Which Layer Is It? Mapping Attacks to the OSI Model, Plus IPsec AH vs. ESP
Chris Rees
25+ years in IT · Pluralsight author, 4.6/5 across 2,000+ ratings
Everyone who sits the CISSP has memorized the OSI layers at least twice before. What Domain 4 adds is the question that separates networking knowledge from security knowledge: at which layer does this attack happen, and therefore which control can see it? ARP poisoning at layer 2 is invisible to a layer 3 firewall. A SQL injection at layer 7 sails through a layer 4 packet filter. Get the layer right and the control follows.
Why the layer matters
A control operates at a layer, and it can only see what that layer exposes. A packet-filtering firewall reads layer 3 and 4 headers; it cannot see the HTTP request inside. A switch operates at layer 2; it cannot see IP addresses. When a scenario describes an attack and asks for the control, the layer of the attack tells you which controls are even candidates. That is what objective 4.1 is testing when it asks you to apply secure design principles in network architectures.
The seven layers, with the attack the exam pins to each
Three rows carry most of the questions. ARP poisoning is layer 2, even though it manipulates the mapping to IP addresses: the frames it forges and the broadcast domain it affects are data link, which is why a router never sees it and a switch with dynamic ARP inspection does. SYN floods and port scans are layer 4, because ports and TCP handshakes are transport concepts, which is why a stateful firewall can see them and a layer 2 switch cannot. Injection is layer 7: it lives inside an application request, so only something that parses the application protocol, a proxy or a web application firewall, has any chance of catching it.
Two rows are deliberately fuzzy and the exam knows it. TLS is usually placed at presentation or session, and different sources disagree; the exam avoids asking the layer of TLS directly and instead asks what it protects. And layers 5 and 6 have few dedicated devices, which is the point: their attacks are software attacks, and the controls are in the application stack.
The control that matches the layer
| Layer of the attack | Control that can see it | Control that cannot |
|---|---|---|
| 7 application | web application firewall, proxy, input validation in code | packet filter, switch |
| 5 and 6 session and presentation | TLS with proper validation, session management in the application | any network device |
| 4 transport | stateful firewall, SYN cookies, rate limiting | switch, physical controls |
| 3 network | router ACLs, packet filter, IPsec, anti-spoofing filters | switch port security |
| 2 data link | switch port security, dynamic ARP inspection, 802.1X, VLANs | firewall, router |
| 1 physical | locked closets, conduit, fiber, tamper-evident seals, shielding | everything above it |
Read the third column when you are eliminating answers. A scenario about ARP poisoning that offers a next-generation firewall as an answer is offering a device that cannot see the attack, however capable it is at the layers it does see.
Secure protocols, by the layer they protect
Objective 4.1 lists the secure protocols you must recognize, and placing them by layer explains what each can and cannot protect. IPsec works at layer 3 and protects every protocol above it, transparently, which is why it suits VPNs. TLS works above the transport layer and protects a single application session, which is why each application has to be configured to use it. SSH is an application-layer protocol that both secures remote administration and tunnels other traffic. 802.1X operates at layer 2 and controls whether a device gets a network at all, before it has an IP address. Wireless security (WPA3) is likewise a layer 2 protection for the radio link and nothing beyond the access point.
The exam's use of this: a scenario where traffic is protected on the wireless link but exposed on the wired network beyond the access point is describing the limit of a layer 2 control, and the answer is a higher-layer protection such as TLS or IPsec end to end.
TCP/IP: the same stack in four layers
The model the internet actually runs on collapses OSI into four: the link layer (OSI 1 and 2), the internet layer (OSI 3), the transport layer (OSI 4), and the application layer (OSI 5, 6 and 7). The exam tests the mapping in both directions. A question about "the internet layer" wants IP and layer 3 attacks; a question about "the application layer" of TCP/IP covers everything OSI splits into three, so it includes session hijacking as well as HTTP attacks.
IPsec, pulled apart
IPsec is the layer 3 security protocol, which means it protects everything above it without the applications knowing. The exam tests its four components and its two modes, and candidates lose marks by treating IPsec as one thing.
Authentication Header (AH) provides integrity and origin authentication for the packet, including most of the IP header. It does not encrypt anything, and because it hashes the header, it breaks when a NAT device rewrites addresses. Encapsulating Security Payload (ESP) provides confidentiality by encrypting the payload, plus integrity and authentication of the payload. ESP is what nearly every real deployment uses. The single most tested IPsec fact: AH does not provide confidentiality; ESP does.
Internet Key Exchange (IKE) is the negotiation: it authenticates the two endpoints, agrees the algorithms and generates the keys. The result of the negotiation is a security association (SA): a one-way agreement covering the algorithms, keys and lifetime for traffic in one direction, so a two-way connection needs two. Each SA is identified by a security parameter index (SPI) carried in the packet header, which is how the receiver knows which keys to apply.
Transport mode encrypts the payload and leaves the original IP header in place. It is used between two hosts that are the actual endpoints of the traffic. Tunnel mode encrypts the entire original packet, header included, and wraps it in a new IP header addressed between two gateways. It is what site-to-site VPNs use, and it hides the internal addresses of both networks from anyone watching the link. A scenario about connecting two offices over the internet is tunnel mode with ESP; one about protecting traffic between a server and a client on the same LAN is transport mode.
Segmentation and direction
Objective 4.1 also names the grades of segmentation and the direction of traffic, and both are layer questions in disguise. Physical separation, including an air gap, is layer 1: no cable, no path. Logical separation with VLANs is layer 2 and with subnets and VPNs is layer 3. Micro-segmentation enforces policy per workload, which is what Zero Trust needs and what a perimeter design never had.
North-south traffic crosses the perimeter and is the traffic every organization inspects. East-west traffic moves laterally inside, between servers and between segments, and is where most of a modern intrusion happens and where the least inspection exists. A scenario about an attacker moving from a compromised workstation to a database server undetected is an east-west problem, and the answer is internal segmentation and inspection, not a bigger perimeter firewall.
Four stems, placed
- Users on one office floor are being redirected to a fake login page, and the router logs show nothing unusual. The router sees nothing because the attack is below it: ARP poisoning at layer 2. Dynamic ARP inspection on the switches.
- Two branch offices need their internal servers to communicate over the internet without exposing internal addressing. Gateway to gateway with hidden internal headers: IPsec in tunnel mode with ESP.
- An auditor asks why the site-to-site VPN uses ESP rather than AH. AH provides no confidentiality and fails through NAT. ESP provides encryption and integrity, and survives address translation.
- A compromised workstation reached a database server two segments away, and the perimeter firewall never saw a packet. East-west traffic never crosses the perimeter. Internal segmentation with inspection between segments, and micro-segmentation for the database tier.
Exam traps
- The device decides the visibility. A control cannot see attacks at a layer above the one it operates on.
- ARP is layer 2, no matter how many IP addresses are involved. Smurf and IP spoofing are layer 3. SYN floods are layer 4.
- AH authenticates and cannot encrypt; ESP does both. AH also breaks through NAT because it hashes the header.
- A security association is one-way and identified by an SPI. Two are needed for a conversation; IKE builds them.
- Tunnel mode for gateways, transport mode for hosts. If the internal addresses must be hidden, it is tunnel mode.
Key takeaways
- Know each OSI layer's unit, device, protocols and attack. Injection at 7, session hijacking at 5, SYN flood at 4, spoofing and smurf at 3, ARP poisoning at 2, tapping at 1.
- TCP/IP collapses OSI to link, internet, transport and application. Questions test the mapping both ways.
- IPsec is AH (integrity, no confidentiality) and ESP (confidentiality plus integrity), negotiated by IKE into one-way security associations identified by SPIs.
- Transport mode protects the payload host to host; tunnel mode wraps the whole packet gateway to gateway.
- East-west traffic is where intrusions move and where inspection is thinnest. Segmentation grades are physical, logical and micro.
Secure network architecture is objective 4.1 of CISSP Domain 4, Communication and Network Security, 13% of the exam, and it is the reference-model view of the zones, firewalls and secure protocols Security+ approaches from the deployment side. Work all 19 topics across three objectives with our CISSP Domain 4 study guide.
#CISSP #ISC2 #OSIModel #IPsec #NetworkSecurity #Segmentation #SecureProtocols #NetworkArchitecture #InfoSec #CyberSecurity
Master the network domain
Secure network architecture is the largest objective in CISSP Domain 4, 13% of the exam. Work all 19 topics across three objectives, from the reference models and IPsec through securing network components and every communication channel, with 60 practice questions.
Get the CISSP Domain 4 guideKeep reading
Ciphertext-Only to Golden Ticket: The Thirteen Cryptanalytic Attacks CISSP Tests, and What Each One Actually Breaks
AES has never been broken and the CISSP knows it, which is why objective 3.7 is not about the math. It is about the nonce that repeated, the comparison that returned early, the certificate nobody validated and the hash that worked as well as the password. Every named cryptanalytic attack sorted by what the attacker holds, why 2DES failed, why a 128-bit hash gives only 64 bits of collision resistance, and the one countermeasure that belongs to each attack.
Read CISSP deep divesRoots of Trust: TPM, Secure Boot vs. Measured Boot, and the Trusted Computing Base
Every security control runs on something, and if that something is compromised the control is theater. CISSP Domain 3 on the hardware and firmware foundations: the trusted computing base and the reference monitor, protection rings, what a TPM actually stores and what a platform configuration register is for, why secure boot and measured boot answer different questions, and where trusted execution environments and hardware security modules fit.
Read CISSP deep divesClear, Purge, Destroy: Data Remanence, Sanitization, and End-of-Life the CISSP Way
Deleted data is not gone, and the CISSP asks about the difference in a dozen ways. Domain 2 on what happens after the useful life of data and the assets that hold it: remanence and why deletion is a lie, the clear, purge and destroy ladder and which media each applies to, cryptographic erasure and the SSD problem, end-of-life versus end-of-support, and the ownership roles that decide who is accountable at every step.
ReadEnjoyed this? Get the AI security news that matters.
Join The AI Security Brief for the top AI security news, plus what's important to the C-suite. Free, straight to your inbox.
No spam. Unsubscribe anytime.