IPsec Explained: AH vs. ESP, Transport Mode vs. Tunnel Mode

IPsec quietly powers most corporate VPNs behind the scenes — here's how AH, ESP, transport mode, tunnel mode, and IKE all fit together.

What IPsec is — a security toolkit that works at the IP layer

Unlike security technologies such as HTTPS that operate at the application (browser) level, IPsec (IP Security) works one layer down, at the IP layer itself. That means it can automatically encrypt and authenticate traffic from any application the moment it goes out as an IP packet, which is why it's the core technology behind most corporate site-to-site and remote-access VPNs.

AH (Authentication Header) — verifies integrity and origin, doesn't encrypt

AH (Authentication Header) attaches signature-like authentication data to a packet, letting the receiver confirm the packet wasn't tampered with in transit and really came from the claimed sender. It doesn't encrypt the actual content, though, so it can't prevent eavesdropping — which is why ESP, which adds encryption, is far more widely used in practice today.

ESP (Encapsulating Security Payload) — adds encryption, the standard in practice

ESP (Encapsulating Security Payload) encrypts the actual data content to prevent eavesdropping, while also providing integrity verification. Because it delivers both encryption and authentication, and handles firewall/NAT environments more gracefully than AH (AH signs the entire header, which tends to break when packets pass through NAT, while ESP is comparatively more flexible here), nearly all modern IPsec implementations use ESP as the default.

Transport Mode — protects only the payload, keeps the original IP header

In transport mode, the original IP header is left untouched, and only the actual data (payload) inside is encrypted and authenticated. It's mainly used for direct communication between two devices, has relatively low overhead, but exposes the real source and destination IP addresses of the two communicating devices.

Tunnel Mode — wraps the entire packet in a new IP header

Tunnel mode encrypts the entire original packet, header included, and wraps it with a brand-new IP header for delivery over a new path between gateways. As a result, the internal network's actual IP ranges stay hidden from outside view, functioning like a secure tunnel between two locations — this is the most common mode used for corporate site-to-site VPNs.

IKE — how both sides negotiate encryption keys safely

IKE (Internet Key Exchange) is the process by which two parties safely agree on which encryption method and key to use before starting IPsec communication. IKE uses techniques like Diffie-Hellman key exchange to generate a shared key securely even in the presence of an eavesdropper, then uses that key to establish the security association (SA) needed for actual AH/ESP communication. The improved IKEv2 is widely used today.

Real-world use — site-to-site VPNs and L2TP/IPsec

In corporate environments, IPsec (typically paired with IKEv2) is widely used for always-on site-to-site VPNs connecting branch offices to headquarters. For personal VPN services and remote-access VPNs, L2TP/IPsec — combining the L2TP tunneling protocol, which has no built-in encryption of its own, with IPsec — is common, with IPsec handling the actual encryption and authentication while L2TP handles building the tunnel itself.

Working one layer below the application makes IPsec universal

Because IPsec operates at the IP layer rather than inside any single application, it protects traffic regardless of which app generated it — a browser, an email client, or a database connection all get the same protection automatically once IPsec is active on the connection. That's the core reason it became the default choice for securing entire network links rather than individual app sessions.

ESP and tunnel mode became the practical default combination

While AH and transport mode both exist as valid parts of the IPsec standard, real-world deployments have converged heavily on ESP (for encryption plus authentication) paired with tunnel mode (for full packet protection and hidden internal addressing) — largely because that combination handles modern network realities like NAT and firewalls far more smoothly than the alternatives.

Frequently Asked Questions

Do I need to choose between AH and ESP myself when setting up a VPN?

Rarely — nearly all modern VPN software and hardware default to ESP because it covers both encryption and authentication, and most consumer or business VPN setups won't even present AH as a practical option.

What's the simplest way to remember the difference between transport and tunnel mode?

Transport mode protects the letter but keeps the original envelope showing; tunnel mode seals the entire original letter and envelope inside a brand-new envelope with a different address on it.