TECHNOLOGY & CERTIFICATION EDITORIAL

IPsec Site-to-Site VPN Tunnels: How They Work and Function

A site-to-site IPsec VPN connects two routed networks across an untrusted path, often the public internet. Its job is not to create a magical private wire: each gateway authenticates the peer, negotiates cryptographic parameters, establishes security associations and then protects the traffic that the configured policy selects. A healthy tunnel therefore needs working routes, compatible traffic selectors, suitable credentials and packet forwarding, not merely a green VPN status icon.

Understanding the separation between key negotiation and user traffic is the fastest way to troubleshoot these links. An IKE security association can exist while no usable data traffic flows; the reverse cannot provide a stable protected site-to-site service.

Tunnel mode protects an inner packet

In common IPsec tunnel mode, a gateway takes an original IP packet from a protected network and places it inside a new outer IP packet whose endpoints identify the VPN gateways. Encapsulating Security Payload (ESP) protects the selected content with encryption and integrity/authentication as negotiated. The peer gateway validates and decrypts the packet, then forwards the original inner packet according to routing and security policy.

The outer path still matters. Public WAN links, ISP filters, NAT devices and MTU constraints affect whether the encapsulated traffic reaches the peer. A tunnel does not make bandwidth infinite or hide all metadata: observers can still see outer IP addresses, packet sizes and timing, even when inner payload confidentiality is correctly provided.

Traditional deployments usually rely on ESP rather than treating IPsec’s Authentication Header as an interchangeable encrypted tunnel format. ESP with contemporary authenticated encryption such as AES-GCM can provide confidentiality and integrity when negotiated and implemented correctly. Algorithm support and recommendations depend on the gateways’ firmware and security policy.

IKEv2 establishes trust before data can move

IKEv2 organizes peer negotiation into two main stages. IKE_SA_INIT negotiates cryptographic proposals and exchanges nonces and key-establishment material. IKE_AUTH authenticates the peers, exchanges identity information and commonly establishes the first Child Security Association for ESP-protected traffic. Later CREATE_CHILD_SA exchanges can create or rekey Child SAs without repeating the original initial handshake.

A Security Association (SA) describes the parameters and state used for protected communication. IKE SA and Child SA are different objects. The IKE SA protects further key-management exchanges; Child SAs protect the application/network traffic selected for ESP. Seeing an established IKE SA but no working Child SA can point to selector, authorization or proposal problems that a connectivity ping alone will not explain.

IKEv2 terminology follows RFC 7296. Understanding the IKEv2 negotiation exchanges and their diagnostic signals helps explain failures inside an IPsec tunnel; the focus here remains the complete site-to-site design.

Traffic selectors and routing must agree

Suppose Office A uses 10.10.0.0/16 and Office B uses 10.20.0.0/16. An IPsec policy can authorize packets from A’s prefix to B’s prefix in one direction and the corresponding reverse traffic in the other. In IKEv2, negotiated Traffic Selector payloads identify which packet addresses and ranges a Child SA may carry. The peers must accept compatible selectors and policy authorization.

A selector mismatch may permit successful IKE authentication but leave the Child SA unable to establish, or protect only a subset of the intended network. If the remote prefix recently changed, both sides need synchronized policy and routing information. Two networks using overlapping private address space may require address redesign, carefully planned translation or segmented routing; an IPsec tunnel by itself does not resolve overlap.

Policy-based VPNs often tie protection to configured local/remote subnet pairs and security-policy databases. Route-based VPNs usually provide a virtual tunnel interface or equivalent route target, making routing policy central to which packets enter the protected path. Product-specific implementation details vary. Neither category removes the need to verify traffic selectors, firewall rules and return routes.

Layer of operation Typical failure Evidence to collect
Underlay transport Peer WAN endpoint not reachable Routing, ISP path, UDP 500/4500, NAT state
IKE authentication Certificate/PSK/identity mismatch IKE_AUTH logs, peer IDs, certificate chain, clock validity
Child SA negotiation Selector or transform mismatch Proposals, TSi/TSr, ESP SA state
Overlay routing Traffic never enters tunnel Route tables, tunnel interface, policy matching
Application connectivity Protected packets arrive but app fails Decapsulation counters, firewalls, DNS and service health
MTU and fragmentation Small packets work, large traffic stalls Path MTU, encapsulation overhead, MSS behavior

NAT traversal and the outer transport

ESP is an IP protocol, not simply another UDP or TCP port. When a gateway sits behind NAT or an intermediate device needs to track state, NAT traversal may encapsulate ESP in UDP, commonly on port 4500, after negotiation detects the relevant environment. IKE signaling commonly begins on UDP 500 and may move or use UDP 4500 as required. Firewalls must allow the actual negotiated path, but blindly opening a broad port range is not a substitute for understanding the peers.

NAT traversal helps carry protected traffic through address translation; it does not automatically make the inner protected subnets reachable or fix a missing return route. Some provider architectures impose endpoint, symmetry or timeout limitations that still affect stable VPN service. Test renegotiation after an idle period and after external IP changes where appropriate.

Build a deployment around operational evidence

Before creating a tunnel, record the public peer endpoints, accepted identities, local and remote prefixes, encryption/authentication proposals, IKE version, lifetime/rekey policy, intended routing, failover and monitoring expectations. Plan certificate management or a protected pre-shared-key process, depending on gateway capabilities and policy. Both parties should verify the same operating assumptions, not just exchange screenshots.

Deploy the tunnel in a controlled change window. Establish peer reachability, authenticate IKE, confirm Child SAs, then send permitted test traffic from a host in each protected subnet. Inspect encapsulated and decapsulated counters and the return path. A one-direction-only counter increase often indicates asymmetric routing, a missing remote policy, a firewall rule or a NAT translation mismatch.

Use a specific test, such as reaching a service host and port, rather than an indiscriminate ping. Some networks block ICMP while allowing the intended application. DNS failures inside the VPN may also masquerade as network reachability failures; resolve the actual destination name and inspect which DNS server the client queries.

Rekeying, replay protection and availability

IPsec SAs are not permanent. Their lifetimes and key-establishment policy trigger rekeying or replacement. A connection may work initially and fail around its rekey interval if peers disagree about proposals, identity or allowable selectors. Anti-replay protection tracks packet sequences to reject certain replays; unusual packet reordering or poor failover design can affect those checks.

For resilience, define what happens when a peer or WAN circuit fails. A redundant gateway may need route convergence, state re-establishment, updated endpoint identity and firewall allowance before protected traffic returns. Health probes and dead-peer detection help identify missing peers but do not themselves prove that the full application path is available. Test an actual controlled failover and document which devices own each recovery step.

Encapsulation reduces usable packet payload relative to the underlay MTU. If small packets work while large requests hang, investigate path MTU discovery, ICMP filtering, MSS clamping and fragmentation policy before changing cryptographic proposals at random. Each gateway vendor has specific counters and logging for these conditions.

Troubleshoot in the order packets travel

Start outside: can each gateway reach the other’s public endpoint, and can the negotiated IKE/NAT-T traffic traverse the path? Next check IKE identities and proposals, then Child SA status and traffic selectors. Once SAs are established, inspect the route that sends the actual source and destination through the tunnel. Finally verify the far-side firewall, endpoint and reverse route.

This order prevents a common mistake: repeatedly deleting and rebuilding a cryptographically healthy tunnel when the actual failure is a new route, changed prefix or denied application port. Preserve logs and counters before reset. Confirm the error from both gateways, because one peer’s “negotiation failed” summary often omits the selector or identity detail visible at the other end.

A secure site-to-site VPN is both a cryptographic relationship and a functioning routed service. Design the two halves together, verify each at the right layer, and maintain a runbook for rekeying, failover and source-prefix changes. Strong encryption cannot compensate for an incorrect route, and a green IKE session is not evidence that every protected application can communicate.

Back to Insights
Explore what matters. Knowledge that goes beyond the exam.
Explore ExamTopics