IPsec protects IP traffic, but the secure tunnel does not appear by magic. Peers must agree on cryptographic parameters, authenticate each other, establish keying material, define which traffic is protected, and maintain that state over time. IKEv2 is the control protocol that coordinates those steps for many modern site-to-site and remote-access VPN deployments.
Understanding the negotiation sequence makes VPN troubleshooting much easier. Instead of treating a failed tunnel as one problem, you can ask whether the peers reached each other, agreed on algorithms, authenticated successfully, created IPsec security associations, matched traffic selectors, and kept the tunnel alive. That reasoning is central to advanced network engineering.
Separate IKE security from IPsec data protection
IKEv2 creates and manages security associations used to protect negotiation traffic and then establishes child security associations that protect application packets. This separation is useful conceptually: the IKE SA secures the relationship between the peers, while CHILD_SA state defines how actual user or application traffic is encrypted and authenticated.
The first exchange builds a secure channel and derives keying material. Later exchanges authenticate identities and negotiate the traffic protection state. Rekeying, additional child SAs, and informational messages then maintain the relationship without rebuilding everything from scratch.
This is why a VPN can show signs of IKE connectivity but still pass no user traffic. The control plane may have succeeded while child-SA selectors, routing, policy, or data-plane encapsulation is wrong.
IKE_SA_INIT establishes the cryptographic foundation
During IKE_SA_INIT, peers exchange proposals for algorithms and a Diffie-Hellman group, provide nonces, and perform the key exchange needed to derive shared secret material. If the peers have no compatible proposal, negotiation stops before identity authentication even begins.
Troubleshooting at this stage focuses on basic reachability and proposal compatibility. Confirm that UDP 500 can pass, the peers are addressing the intended endpoints, NAT behavior is understood, and both sides share at least one compatible set of encryption, integrity or authenticated-encryption, pseudorandom-function, and Diffie-Hellman parameters as required by the implementation.
Algorithm choice should follow current platform guidance rather than habit. Old ciphers or weak groups may still exist for interoperability, but a working tunnel is not automatically a well-designed tunnel. Security policy should define acceptable suites and a controlled migration path for legacy peers.
IKE_AUTH proves identity and creates the first protected relationship
After the initial key exchange, IKE_AUTH authenticates the peers and commonly creates the first CHILD_SA. Authentication can use pre-shared keys, digital certificates, or other methods supported by the VPN design. The peers also exchange identities, which means a certificate can be valid yet still fail if the expected identity does not match policy.
Certificate-based VPNs add dependencies on certificate validity, trust chains, key usage, revocation handling, and time. Pre-shared keys remove PKI complexity but create their own lifecycle and distribution problems, especially at scale. The choice should reflect the number of peers, automation needs, and assurance requirements.
When authentication fails, do not immediately change encryption proposals. Inspect the exact stage. Logs that show successful IKE_SA_INIT followed by authentication failure point toward credentials, certificates, identity matching, or authorization rather than basic network reachability.
Traffic selectors define what the tunnel actually protects
IPsec needs a definition of protected traffic. Depending on the platform and tunnel model, peers negotiate selectors that represent local and remote address ranges and sometimes protocol or port details. If the two sides disagree about what should be protected, the IKE SA may stay up while the expected application flow has no usable child SA.
This issue commonly appears when one side summarizes networks differently, an administrator reverses local and remote prefixes, or a route-based implementation interacts with a policy-based peer. The configuration can look correct in isolation and still fail because the peers describe the protected traffic differently.
The existing explanation of site-to-site IPsec tunnels is useful background, while negotiation analysis goes deeper into why the peers either accept or reject each other’s security and traffic proposals.
NAT traversal changes encapsulation, not the security goal
IKEv2 peers can detect network address translation in the path and use NAT traversal, commonly moving protected traffic to UDP 4500. This allows IPsec to cross devices that would otherwise have difficulty handling ESP through address translation.
A common troubleshooting mistake is to permit UDP 500 but forget UDP 4500 or to assume that seeing 4500 means the tunnel is misconfigured. In a NATed design, that behavior can be expected. Packet captures and VPN logs should show whether NAT was detected and whether the peers switched transport as intended.
For cloud and hybrid designs, NAT placement matters because it can change peer identity, routing symmetry, and which public address the remote side expects. The tunnel configuration must match the architecture that packets actually traverse.
Rekeying and liveness are part of normal operation: Security associations have lifetimes. Peers periodically create new keys and replace old SAs so long-lived tunnels do not use the same keying material indefinitely. Rekey behavior is normal, but mismatched lifetimes, platform bugs, or stale state can produce intermittent failures that appear only after the tunnel has been healthy for hours.
IKEv2 also supports liveness detection and informational exchanges that help peers recognize when the other side is unavailable. Implementations may expose dead-peer detection settings, retry timers, and tunnel monitoring features. Aggressive timers can cause unnecessary churn across unstable links, while very slow detection can delay failover.
High-availability VPN design should therefore test steady state and transition state. A tunnel that forms successfully during a maintenance window may still fail to recover cleanly after a link flap, peer restart, address change, or routing failover.
Route-based VPNs still depend on correct security negotiation
Route-based VPNs make forwarding decisions through virtual tunnel interfaces and routing tables, which often simplifies large or dynamic environments. They do not eliminate IKEv2 or IPsec state. The routing plane decides which traffic enters the tunnel; IKE and IPsec still establish the protection used to carry it.
This distinction is important in Microsoft AZ-700 and Fortinet environments such as FortiGate Administrator 7.6. Route propagation, BGP, static routes, security policy, and tunnel negotiation have to agree. A route can point perfectly toward a tunnel that has no valid child SA, or the SA can be healthy while routing sends traffic somewhere else.
For engineers coming from host-oriented IPsec, the distinction between transport mode and tunnel mode is also useful. Site-to-site gateways normally protect traffic on behalf of other systems, so encapsulation and selector design must reflect that role.
Troubleshoot the negotiation in order
A disciplined workflow starts outside the cryptography. Verify addresses, routing, UDP reachability, firewalls, and NAT. Then confirm that the peers exchange IKE_SA_INIT messages and select a proposal. Next check identity authentication, certificate or key validity, and authorization. Only after that should you inspect child-SA selectors, routes, firewall policy, and application return traffic.
Use both peer logs when possible. One device may report a generic timeout while the other states exactly which proposal, identity, or selector it rejected. Packet captures can confirm whether messages are sent and whether replies return, but encrypted IKE traffic means captures complement rather than replace device logs.
The key skill is to locate the failed stage before changing configuration. Randomly weakening proposals, widening selectors, or disabling validation may make the tunnel form while hiding the real problem. A secure VPN should be both interoperable and explainable.
Read logs as a state machine
VPN logs become far easier to interpret when you treat IKEv2 as a sequence of states. A timeout before any response points toward reachability, filtering, wrong peer addresses, or NAT. A proposal rejection during the initial exchange points toward cryptographic mismatch. A failure after a secure channel begins points toward authentication or identity. A healthy IKE SA with no application traffic points toward child SAs, selectors, routing, or policy.
Do not focus only on the final error string. Review the messages immediately before it and compare both peers. One side may say “peer not responding” because the other side deliberately rejected a proposal and logged the real reason locally. Correlating timestamps, SPI values, peer addresses, and child-SA identifiers prevents unrelated tunnel attempts from being mixed together.
Packet counters can also isolate the direction of failure. If encrypted outbound packets increase but inbound counters remain static, investigate the remote path. If decrypted packets arrive but the application never sees them, inspect local routing and firewall policy. This step-by-step method is faster than repeatedly clearing the tunnel and hoping it recovers.
Interoperability problems often appear after the basic tunnel is already working. Rekey timing, security-association lifetimes, NAT traversal, dead-peer detection, fragmentation, and path MTU can turn a stable lab configuration into an unreliable production VPN. Troubleshooting is easier when the negotiation is split into phases: confirm peer reachability, verify IKE authentication and proposals, inspect the IPsec child security association, then test the protected traffic and return path. That sequence prevents teams from changing cryptographic settings when the real problem is routing, MTU, or an asymmetric policy outside the tunnel itself.
Document the negotiated result, not just the intended configuration
Large VPN estates often contain years of compatibility decisions. The configuration may advertise several proposals while actual peers consistently select one combination. Periodically inventory which algorithms, groups, authentication methods, and lifetimes are really in use so weak legacy settings can be removed safely rather than left forever “just in case.”
Record peer ownership and business purpose alongside technical parameters. When a certificate approaches expiration or a cryptographic standard changes, administrators need to know who can coordinate the remote change. A tunnel with no known owner is operational debt even if it is healthy today.
The strongest design combines secure negotiation with route and application evidence. Engineers should be able to state which peer identities authenticated, which algorithms were selected, which networks or routes are protected, where NAT traversal occurs, how failover behaves, and which logs confirm the tunnel’s health. That is the level of understanding advanced network and security scenarios reward.