How VPNs Actually Work: From Tunneling Protocols to the Hidden Latency Costs

In 2019, a network engineer at a major financial institution noticed something odd. Their newly deployed VPN, configured with OpenVPN over TCP, was causing a 40% drop in throughput for database replication traffic. The latency between their New York and London data centers had jumped from 75ms to over 200ms. After weeks of troubleshooting, they discovered the culprit wasn’t bandwidth or hardware—it was TCP-over-TCP meltdown, a fundamental interaction between the VPN protocol and the underlying transport layer.

This isn’t an isolated incident. Every VPN connection carries hidden costs: encapsulation overhead, encryption latency, and protocol-specific quirks that can transform a 100Mbps link into something that struggles to deliver 30Mbps. Understanding these mechanics isn’t just academic—it’s essential for building systems that actually work at scale.

What a VPN Actually Does at the Packet Level

A VPN doesn’t create a “secure pipe” in any physical sense. It creates a logical tunnel through packet encapsulation. Your original IP packet—the one carrying your HTTP request or database query—gets wrapped inside another IP packet with different source and destination addresses.

Consider a simple scenario: your laptop (192.168.1.100) wants to reach a server at 10.0.0.50 through a VPN gateway at 203.0.113.1. Without a VPN, the packet travels directly. With a VPN, the original packet becomes the payload of a new packet:

Original packet:
[IP header: src=192.168.1.100, dst=10.0.0.50][TCP header][HTTP data]

Encapsulated packet:
[IP header: src=192.168.1.100, dst=203.0.113.1][VPN header][ENCRYPTED:
  [IP header: src=192.168.1.100, dst=10.0.0.50][TCP header][HTTP data]
]

This encapsulation happens at layer 3 (IPsec) or layer 4 (OpenVPN, WireGuard over UDP). The choice matters profoundly for performance, NAT traversal, and failure modes.

IPsec: The Kernel-Level Heavyweight

IPsec operates at the network layer, making it nearly invisible to applications. It consists of two main protocols:

ESP (Encapsulating Security Payload) provides encryption, authentication, and integrity. It’s the workhorse of IPsec VPNs, supporting both transport mode (encrypts only payload) and tunnel mode (encrypts the entire original packet).

IKE (Internet Key Exchange) handles the authentication and key exchange. IKEv2, the current standard, establishes security associations through a two-phase process:

1. IKE_SA (Phase 1): Peers authenticate each other and establish a secure channel for negotiation. This typically involves Diffie-Hellman key exchange and can use pre-shared keys, certificates, or EAP authentication.

2. CHILD_SA (Phase 2): Within the protected IKE_SA channel, peers negotiate the actual IPsec security association that will carry user traffic.

The ESP packet format adds substantial overhead:

Image source: NetworkLessons - IPsec (Internet Protocol Security)

In tunnel mode, ESP adds:

• SPI (Security Parameters Index): 4 bytes
• Sequence Number: 4 bytes
• IV (Initialization Vector): typically 8-16 bytes depending on cipher
• Padding: 0-255 bytes (to align to cipher block size)
• Pad Length: 1 byte
• Next Header: 1 byte
• ICV (Integrity Check Value): typically 12-16 bytes

Plus a new IP header (20 bytes for IPv4). That’s 50-70 bytes of overhead per packet before you even count the encryption expansion.

NAT Traversal: UDP Encapsulation

IPsec’s ESP protocol uses protocol number 50, which NAT devices typically cannot translate—they can’t modify the encrypted payload to update checksums. RFC 3948 solved this by wrapping ESP inside UDP packets on port 4500:

[IP header][UDP header (port 4500)][ESP header][Encrypted payload]

This adds another 8 bytes for the UDP header, bringing total overhead to 60-80 bytes for a typical IPsec tunnel mode packet.

OpenVPN: TLS Over UDP (or TCP)

OpenVPN takes a different approach. Instead of operating at the IP layer, it runs as a userspace process that creates a virtual network interface (TUN/TAP). All traffic through this interface gets encrypted and sent over a TLS channel.

The architecture is elegantly simple:

Application → TUN device → OpenVPN process → TLS encryption → UDP/TCP socket → Network

OpenVPN’s packet format reflects its TLS heritage:

P_DATA message:
[Packet opcode/key_id (1 byte)][HMAC (16-20 bytes, optional)][IV (variable)][Encrypted payload][Auth tag]

The control channel uses a reliable transport layer built on top of UDP, implementing ACK and retransmission for TLS handshake packets. This is where things get interesting—and problematic.

The TCP Meltdown Problem

When OpenVPN runs over TCP, you create a situation where TCP congestion control operates twice: once for the outer TCP connection carrying the VPN tunnel, and once for each inner TCP connection inside the tunnel.

Here’s what happens when packet loss occurs:

1. The outer TCP detects loss and reduces its congestion window
2. The inner TCP also detects loss (because the outer TCP’s retransmission delay appears as increased RTT) and reduces its congestion window
3. Both layers compete for retransmission, creating exponential backoff

Research from Carnegie Mellon University demonstrated that TCP-over-TCP can reduce throughput by 40-60% compared to TCP-over-UDP in lossy networks. The meltdown isn’t theoretical—it’s measurable and predictable.

sequenceDiagram
    participant App as Application
    participant InnerTCP as Inner TCP
    participant VPN as VPN Layer
    participant OuterTCP as Outer TCP
    participant Net as Network

    Note over App,Net: Normal operation
    App->>InnerTCP: Send data
    InnerTCP->>VPN: Segment
    VPN->>OuterTCP: Encapsulate
    OuterTCP->>Net: Transmit

    Note over App,Net: Packet loss scenario
    Net--xOuterTCP: Packet lost
    OuterTCP->>OuterTCP: Reduce cwnd, retransmit
    InnerTCP->>InnerTCP: Timeout (long RTT), reduce cwnd
    Note over InnerTCP,OuterTCP: Double congestion control = meltdown

This is why OpenVPN over UDP is the recommended configuration for production deployments. The UDP outer layer doesn’t implement congestion control, letting the inner TCP behave naturally.

WireGuard: Modern Cryptography, Minimal Overhead

WireGuard represents a fundamental rethink of VPN protocol design. Jason Donenfeld’s creation, now part of the Linux kernel, achieves in roughly 4,000 lines of code what OpenVPN requires over 100,000 lines to accomplish.

The key innovations:

Noise Protocol Framework

WireGuard uses the Noise_IK handshake pattern, combining Curve25519 for key exchange with ChaCha20-Poly1305 for authenticated encryption. The handshake completes in just 1.5 round trips:

Message 1 (Initiator → Responder):

handshake_initiation {
    message_type: 1
    sender_index: random_32bit
    unencrypted_ephemeral: 32 bytes (Curve25519 public key)
    encrypted_static: 48 bytes (initiator's static public key)
    encrypted_timestamp: 28 bytes (TAI64N timestamp)
    mac1: 16 bytes
    mac2: 16 bytes
}

Message 2 (Responder → Initiator):

handshake_response {
    message_type: 2
    sender_index: random_32bit
    receiver_index: initiator's sender_index
    unencrypted_ephemeral: 32 bytes
    encrypted_nothing: 16 bytes
    mac1: 16 bytes
    mac2: 16 bytes
}

After these two messages, both parties derive session keys using HKDF and can begin sending data. The entire handshake is just 148 bytes in each direction.

Cryptographic Primitives

WireGuard hardcodes its cryptographic choices, eliminating negotiation overhead:

Component	Algorithm	Rationale
Key Exchange	Curve25519	Fast, constant-time ECDH
Encryption	ChaCha20-Poly1305	AEAD, no hardware acceleration needed
Hashing	BLAKE2s	Faster than SHA-256, same security margin
Key Derivation	HKDF	Standard KDF construction

The choice of ChaCha20-Poly1305 is significant. On systems without AES-NI hardware acceleration, ChaCha20-Poly1305 outperforms AES-GCM by 2-3x. Even on systems with AES-NI, the performance difference is negligible, making ChaCha20-Poly1305 the better default choice.

Data Packet Format

WireGuard data packets are remarkably compact:

packet_data {
    message_type: 4 (1 byte)
    reserved: 0 (3 bytes)
    receiver_index: 4 bytes
    counter: 8 bytes
    encrypted_encapsulated_packet: variable + 16 bytes Poly1305 tag
}

Total overhead: 32 bytes per packet (including the Poly1305 authentication tag). Compare this to IPsec’s 60-80 bytes or OpenVPN’s 50-80 bytes.

The Latency Mathematics

Why does VPN latency matter more than throughput for many applications? Because of how TCP congestion control interacts with round-trip time.

TCP throughput is bounded by:

$$\text{Throughput} \approx \frac{\text{MSS} \times 1.22}{\text{RTT} \times \sqrt{\text{Loss}}}$$

If your VPN adds 30ms of latency to each round trip, you’re not just paying 30ms—you’re reducing maximum throughput. For a connection with 0.1% loss and 1460-byte MSS:

• RTT = 50ms: Throughput ≈ 80 Mbps
• RTT = 80ms (with VPN): Throughput ≈ 50 Mbps

That’s a 37% reduction from latency alone, before accounting for encryption overhead or packet expansion.

The Speed of Light Isn’t Negotiable

Light travels at approximately 200,000 km/s in fiber optic cable (about 2/3 of vacuum speed). For a New York to London connection (5,585 km), the theoretical minimum one-way latency is:

$$\frac{5,585 \text{ km}}{200,000 \text{ km/s}} = 27.9 \text{ ms}$$

In practice, routers, switches, and protocol overhead add 20-40ms. A VPN server that routes you through an additional hop adds its own latency contribution. If your VPN provider’s server is in a different city than your optimal route, you might add 50-100ms just from suboptimal routing.

MTU, Fragmentation, and Performance Degradation

When a VPN adds headers to each packet, those packets become larger. If they exceed the path MTU (typically 1500 bytes on Ethernet), something must give.

Options:

1. Fragmentation: The packet is split into multiple fragments. This is expensive—each fragment requires its own header, and loss of any fragment means retransmission of the entire original packet.

2. Path MTU Discovery: ICMP messages discover the smallest MTU along the path. But many firewalls block ICMP, causing black-hole fragmentation.

3. Reduced tunnel MTU: Configure the VPN interface with a smaller MTU (e.g., 1400 bytes) to leave room for encapsulation.

WireGuard recommends an MTU of 1420 bytes for IPv4 and 1400 bytes for IPv6. Here’s the calculation:

Ethernet MTU: 1500 bytes
- IPv4 header: 20 bytes
- UDP header: 8 bytes
- WireGuard header: 32 bytes
= Inner MTU: 1440 bytes (conservative: 1420)

For IPsec ESP in tunnel mode with NAT-T:

Ethernet MTU: 1500 bytes
- IPv4 header: 20 bytes
- UDP header: 8 bytes (NAT-T)
- ESP overhead: ~50 bytes
= Inner MTU: ~1422 bytes (conservative: 1400)

Performance Comparison: What the Numbers Say

A 2025 empirical study published in MDPI Cloud Computing compared WireGuard and OpenVPN across Azure and VMware environments. Key findings:

Metric	WireGuard	OpenVPN	Difference
TCP Throughput (VMware baseline)	210.64 Mbps	110.34 Mbps	+91%
TCP Throughput (Azure baseline)	~280 Mbps	~290 Mbps	-3%
CPU Utilization	~4.76%	~3.97%	+20%
Packet Loss (VMware)	12.35%	47.01%	-74%
Connection Initiation	Fastest	Slowest	-

The results show that WireGuard’s advantages are environment-dependent. In virtualized environments like VMware, WireGuard’s kernel-space implementation shows clear throughput benefits. In cloud environments with different virtualization stacks, the difference narrows significantly.

Hardware Acceleration: AES-NI

Modern Intel and AMD processors include AES-NI instructions that accelerate AES encryption. On a system with AES-NI support:

• AES-128-GCM: ~5 cycles per byte
• AES-256-GCM: ~7 cycles per byte
• ChaCha20-Poly1305: ~10 cycles per byte (no hardware acceleration)

Without AES-NI:

• AES-128-GCM: ~15-20 cycles per byte
• ChaCha20-Poly1305: ~10 cycles per byte

This is why WireGuard’s ChaCha20-Poly1305 choice is pragmatic—it performs well everywhere, not just on newer hardware with AES-NI.

Security Trade-offs

Code size directly impacts audit complexity. WireGuard’s ~4,000 lines can be thoroughly reviewed by a small team. OpenVPN’s ~100,000+ lines require more extensive review processes.

But smaller code doesn’t automatically mean more secure. Consider:

WireGuard advantages:

• Formal verification of the Noise protocol
• Minimal attack surface
• Modern cryptographic primitives
• No algorithm negotiation (eliminates downgrade attacks)

OpenVPN advantages:

• Battle-tested over 20+ years
• Flexible authentication (certificates, username/password, OTP)
• Can run over TCP for restrictive networks
• Mature ecosystem and tooling

Both protocols have undergone security audits with no critical vulnerabilities found in their current implementations. The choice often comes down to operational requirements rather than security.

Choosing the Right Protocol

Scenario	Recommended Protocol	Rationale
Site-to-site VPN between data centers	WireGuard or IPsec	Kernel-space efficiency, minimal overhead
Remote access for employees	OpenVPN or IKEv2/IPsec	Flexible authentication, TCP fallback option
Mobile devices on unreliable networks	IKEv2/IPsec with MOBIKE	Native OS support, seamless roaming
Embedded systems with limited CPU	WireGuard	ChaCha20-Poly1305 needs no hardware acceleration
Networks blocking UDP	OpenVPN over TCP	Only option when UDP is filtered
High-security environments	IPsec with AES-256-GCM	FIPS 140-2 compliance available

The fastest VPN protocol is the one correctly configured for your specific constraints. WireGuard’s elegance means nothing if your network blocks UDP. OpenVPN’s flexibility is worthless if you’re paying CPU overhead you don’t need.

The Hidden Costs Summary

When you connect through a VPN, you’re paying several taxes:

1. Encapsulation overhead: 32-80 bytes per packet, reducing effective bandwidth by 2-5%

2. Encryption latency: 0.1-1ms per packet depending on hardware and cipher

3. Protocol handshaking: Initial connection takes 50-500ms depending on protocol and authentication method

4. MTU reduction: Smaller packets mean more packets for the same data, increasing header overhead

5. Routing detours: If your VPN server isn’t on the optimal path, you’re adding unnecessary distance

For a 1500-byte packet on a 100Mbps link with 50ms RTT:

• Without VPN: ~166 packets/second, 200Mbps theoretical throughput
• With VPN (40-byte overhead, 30ms added RTT): ~143 packets/second, 143Mbps theoretical throughput

That’s a 28% throughput reduction from overhead and latency combined.

Understanding these mechanics transforms VPN selection from “pick the fastest one” to “pick the one whose trade-offs match your constraints.” The 4,000-line protocol isn’t always better than the 100,000-line one—but knowing why helps you build systems that actually perform.

---

References

1. Donenfeld, J. A. (2017). “WireGuard: Next Generation Kernel Network Tunnel.” NDSS Symposium 2017.

2. Kent, S. (2005). “IP Encapsulating Security Payload (ESP).” RFC 4303, IETF.

3. Kaufman, C. et al. (2005). “Internet Key Exchange (IKEv2) Protocol.” RFC 4306, IETF.

4. Huttunen, A. et al. (2005). “UDP Encapsulation of IPsec ESP Packets.” RFC 3948, IETF.

5. Perrin, T. (2018). “The Noise Protocol Framework.” Noise Protocol Specification.

6. Dowling, B., Paterson, K. (2018). “A Cryptographic Analysis of the WireGuard Protocol.” IACR Cryptology ePrint Archive.

7. MDPI (2025). “Empirical Performance Analysis of WireGuard vs. OpenVPN in Cloud and Virtualised Environments.” Cloud Computing, 14(8), 326.

8. OpenVPN Technologies. “OpenVPN Protocol Specification.” OpenVPN Community Documentation.

9. Cisco Systems. “IPsec VPN Design Guide.” Cisco Technical Documentation.

10. Cloudflare. “How VPNs Affect Internet Speed.” Cloudflare Learning Center.

11. Bernstein, D. J. (2008). “ChaCha20 and Poly1305.” IETF RFC 7539.

12. Intel Corporation. “Intel AES-NI Instructions: Performance Analysis.” Intel Software Developer Documentation.

13. NetworkLessons. “IPsec (Internet Protocol Security).” Technical Tutorial Series.

14. IVPN. “PPTP vs IPSec IKEv2 vs OpenVPN vs WireGuard.” Protocol Comparison Guide.

15. NIST. “Guide to IPsec VPNs.” Special Publication 800-77 Revision 1.