VPN Tunneling & Encryption: How It Really Works

The initial contact phase is as follows:

Once these steps are complete, the handshake to fully set up encryption can now take place. Here’s how the major VPN protocols handle the initial exchanges:

• OpenVPN, SSL/TLS, SSTP:
  • Client sends Client Hello packet.
  • Server responds with Server Hello packet.
• WireGuard:
  • Client and server exchange handshake initiation and response messages.
  • Messages include ephemeral public keys and cryptographic parameters.
• L2TP/IPSec and IKEv2:
  • Client and server exchange IKE (Internet Key Exchange) initiation packets IKE_SA_INIT.

Many leading VPNs also offer their own proprietary protocols, which are often based on modified versions of WireGuard or OpenVPN and follow initiation processes similar to their base protocols.

Here’s what that first packet actually looks like – an OpenVPN Client Hello captured during one of my own test connections, over TCP on port 443:

^{Back to VPN connection phases overview ↩}

The authentication phase ensures that clients only connect to genuine VPN servers. Authentication methods vary in complexity and security, often combining multiple techniques to ensure a robust connection.

While VPN protocols like OpenVPN and IPSec offer high configurability, most authentication processes involve some combination of certificates, pre-shared keys (PSK), and username/password verification.

VPN Protocol	Primary Authentication Methods	Key Features
OpenVPN	Certificate-based authentication, Username and password	Uses HMAC for packet authentication, creates separate TLS control channel for ongoing session authentication
WireGuard	Public-key cryptography	No traditional handshake process, authentication occurs with the first packet sent, uses pre-shared private and public keys for each peer
IKEv2	Pre-shared keys (PSK), X.509 digital certificates, user-based authentication via EAP (Extensible Authentication Protocol)	Supports sequential authentication for enhanced security, uses encrypted IKE_AUTH packets for privacy
L2TP/IPSec	Two-phase authentication: L2TP authenticates VPN endpoints, IPSec authenticates the user	L2TP uses PPP (Point-to-point Protocol) methods (e.g. PAP, CHAP, MS-CHAP), IPSec establishes an encrypted tunnel before L2TP authentication, ensuring overall security

^{Back to VPN connection phases overview ↩}

Key exchange is the cryptographic process where two parties securely establish a shared secret key over an insecure communication channel. The name is slightly misleading – nothing secret is ever actually sent.

That shared secret key enables symmetric encryption for everything that follows, effectively creating the secure tunnel between client and server.

For a more detailed explanation of how various key exchange methods work, refer to the VPN handshakes section.

Here, I’ll focus on how the most popular VPN protocols handle key exchange:

As you can see, key exchange methods vary significantly between protocols in terms of complexity, performance and most importantly, security.

That flexibility allows VPNs to be configured for various use cases. The downside is that complexity invites mistakes: the more options a protocol exposes, the more ways a less-experienced developer can misconfigure it and leave a security hole.

I’ve seen plenty of real-world examples of that – our investigation into security flaws in free Android VPNs is full of them.

^{Back to VPN connection phases overview ↩}

Once the VPN handshake is complete, a secure tunnel is established between the client and the server.

For a more detailed explanation of VPN tunnels, refer to the VPN tunnels section. Here, I’ll focus on the how the tunnel gets built, and how the different protocols go about it.

Here’s how the main VPN protocols handle VPN tunnel creation:

• OpenVPN:
  • Uses tun/tap virtual network interface, configurable for IP or Ethernet (layer 2) modes.
  • Exchanges IP addresses, routes, and other network settings.
  • Activates the VPN tunnel, routing traffic through the virtual interface.
  • Sets up additional features like compression or fragmentation, if configured.
• WireGuard:
  • Employs a minimalist tunnel setup focused on performance, with low overhead compared to other protocols.
  • Creates a simple virtual network interface, typically named wg0.
  • Can send persistent keepalive packets to maintain the connection.
• L2TP/IPSec:
  • IPSec layer creates the tunnel for L2TP packets.
  • Establishes L2TP control connection within the IPSec tunnel.
  • Creates L2TP session for data transfer.
  • Activates VPN tunnel, routing traffic through the L2TP tunnel protected by IPSec.
• IKEv2:
  • Establishes IPSec tunnel.
  • Can route different traffic flows through additional IPSec Child SAs.

^{Back to VPN connection phases overview ↩}

Being assigned a new VPN IP address is a crucial stage in the connection process. For a general overview of IP masking, refer to the section on how a VPN hides your IP address.

Each VPN protocol handles IP assignment differently. Here’s a breakdown of how the main protocols manage it:

Your device’s routing table is updated as part of this process, with one rule to carry traffic between your device and the VPN server over your normal internet connection, and further rules to push all other internet traffic through the tunnel.

If split-tunneling is available and enabled in the client, additional rules are added to keep traffic from the apps you’ve specified outside the tunnel and on your normal connection.

^{Back to VPN connection phases overview ↩}

To keep your internet activity private, a VPN needs to send your DNS queries through the encrypted tunnel along with everything else. Ideally, the provider also resolves those queries on its own DNS servers, rather than handing them to another third party.

When those queries escape the tunnel, that’s known as a DNS leak, handing a log of every site you visit to a third party you were trying to hide it from, compromising your privacy.

Here’s how the main VPN protocols protect DNS resolution:

As you can see, the choice of VPN protocol significantly impacts DNS handling, affecting both security and functionality.

Modern VPN protocols offer robust DNS protection but they’re easy to misconfigure – which is why a leak test is the first check I run on any VPN app I install.

A leak isn’t the only way your queries end up in someone else’s logs though. It’s even more common to find VPN providers, especially free services, who don’t invest in running their own DNS servers. Your queries can travel through the tunnel exactly as they should, and still be logged at the far end by whichever third-party resolver the provider is using.

Over 80% of free Android VPNs leaked DNS requests in some way, according to our research.

^{Back to VPN connection phases overview ↩}

With the secure VPN connection fully established, the client encrypts your data and sends it through the tunnel to the VPN server. The server decrypts it and forwards it to its final destination, i.e. the website or app you’re using.

Responses come back the same way, but in reverse: the VPN server encrypts them and sends them through the tunnel for the client to decrypt.

The specific cipher used for encryption, along with packet structure and handling, is determined by the VPN protocol in use. For a general overview, refer to the section on how VPN encryption works.

The following table compares how popular VPN protocols handle data transmission and encryption/decryption. Hit the button to expand it then scroll down to see how each protocol handles the various aspects of the process.

Periodic key renegotiation – also known as key rotation or rekeying – is an important security feature that helps maintain the confidentiality of the connection over time. The longer a single key stays in use, the more data a single compromise would expose.

Here’s how each protocol handles it. Hit the button above the table to launch a fullscreen view.

A VPN tunnel is the secure channel established between your device and the VPN server: a private route through the public internet.

It’s called a tunnel because your original traffic is encrypted and then wrapped inside an ordinary, unencrypted outer packet addressed to the VPN server. Think of it as placing a letter inside one envelope, then placing that envelope inside a second one addressed somewhere else entirely.

The encryption is what makes your data unreadable; the outer envelope is what carries it without revealing who it’s really for. It also explains why anyone monitoring your connection can see that you’re connected to a VPN, but not what you’re doing inside it.

This process, called encapsulation, is performed by tunneling protocols.

Encryption transforms plaintext data into a secret code, unreadable to anyone without the key to decrypt it. In a VPN, that’s what conceals your browsing activity on the leg between your device and the VPN server, protecting you from various forms of surveillance.

The work is done by a cipher: a mathematical algorithm (i.e. a set of rules) that encrypts and decrypts data, often paired with a specific key length.

Very few apps let you choose your own cipher – Private Internet Access is one of the exceptions:

The key itself is a string of characters used to scramble the data. Only someone holding the same key can unscramble it.

Longer key lengths generally offer more security. For example, AES-256 is considered stronger than AES-128.

Some VPN protocols use static keys, designed for long-term use, while others use ephemeral keys that expire after a single session or transaction.

Here’s how encryption ciphers work:

1. The cipher uses a key to encrypt the data before sending it.
2. The same or a related key is used to decrypt the data at the receiving end.
3. Only those with the correct key can decrypt and read the information.

Which cipher a VPN uses often comes down to how its developers prioritized security over performance, since stronger ciphers demand more processing power. Newer ciphers also emerge over time, and the better services adopt them. How quickly a provider does so tells you a lot about how much attention it pays to its apps.

The most commonly used ciphers in VPN services are:

• Advanced Encryption Standard (AES)
  • Gold standard for online encryption
  • Available in 128-bit, 192-bit, and 256-bit key-lengths
  • AES-256 preferred for stronger protection, and considered resistant to known quantum attacks
• Blowfish
  • Designed in 1993, largely replaced by AES-256
  • 128-bit key length (32 to 448 bits possible)
  • Has known weaknesses
  • Still turns up as a fallback in older OpenVPN configurations; I treat its presence as a red flag
• ChaCha20
  • Newer cipher, now very widely adopted
  • 256-bit key length
  • Comparable security to AES-256
  • Noticeably faster performance, especially on mobile devices
• Camellia
  • Developed in 2000
  • Security and speed comparable to AES
  • Not certified by NIST
  • Rarely offered by VPN services

Encryption modes determine how a cipher like AES is actually applied to your data. They matter more than most people realize, affecting both privacy and performance. Here are the most common modes used in VPNs:

• CBC (Cipher Block Chaining)
  • Traditional mode
  • Creates chain of dependent blocks
  • Can be slower than newer modes
  • Requires additional security measures
• GCM (Galois/Counter Mode)
  • Provides encryption and authentication
  • Faster than CBC
  • Considered very secure
  • Widely used in modern VPNs
• CTR (Counter)
  • Turns block cipher into stream cipher
  • Fast and allows parallel processing
  • Doesn’t provide authentication alone
  • ChaCha20 uses the same counter-based construction
• OCB (Offset Codebook Mode)
  • Provides authenticated encryption in single pass
  • Very efficient
  • Limited adoption due to patent concerns

Other modes are best avoided in VPNs. ECB (Electronic Codebook) in particular is too simple to be secure, while CFB (Cipher Feedback) and OFB are both outdated and easy to misconfigure. Any mode without integrity checks should also be avoided, unless it’s paired with a separate authentication mechanism.

Most top-tier VPNs now pair AES with GCM, or the ChaCha20-Poly1305 combination, as both strike a good balance between security and performance. These are the combinations I look for whenever I review a VPN.

VPN protocols are the sets of rules and processes used to establish secure connections between your device and the VPN server.

They determine how the VPN tunnel is formed and managed, while encryption ciphers secure the data traveling inside it.

Each protocol offers a unique balance of speed, security and compatibility. Most VPN services will default to a particular protocol, but let you switch in the app settings.

VPN handshakes are crucial for initiating secure connections. This involves:

1. Authentication: Verifying the identity of both the VPN client and server.
2. Key Exchange: Deriving a shared session key at each end, without ever sending it across the connection.
3. Parameter Negotiation: Agreeing on communication rules and cipher suites.

During a VPN handshake, your device and the VPN server exchange information to establish a secure channel, and both ends come away holding the same secret key for encrypting and decrypting data throughout the VPN session.

TLS-based handshakes – the kind used by OpenVPN, SSTP and certificate-based IKEv2 – typically rely on the RSA (Rivest-Shamir-Adleman) algorithm, which has underpinned internet security for decades. However, key length is critical.

For adequate security, use VPN services implementing RSA-2048 or RSA-4096. RSA-1024 is no longer considered secure due to advances in computing power.

Broadly secure as the process is, the handshake generates a ‘master key’ that could decrypt every session on that server if compromised. In such a scenario, an attacker could hack into the VPN server and gain access to all data flowing through the VPN tunnel.

To mitigate this risk, I recommend using VPN services with Perfect Forward Secrecy.

Perfect Forward Secrecy (PFS) is a critical security feature in modern VPN protocols. It uses Diffie-Hellman (DH) or Elliptic Curve Diffie-Hellman (ECDH) algorithms to generate a unique, temporary session key for every connection.

With Perfect Forward Secrecy, encryption keys are never sent across the connection at all.

Key benefits of PFS include:

1. Independent key generation: Both the VPN server and client independently derive the same encryption key without exchanging it.
2. Session isolation: Each connection uses a unique key, limiting the potential damage from a single compromised session.
3. Temporal security: Keys are temporary and discarded after use, preventing future decryption of any captured data.

RSA remains crucial for authentication, but it can’t deliver PFS on its own: that requires DH or ECDH in the cipher suite. ECDH offers robust security for handshakes, while plain DH should only be used together with other measures, given its potential vulnerabilities.

OpenVPN and WireGuard – the two VPN protocols I recommend – both support Perfect Forward Secrecy.

Encryption keeps your data private, and hashing proves it hasn’t been altered in transit.

Secure Hash Algorithms (SHA) play a vital role in VPN security by:

1. Verifying data integrity during transmission
2. Authenticating client-server connections
3. Preventing unauthorized data manipulation

SHAs use a hash function to convert source data into a fixed-length string of characters, known as the “hash value”. This is a one-way process and can’t be reversed; changing even one character in the input drastically alters the output.

This is how it works in VPN communications:

1. The sender applies an agreed-upon hash function to the data.
2. The recipient generates a hash from the received data using the same function.
3. If the generated hash matches the transmitted hash, the data’s integrity is confirmed.
4. Mismatched hashes indicate potential tampering, and the data is discarded.

SHA hash authentication prevents man-in-the-middle attacks: it detects certificate tampering, which stops hackers from impersonating legitimate VPN servers.

For maximum security, I recommend VPN services using SHA-2 or higher. SHA-1 has known weaknesses that can compromise security.