VPN Encryption & VPN Protocols

Callum Tennent
By Callum TennentUpdated

Encryption is at the very heart of VPN technology. It’s what protects your privacy from intrusive surveillance and prevents hackers from intercepting your personal data on public networks. In this guide we will break down how encryption works and how it protects you.

Guide to VPN Encryption

Encryption is the process of encoding data so that only authorized parties can access it and preventing anyone else from doing so. There is a long history of using cryptography to communicate in secret that long predates computer encryption.

At its core, encryption still works in the same way as the earliest ciphers, specific codes involving the substitution of letters and numbers, whereby messages looked like gibberish unless you had the correct cipher allowing you to decode their contents.

Given the processing power of modern computers, human-based ciphers are too easy to break. We now use powerful algorithms to create different methods of encryption (ciphers), also generating the key required to decrypt the data.

When we talk about encryption, we use a combination of cipher and key-length. For example, AES-256 is the AES cipher with a 256-bit key length. Other ciphers include Blowfish and, much less commonly used in VPN software, Camellia.

Types of Encryption

Illustration of a padlock surrounded by clouds and various security-related symbols
There are two types of encryption:

Symmetric-key encryption

This is where encryption and decryption keys are the same. Both parties must have the same key to communicate. This kind of encryption is used in VPN technology.

The power of modern computers means that the keys have had to become increasingly long to prevent brute force attacks (trying every combination to find the right key).

The current gold standard are 256-bit keys which cannot be brute-forced as it would take billions of years to run through the possible combinations.

Public-key encryption

Software is used to generate sets of public keys and private keys. The public key is used to encrypt data sent to the holder of a private key, which allows them to decode the messages. This system is increasingly used by journalists for example who publish their public key on their social media for sources to send them messages that only they can decrypt with their private key.

The most well-known such system is Pretty Good Privacy (PGP). There are tons of different software tools that use OpenPGP, the open source version of the standard.

SSL and TLS

The encryption that most of us have some experience with is when browsing the web on a secure site that implements HTTPS (when the padlock is displayed in the browser’s address bar).

The security protocol in use here is TLS (Transport Layer Security), which incorporates the older Secure Sockets Layer (SSL), which use a combination of public-key and symmetric encryption to safeguard your data.

Your browser uses public-key encryption to communicate with the secure page’s server and securely share symmetric keys that are in turn used to encrypt the data being transferred.

This is much more efficient as public-key encryption requires too much computing power to encrypt all the necessary data in a session.

The only problem arises when a server only uses a single private key to generate the secure session. If that key is ever compromised, the hacker can decrypt any secure session on that server.

Fortunately, it’s becoming increasingly the case that unique private keys are generated and then discarded after each session. This is known as Perfect Forward Secrecy (PFS).

VPN Protocols

Illustration of miniature workmen surrounding a computer screen displaying a padlock inside a shield
VPN protocols represent the processes and sets of instructions VPN clients rely on to establish secure connections between a device and a VPN server in order to transmit data.

In other words, a VPN protocol is a mix of transmission protocols and encryption standards.

Here are the main VPN tunnelling protocols you need to know about:

OpenVPN – Very Secure and Fast

OpenVPN is the industry gold standard VPN protocol and we recommend you use it whenever you can. It’s one of the most secure, and importantly is open-source, which means it’s completely transparent and continues to be publicly tested and improved.

It’s very configurable and while it’s not supported natively by any platform, most VPN providers offers free apps that support it across most major platforms.

Importantly, OpenVPN works on on both the UDP port and TCP 443 port, which is used by HTTPS traffic. This makes it tough to block as it’s hard to distinguish it from banking and e-commerce traffic using that port.

OpenVPN TCP (Transmission Control Protocol) is the most used connection protocol on the internet. It has integrated error correction meaning that when data is transferred, a confirmation of the data packet’s arrival is required before the next packet is sent. The current packet keeps being resent until a confirmation of arrival is received.

OpenVPN UDP (User Datagram Protocol) simply transmits data packets without requiring confirmation of arrival, before transmitting the next data packet. The fewer checks result in lower latency, and make it well suited for streaming and gaming.

OpenVPN also uses the OpenSSL library, which supports a range of ciphers.

Its encryption has two elements: data channel encryption and control channel encryption:

  • Data channel encryption is what protects the information itself that’s been transmitted.
  • Control channel encryption is what secures the connection between your device and the VPN server itself.

Be aware that some VPN services don’t use anywhere near the same level of encryption on both channels. Using weaker encryption on the data channel is a cheap shortcut to a faster connection as security comes at the expense of speed.

Unfortunately, a VPN is only as secure as its weakest element, so you should look for a VPN that’s as strong as possible in its encryption of both channels.

L2TP/IPsec – Secure, But Can Be Slow

Layer 2 Tunneling Protocol (LT2P) is a Microsoft-proprietary protocol that is typically implemented with the IPsec authentication as it doesn’t provide any encryption by itself. It’s usually used with the AES cipher.

The protocol uses a limited number of fixed ports, which makes it relatively easy to block.

In terms of speed, while its double encapsulation should make it slower than OpenVPN, it is actually theoretically faster because the encryption and decryption take place in the kernel and it supports multi-threading, which OpenVPN does not.

It has no known vulnerabilities when used with AES but there have been suggestions that it has been compromised by the NSA and was deliberately weakened during its creation.

The main problem though lies with VPN services that use pre-shared keys that can be downloaded from their websites. While these keys are only used to authenticate the connection with the VPN servers (the data itself remaining encrypted), it does open the door to potential hacks.
Despite these issues, LT2P/IPsec is a solid choice given that it’s supported natively by so many platforms as long as pre-shared keys are not used.

SSTP – Faster and More Secure than L2TP

Secure Socket Tunneling Protocol (SSTP) is a Microsoft-owned proprietary protocol that’s based on SSL 3.0, meaning that like OpenVPN it can use TCP port 443.

Unlike OpenVPN though, SSTP is not transparent, which means it’s not possible to disprove suggestions of backdoors or other vulnerabilities. This risk far outweighs any benefits from its close integration with Windows.

Another red flag is that SSL 3.0 is vulnerable to a man-in-the-middle attack known as POODLE. It’s not confirmed whether SSTP is affected but it’s not worth the risk in our view, despite sharing benefits with OpenVPN.

IKEv2 – Very Fast and Secure

Internet Key Exchange version 2 (IKEv2) is another closed standard, this time developed by Microsoft and Cisco. It’s natively supported by iOS, Windows 7 and later, and Blackberry.

There are open source versions developed for Linux, which don’t carry same the trust issues of the proprietary version.

On the plus side, it’s copes well with changing networks and is very good at reconnecting when users drop their internet connections making it particularly useful for mobile users.

It’s a shame that it’s closed source as IKEv2 is a very fast and secure protocol that’s particularly good for mobile users.

IKEv2 often won’t cut it when you’re trying to connect out of a highly censored country, though.

WireGuard – Promising New Protocol

Wireguard is a new tunnelling protocol that aims to be faster and more performant than the current most popular protocol, OpenVPN.

WireGuard aims to tackle the issues often associated with OpenVPN and IPsec: complex setup, disconnections (without additional configuration) and longer reconnection times, and have a heavy codebase (400,000 to 600,000 lines) which can make it harder to find bugs.

It seeks to improve on the other protocols by using more up-to-date ciphers and it has a code base of around 4,000 lines (~1% of OpenVPN’s and IPsec’s).

While it is still in development, early tests seem to suggest that WireGuard is very fast, very secure and reconnections are instant.

It’s early days and it has to still fully ‘prove itself’ however an increasing number of VPN providers are adding it to their clients, including ExpressVPN and IVPN.

PPTP – Weak Security, Avoid

The Point-to-Point Tunneling Protocol (PPTP) is an outdated VPN protocol developed by a Microsoft-funded team for creating VPN over dial-up and bundled with Windows in 1999.

It does have some positives: it’s compatible with pretty much everything, doesn’t need additional software and is very fast.

The major problem is that it’s proven to be insecure and easy to crack. It’s also simple to block as it relies on the GRE protocol that’s easily firewalled.

Avoid using this protocol unless absolutely necessary to change your IP address for non-sensitive reasons.

Ciphers

Illustration of man inputting a password into a laptop
A cipher is essentially an algorithm for encrypting and decrypting data. VPN protocols use a variety of ciphers, and these are the most commonly used:

AES

The Advanced Encryption Standard (AES) is a symmetric-key cipher established by the U.S. National Institute of Standards and Technology (NIST) in 2001. It’s become the gold standard in the VPN industry.

Its 128-bit block size means it can handle larger file sizes than other ciphers, such as Blowfish for example.

While AES-128 is still considered secure, AES-256 is generally preferred as it gives greater protection.

Blowfish

This is the default cipher used in OpenVPN (it can be configured to use other ciphers) and is typically Blowfish-128, although it theoretically ranges from 32 bits to 448 bits.

While generally secure, it does have some weaknesses that mean it really should only be used as back up to AES-256.

Camellia

Camellia is very similar to AES, the main difference being that it is not certified by NIST, the US organization that created AES.

While there is an argument for using a cipher not associated with the US government, it’s rarely available in VPN software. Nor has it been as thoroughly tested as AES.

Handshake Encryption

Just as with initiating a secure session on a HTTPS website, securely connecting to a VPN server requires the use of public-key encryption (typically the RSA cryptosystem) via a TLS handshake.

RSA has been the basis for internet security for the last two decades but unfortunately, it now seems likely that RSA-1204 has been cracked by the NSA.

While most VPN services have moved away from RSA-1024, a minority continue to use it and should therefore be avoided. Look for RSA-2048, which is still considered secure.

Ideally, additional encryption will be used to create Perfect Forward Secrecy. Typically this will be via the inclusion of Diffie-Hellman (DH) or Elliptic curve Diffie-Hellman (ECDH) key exchange.

While ECDH can be used on its own to create a secure handshake, DH alone should be avoided as it is vulnerable to being cracked. This is not an issue when used with RSA.

SHA Hash Authentication

Secure Hash Algorithm (SHA) is used to authenticate data and SSL/TLS connections, such as OpenVPN connections.

It prevents man-in-the-middle attacks by being able to detect any tampering with a valid TLS certificate. Without it a hacker could divert your connection to their own server instead of that of your VPN provider.

While SHA-1 is no longer secure for protecting websites, it is still considered secure when used by OpenVPN. There are more secure versions available, such as SHA-2 and SHA-3.