Guide to VPN Encryption & VPN Protocols

Simon Migliano
Simon MiglianoUpdated

Encryption is at the very heart of VPN technology. It’s what protects your privacy from intrusive surveillance and prevents hackers from intercepting your personal data on public networks. Read this guide if you would like to know more about how encryption works and how it protects you.

Encryption is the process of encoding data so that only authorized parties can access it and preventing anyone else from doing so. There is a long history of using cryptography to communicate in secret that long predates computer encryption.

At its core, encryption still works in the same way as the earliest ciphers, specific codes involving the substitution of letters and numbers, whereby messages looked like gibberish unless you had the correct cipher allowing you to decode their contents.

Given the processing power of modern computers, human-based ciphers are too easy to break. We now use powerful algorithms to create different methods of encryption (ciphers), also generating the key required to decrypt the data.

When we talk about encryption, we use a combination of cipher and key-length. For example, AES-256 is the AES cipher with a 256-bit key length. Other ciphers include Blowfish and much less commonly in VPN software, Camellia.

Types of Encryption

There are two types of encryption:

Symmetric-key encryption

This is where encryption and decryption keys are the same. Both parties must have the same key to communicate. This kind of encryption is used in VPN technology.

The power of modern computers means that the keys have had to become increasingly long to prevent brute force attacks (trying every combination to find the right key). The current gold standard are 256-bit keys which cannot be brute-forced as it would take billions of years to run through the possible combinations.

Public-key encryption

Software is used to generate sets of public keys and private keys. The public key is used to encrypt data sent to the holder of a private key, which allows them to decode the messages. This system is increasingly used by journalists for example who publish their public key on their social media for sources to send them messages that only they can decrypt with their private key.

The most well-known such system is Pretty Good Privacy (PGP). There are tons of different software tools that use OpenPGP, the open source version of the standard.

SSL and TLS

The encryption that most of have some experience with is when browsing the web on a secure site that implements HTTPS (when the padlock is displayed in the browser’s address bar).

The security protocol in use here is TLS (Transport Layer Security), which incorporates the older Secure Sockets Layer (SSL), which use a combination of public-key and symmetric encryption to safeguard your data.

Your browser uses public-key encryption to communicate with the secure page’s server and securely share symmetric keys that are in turn used to encrypt the data being transferred. This is much more efficient as public-key encryption requires too much computing power to encrypt all the necessary data in a session.

The only problem arises when a server only uses a single private key to generate the secure session. If that key is ever compromised, the hacker can decrypt any secure session on that server.

Fortunately, it’s becoming increasingly the case that unique private keys are generated and then discarded after each session. This is known as Perfect Forward Secrecy (PFS).

VPN Encryption

OpenVPN

OpenVPN is the industry gold standard VPN protocol and we recommend you use it whenever you can. It’s the most secure and importantly is open-source, which means it’s completely transparent and continues to be been publicly tested and improved.

It’s very configurable and while it’s not supported natively by any platform, every VPN provider offers free apps that support it across most major platforms.

Importantly, OpenVPN works on on both the UDP port and TCP 443 port, which is used by HTTPS traffic. This makes it tough to block as it’s hard to distinguish it from banking and ecommerce traffic using that port.

It also uses the OpenSSL library, which supports a range of ciphers.

OpenVPN encryption has two elements: data channel encryption and control channel encryption. Data channel encryption is what protects the information itself that’s been transmitted. Control channel encryption is what secures the connection between your device and the VPN server itself.

It’s important to be aware that some VPN services don’t use anywhere near the same level of encryption on both channels. Using weaker encryption on the data channel is a cheap shortcut to a faster connection as security comes at the expense of speed.

Unfortunately, a VPN is only as secure as its weakest element, so you should look for a VPN that’s as strong as possible in its encryption of both channels.

PPTP

The Point-to-Point Tunneling Protocol (PPTP) is an outdated VPN protocol developed by a Microsoft-funded team for creating VPN over dial-up and bundled with Windows in 1999. It does have some positives: it’s compatible with pretty much everything, doesn’t need additional software and it’s very fast.

The major problem is that it’s proven to be insecure and is easy to crack. It’s also simple to block as it relies on the GRE protocol that’s easily firewalled.

Avoid using this protocol unless absolutely necessary to change your IP address for non-sensitive reasons.

L2TP/IPsec

Layer 2 Tunneling Protocol (LT2P) is a Microsoft-proprietary protocol that is typically implemented with the IPsec authentication as it doesn’t provide any encryption by itself. It’s usually used with the AES cipher.

The protocol uses a limited number of fixed ports, which makes it relatively easy to block.

In terms of speed, while its double encapsulation should make it slower than OpenVPN, it is actually theoretically faster because the encryption and decryption take place in the kernel and its supports multi-threading, which OpenVPN does not.

It has no known vulnerabilities when used with AES but there have been suggestions that it has been compromised by the NSA and was deliberately weakened during its creation.

The main problem though lies with VPN services that use pre-shared keys that can be downloaded from their websites. While these keys are only used to authenticate the connection with the VPN servers (the data itself remaining encrypted), it does open the door to potential hacks.
Despite these issues, LT2P/IPsec is a solid choice given that it’s supported natively by so many platforms as long as pre-shared keys are not used.

SSTP

Secure Socket Tunneling Protocol (SSTP) is a Microsoft-owned proprietary protocol that’s based on SSL 3.0, meaning that like OpenVPN it can use TCP port 443.

Unlike OpenVPN though, SSTP is not transparent, which means it’s not possible to disprove suggestions of backdoors or other vulnerabilities. This risk far outweighs any benefits from its close integration with Windows.

Another red flag is that SSL 3.0 is vulnerable to a man-in-the-middle attack known as POODLE. It’s not confirmed whether SSTP is affected but it’s not worth the risk in our view, despite sharing benefits with OpenVPN.

IKEv2

Internet Key Exchange version 2 (IKEv2) is another closed standard, this time developed by Microsoft and Cisco. It’s natively supported by iOS, Windows 7 and later, and Blackberry.

There are open source versions developed for Linux, which don’t carry same the trust issues of the proprietary version.

On the plus side, it’s copes well with changing networks and is very good at reconnecting when users drop their internet connections making it particularly useful for mobile users.

It’s a shame that it’s closed source as IKEv2 is a very fast and secure protocol that’s particularly good for mobile users.

Ciphers

AES

The Advanced Encryption Standard (AES) is a symmetric-key cipher established by the U.S. National Institute of Standards and Technology (NIST) in 2001. It’s become the gold standard in the VPN industry.

Its 128-bit block size means it can handle larger file sizes than other ciphers, such as Blowfish for example.

While AES-128 is still considered secure, AES-256 is generally preferred as it gives greater protection.

Blowfish

This is the default cipher used in OpenVPN (it can be configured to use other ciphers) and is typically Blowfish-128, although it theoretically ranges from 32 bits to 448 bits.

While generally secure, it does have some weaknesses that mean it really should only be used as back up to AES-256.

Camellia

Camellia is very similar to AES, the main difference being that it is not certified by NIST, the US organization that created AES.

While there is an argument for using a cipher not associated with the US government, it’s rarely available in VPN software. Nor has it been as thoroughly tested as AES.

Handshake Encryption

Just as with initiating a secure session on a HTTPS website, securely connecting to a VPN server requires the use of public-key encryption (typically the RSA cryptosystem) via a TLS handshake.

RSA has been the basis for internet security for the last two decades but unfortunately, it now seems likely that RSA-1204 has been cracked by the NSA.

While most VPN services have moved away from RSA-1024, a minority continue to use it and should therefore be avoided. Look for RSA-2048, which is still considered secure.

Ideally, additional encryption will be used to create Perfect Forward Secrecy. Typically this will be via the inclusion of Diffie-Hellman (DH) or Elliptic curve Diffie-Hellman (ECDH) key exchange.

While ECDH can be used on its own to create a secure handshake, DH alone should be avoided as it is vulnerable to being cracked. This is not an issue when used with RSA.

SHA Hash Authentication

Secure Hash Algorithm (SHA) is used to authenticate data and SSL/TLS connections, such as OpenVPN connections.

It prevents man-in-the-middle attacks by being able to detect any tampering with a valid TLS certificate. Without it a hacker could divert your connection to their own server instead of that of your VPN provider.

While SHA-1 is no longer secure for protecting websites, it is still considered secure when used by OpenVPN. There are more secure versions available, such as SHA-2 and SHA-3.