The Essentials

A Beginner’s Guide to VPN Encryption

JP Jones
JP JonesUpdated

Encryption is at the very heart of VPN technology. In this guide we break down how VPN encryption works and how it protects you.

Guide to VPN Encryption

What actually is encryption?

Encryption is the process of encoding data so that only those with authorization can access it, preventing anyone else from doing so.

It’s a process which involves the substitution of letters and numbers, making messages look like gibberish unless you apply the correct process to decode it – that process is known as a ‘cipher’.

Given the processing power of modern computers, any cipher a human could come up with would be way too easy to break.

Powerful mathematical algorithms are now used to create different methods of encryption, plus the secret ciphers required to decrypt them.

VPN services use encryption to create a secure connection (the VPN tunnel) between your device and a VPN server, keeping your internet data private from your ISP, hackers, and any other snooping third parties.

In this guide, you’ll learn all about ciphers and VPN protocols, and find out which ones are the best and safest to use.

We’ll break it all down into plain English and explain all of the jargon in simple terms so that you can get a better understanding of VPN encryption and all that it entails.

Encryption Glossary

Screenshot of encrypted data on a black background

Let’s start by defining some of the important terms we use when talking about encryption:

Term Definition
Algorithm A set of instructions that need to be followed (by software) in order to solve a problem. A cipher is an algorithm for performing encryption or decryption.
Authentication A process to ensure and confirm a user’s identity.
Bit Short for ‘binary digit’, it’s the smallest unit of data in a computer.
Block A group of bits of fixed length (e.g. 64 bits)
Block Size The maximum length of a group of bits.
Brute Force Attack A method to try and guess a secret by trying every possible combination of characters. It’s a crude, exhaustive effort rather than a more complicated or strategic one. Imagine you can’t open the padlock on your locker – rather than trying to remember the code, by brute forcing it you simply try every combination from 000 through to 999 until it opens up.
Cipher An algorithm for performing encryption and decryption.
Cryptography The art of writing and solving codes to create secure messaging, designed so that only the intended parties can process and read it.
Encryption The process of converting data into a code in order to prevent unauthorized access to it.
Firewall Software which monitors and controls packets of data incoming or outgoing from a network. You’ll likely be familiar with the one on your desktop or laptop computer, which is designed to watch out for hackers and viruses.
Handshake A process that initializes the connection between two computers – like its name suggests, it’s a greeting that establishes the rules for communication.
HTTPS It stands for ‘Hypertext Transfer Protocol Secure’, but no one ever calls it that. It’s a secure version of HTTP, the protocol that is the foundation of the web, and is used to ensure communication between devices and websites stays authenticated, private, and secure.
Kernel The core of an operating system. In a computer it controls the operation of the computer and its hardware.
Key A string of bits used by a cipher to convert ‘plain text’ (unencrypted information) into ciphertext (encrypted information), and vice versa. A key can vary in length – generally, the longer it is, the longer it would take to crack it.
Packet A packet is a unit of data routed between an origin and destination on a network, and made up of a header, the payload (your data), and the trailer, and normally of a specific set size, which matches the network MTU (Maximum Transmission Unit) size. When data needs to be sent over a network, if it’s larger than a packet it may need to broken down into multiple data packets, which are then sent individually and the data reassembled at the destination.
Port Just as a boat docks at a physical port, computing ports represent the ‘endpoint’ of communication. Any information reaching your device does so via a port.
Protocol A set of rules used to negotiate a connection between a VPN client and a VPN server. Some are more complex or secure than others – OpenVPN and IKEv2 are some popular choices.

Encryption Glossary

Now we know what the terms mean, it’s time to explain what encryption is and does in more depth.

Types of Encryption

Illustration of two keys, one on top of the other on a pale blue background

There are two types of encryption: symmetric and asymmetric.

Symmetric-key Encryption

Symmetric-key encryption is where only one key is used to handle both encryption and decryption of data.

Both parties require the same key to communicate. This is the kind of encryption used in VPN technology.

The power of modern computers means that the keys have had to become increasingly long to prevent brute-force attacks (trying every combination to find the right key).

The current gold standard is the 256-bit key, which cannot be brute-forced as it would take billions of years to run through all of the possible combinations using computers available today.

Asymmetric Encryption

With asymmetric cryptography (or public-key cryptography), each participant who wants to communicate securely uses software to generate a public key and corresponding private key.

As an example, let’s take two people: Person A and Person B

When Person A wants to send a secure message to Person B, Person B’s public key is used in a cipher to convert the plain text message into an encrypted message.

While the encrypted message might travel from A to B through various other parties (Persons C, D, E, and F), anyone who tries to read the message will only see the encrypted text.

When Person B receives the encrypted message, they use their private key to decode the encrypted message back into plain text.

This system is increasingly used by journalists, for example, who publish their public key on their social media profiles for sources to send them messages that can only be decrypted with the journalist’s private key.

The most well-known such system is Pretty Good Privacy (PGP). There are lots of different software tools that use OpenPGP, the open source version of the standard.

Public-key encryption is used during a TLS handshake in order to securely share a symmetric key between client and server.

SSL & TLS

TLS/SSL is the encryption that most of us have experienced when browsing the web on a secure website that uses HTTPS.

You’ll know when a website is using HTTPS by the padlock symbol in the browser’s address bar.

The security protocol in use here is TLS (Transport Layer Security), which is based on its predecessor Secure Sockets Layer (SSL Version 3.0).

TLS uses a combination of public-key and symmetric encryption to safeguard your data.

During the TLS handshake, your browser uses asymmetric encryption to communicate with the secure page’s server and securely generate a symmetric key.

This is then used to encrypt the data being transferred between your browser and the server.

Generating a symmetric key for use is much more efficient than using asymmetric keys for all data transfer.

This is because asymmetric encryption requires an incredible amount more of computing power to encrypt and decrypt all the necessary data when compared to using a symmetric key.

It can therefore be said that symmetric key encryption is the faster encryption method.

While the above is good and generates secure encryption, every secure session generated by the server is decryptable with the server’s private key.

If that private key were ever to be compromised, the stolen private key can then be used to decrypt any secure session on that server, past or present.

To avoid that, HTTPS and OpenVPN connections are now commonly set up using Perfect Forward Secrecy, utilizing an algorithm called Diffie-Hellman Key Exchange which generates the symmetric key.

It may sound confusing, but all you really need to understand is that this is the more secure way to go about things as the symmetric key is never exchanged over the connection, instead generated independently by both the server and web browser.

The browser generates a temporary private key, and a corresponding public key.

The symmetric key for the TLS session is based on the output of an algorithm that operates on a private key of the device (e.g. a server), and the public key of the other device (e.g. your browser).

Due to the fancy mathematical properties of this algorithm and some technical magic the key generated by this process will match on both the server and browser.

That means if a third party such as your ISP or the government is storing your encrypted data from previous sessions, and the private key is otherwise compromised or stolen, they won’t be able to decrypt that data.

ExpressVPN is one VPN service that uses Perfect Forward Secrecy, negotiating a new private key each time you connect, and every 60 minutes while a VPN connection is being used.

ExpressVPN's illustrating demonstrating how the app uses public keys

This ExpressVPN illustration shows how the VPN app uses the server’s public key to produce a symmetric key pair, using asymmetric encryption.

Now that we’ve explained the different methods of encryption available, let’s talk about the different VPN protocols that are available.

VPN Protocols

What is a VPN protocol?

VPN protocols represent the processes and sets of instructions (or rules) VPN clients rely on to establish secure connections between a device and a VPN server in order to transmit data.

VPN protocols are formed from a combination of transmission protocols and encryption standards.

Which VPN protocols are currently available?

Here are the main VPN tunnelling protocols you need to know about:

OpenVPN – Very Secure and Fast

 

OpenVPN Logo

OpenVPN is the industry gold standard VPN protocol and we recommend you use it whenever you can.

It’s one of the most secure and safe VPN protocols, and importantly is open-source, which means it’s completely transparent and continues to be publicly tested and improved.

OpenVPN is very configurable and, while it’s not supported natively by any platform, most VPN providers offer free apps that support it.

These custom VPN apps are available across most major platforms like Microsoft Windows, Apple MacOS, Android, Linux, and iOS.

Some providers also offer OpenVPN configuration files, meaning you can download the original OpenVPN client for your platform from https://openvpn.net/ and use it to connect to your chosen VPN service.

OpenVPN works on both UDP and TCP, which are types of communication protocols.

TCP (Transmission Control Protocol) is the most used connection protocol on the internet. The data that is being sent is transferred in chunks, typically made up of several packets.

TCP is designed to deliver the transferred data to the OpenVPN client in the order it was sent from the OpenVPN server (e.g. packets 1, 2, 3, 4, and 5 sent from OpenVPN are received by the OpenVPN client in the same order – 1, 2, 3, 4, 5).

To do this, TCP can delay the delivery of the packets it has received over the network to the OpenVPN Client until it has received all the packets expected and rearranged any out-of-order packets back into place.

TCP will re-request (and then wait to receive) packets that may have been lost in transmission between server and client, too.

This processing and wait time adds latency to the VPN connection, making the connection slower than UDP.

UDP (User Datagram Protocol) simply transmits data packets without requiring confirmation of arrival, and the UDP packet sizes are smaller than TCP.

By using OpenVPN UDP, the smaller packet size, lack of checks, and re-organization results in a faster connection.

So, which is better: TCP or UDP?

It depends on your desired outcome.

If you’re using a VPN to game, stream, or use VoIP services, then UDP is your best bet, as it’s quicker than TCP.

The downside is that you may experience some lost packets, which for example might mean on a VOIP call you hear the voice of the person you’re talking to cut out for a fraction of a second mid-speech.

However, you should switch to TCP if you experience connection problems. TCP port 443 is also useful for bypassing censorship, as this port is the default port for HTTPS, and so is less likely to be blocked by firewalls.

For encryption, OpenVPN uses the OpenSSL library, which supports a range of ciphers.

OpenVPN encryption is made up of several elements: data channel, control channel, server authentication, and HMAC authentication:

  • Server authentication functions in the same was as TLS or HTTPS. OpenVPN can use certificates to verify that the server you are talking to is cryptographically trusted.
  • The control channel is used in the initial phase, performing the TLS handshake to agree on the encryption parameters to securely transmit data, and authenticating the client to the server.
  • The data channel is the layer that transmits information between your device and the OpenVPN Server. This layer is encrypted using a symmetric encryption scheme for performance, the key for which was obtained via the control channel.
  • HMAC authentication is used to ensure that packets being sent have not been altered in transit by a man-in-the-middle attacker who has the ability to read or alter the data in real time.

Be aware that some VPN services don’t use anywhere near the same level of encryption on both channels.

Using weaker encryption on the data channel can be a cheap shortcut to a faster connection as better security comes at the expense of speed.

Unfortunately, a VPN is only as secure as its weakest element, so you should look for a VPN that’s as strong as possible in its encryption of both channels.

We’ll go into more detail about that in the sections below on ciphers and handshakes.

Now that you know what the safest VPN protocol is, you should know what the others are – plus which one to avoid at all costs.

PPTP – Weak Security, Avoid

The Point-to-Point Tunneling Protocol (PPTP) one of the oldest VPN protocols still in use today. It was developed by a Microsoft-funded team and published in 1999.

Despite being obsolete, PPTP does have some positives: it’s compatible with pretty much everything, it doesn’t need additional software as it’s included in modern operating systems, and it’s very fast.

The major problem is that it’s proven to be insecure and easy to crack (an attack would typically take between one minute and 24 hours).

Illustration of hackers stealing credit card and identity, with a unlocked padlock on the computer screen

PPTP is also simple to block as it relies on the GRE protocol, which is easily firewalled.

You should avoid using this protocol unless absolutely necessary to change your IP address for non-sensitive reasons. We consider PPTP to be insecure.

L2TP/IPsec – Secure, But Can Be Slow

Layer 2 Tunneling Protocol (LT2P) takes the best features of both Microsoft’s Point-to-Point Tunneling Protocol (PPTP) and Cisco’s Layer 2 Forwarding Protocol (L2F) and is used to create a tunnel between a client device and a server over the network.

L2TP can handle authentication, but does not provide any encryption capabilities.

Therefore, L2TP is typically implemented with the Internet Protocol Security (IPsec) to create secure packets that provide authentication, integrity, and encryption of data.

This is more commonly known as L2TP/IPsec, and data is usually encrypted using the AES cipher, which you can read more about here.

When connecting to a VPN server with L2TP/IPsec, IPsec is used to create a secure control channel between client and server.

Packets of data from your device application (like your web browser, for example) are encapsulated by L2TP. IPSec then encrypts this L2TP data and sends it to the server, which then performs the reverse process, decrypting and decapsulating the data.

In terms of speed, L2TP/IPsec’s double encapsulation (essentially a tunnel within a tunnel) should make it slower than OpenVPN.

However, it is actually theoretically faster because the encryption and decryption take place in the kernel, which can process packets efficiently with minimum overhead.

L2TP/IPsec is generally considered safe when used with the AES cipher.

But there have been suggestions that the protocol has been compromised by the NSA, and that IPsec was deliberately weakened during its creation.

There hasn’t been any official confirmation of this, though.

The main problem with L2TP/IPsec and its use in VPN services lies with those services that use pre-shared keys (also known as a shared secret) that can be downloaded from the VPN service websites and are therefore available to anyone.

While these keys are only used to authenticate the connection with the VPN servers, and the data itself remains encrypted via a separate key, it does open the door to potential MITM (Man in-the-middle) attacks.

This is where the attacker impersonates a VPN server in order to decrypt the traffic and eavesdrop on the connection.

L2TP/IPsec also uses a limited number of fixed ports, which makes it relatively easy to block.

Despite these issues, LT2P/IPsec is a solid choice given that it’s supported natively by so many platforms as long as pre-shared keys are not used.

SSTP – Closed Source with Potential Risks

Secure Socket Tunneling Protocol (SSTP) is a Microsoft-owned proprietary protocol that’s based on SSL 3.0, meaning that, like OpenVPN, it can use TCP port 443.

As SSTP is not open-source, it is impossible to disprove suggestions of backdoors or other vulnerabilities present in the protocol.

This risk far outweighs any benefits from its close integration with Windows.

Another red flag is that SSL 3.0 is vulnerable to a man-in-the-middle attack known as POODLE.

It has not been confirmed whether SSTP is affected, but in our view it’s not worth the risk.

IKEv2/IPSec – Very Fast, Secure, and Stable

Internet Key Exchange version 2 (IKEv2) is a newer VPN protocol, and another closed-source standard developed in collaboration between Microsoft and Cisco.

IKEv2 is natively supported by iOS, BlackBerry, and Windows versions 7 and later.

However, there are open-source versions of IKEv2 developed for Linux which don’t carry the same trust issues as the proprietary version.

Similarly to L2TP/IPsec, IKEv2 is used alongside IPsec when a part of a VPN solution, but offers more functionality.

IKEv2/IPSec can handle changing of networks through something called the MOBIKE protocol – useful for mobile users who are prone to their connection dropping, and is faster due to being programmed to make better use of bandwidth.

IKEv2/IPSec also supports a wider range of encryption ciphers than L2TP/IPSec.

IKEv2 often won’t cut it when you’re trying to connect out of a highly censored country, though. This is because IKEv2 uses specified ports that are very easy for the Great Firewall to block.

WireGuard – Promising New Protocol

 

WireGuard Logo

Wireguard is a new tunneling protocol that aims to be faster and higher-performing than the current most popular protocol, OpenVPN.

WireGuard aims to tackle the issues often associated with OpenVPN and IPsec: namely complicated setup, plus disconnections (without additional configuration) and the associated long reconnection times that follow.

While OpenVPN+OpenSSL and IPsec have a large codebase (~100,000 lines of code for OpenVPN and 500,000 for SSL) and IPsec (400,000 lines of code), which makes it hard to find bugs, Wireguard currently weighs in at less than 5,000 lines in size.

But Wireguard is still under development.

While Wireguard benchmarks show that it is very fast, there are issues in the implementation that may make it unsuitable for use by a commercial VPN provider.

One of which is that it requires a non-public IP address to be assigned to each user, which adds an element of logging any serious VPN user would be uncomfortable with.

Why?

Because this non-public IP address can be used to identify you.

Work is underway to address this, though.

It’s still early days for WireGuard, and it has yet to fully prove itself – however, an increasing number of VPN providers are adding it to their clients for test purposes only, including IVPN and AzireVPN.

Promotion on IVPN's website talking about its WireGuard implementation

VPN protocols provide the framework for secure encryption, now let’s find out what role ciphers play and the varieties of ciphers available.

Ciphers

A cipher is essentially an algorithm for encrypting and decrypting data. VPN protocols use a variety of ciphers, and the following are the most commonly used:

AES

The Advanced Encryption Standard (AES) is a symmetric-key cipher established by the U.S. National Institute of Standards and Technology (NIST) in 2001.

It is the gold standard for online encryption protocols, and is used heavily by the VPN industry. AES is considered one of the safest ciphers to use.

AES has a block size of 128 bits, which means AES can handle larger file sizes than other ciphers, such as Blowfish which has a 64-bit block size.

AES can be used with varying key lengths. While AES-128 is still considered secure, AES-256 is preferred as it gives greater protection. AES-192 is also available.

Illustration of a bank safe on a blue background

When you read about ‘military-grade’ or ‘bank-grade’ encryption on a VPN service’s website, it generally refers to AES-256, which is used by the US government for Top Secret data.

Blowfish

Blowfish is another symmetric-key cipher, designed in 1993 by American cryptographer Bruce Schneier.

Blowfish used to be the default cipher used in OpenVPN, but has largely been replaced by AES-256.

When Blowfish is used, you’ll typically see it used with a 128-bit key length, although it can range from 32 bits to 448 bits.

Blowfish has some weaknesses, including its vulnerability to ‘birthday attacks’, which means that it really should only be used as a fallback to AES-256.

Camellia

Camellia is also a symmetric-key cipher, and it’s very similar to AES in terms of security and speed.

The main difference is that Camellia is not certified by NIST, the US organization that created AES.

While there is an argument for using a cipher not associated with the US government, it’s rarely available in VPN software, nor has it been as thoroughly tested as AES.

VPN Handshake

Illustration of two hands shaking

Just as with initiating a secure session on a HTTPS website, securely connecting to a VPN server requires the use of public-key encryption (typically using the RSA cryptosystem) via a TLS handshake.

RSA has been the basis for internet security for the last two decades but unfortunately, it now seems likely that the weaker version, RSA-1024, has been cracked by the NSA.

While most VPN services have moved away from RSA-1024, a minority continue to use it and should therefore be avoided. Look for RSA-2048, which is still considered secure.

Ideally additional encryption will be used to create Perfect Forward Secrecy. Typically this will be via the inclusion of Diffie-Hellman (DH) or Elliptic curve Diffie-Hellman (ECDH) key exchange.

While ECDH can be used on its own to create a secure handshake, DH alone should be avoided as it is vulnerable to being cracked. This is not an issue when used with RSA.

SHA Hash Authentication

Secure Hash Algorithms (SHA) are used to guarantee the integrity of transmitted data and SSL/TLS connections, such as OpenVPN connections, to ensure that information has not been changed in transit between source and destination.

Secure Hash Algorithms work by transforming source data using what is known as a hash function, whereby the original source message is run through an algorithm and the result is a fixed-length string of characters that looks nothing like the original – the “hash value”.

It is a one way function – you cannot run a de-hash process to determine the original message from the hash value.

Hashing is useful because changing just one character of the input source data will totally change the hash value that is output from the hash function.

A VPN client will run the data received from the server, combined with the secret key, through the hash function agreed during the VPN handshake.

If the hash value the client generates differs from the hash value in the message, the data will be discarded as the message has been tampered with.

SHA hash authentication prevents man-in-the-middle attacks by being able to detect any tampering with a valid TLS certificate.

Without it a hacker could impersonate the legitimate server and trick you into connecting to an unsafe one, where your activity could be monitored.

It is recommended that SHA-2 (or higher) should be used, to improve security, as SHA-1 has proven weaknesses that can compromise security.

What is a VPN? Explanation Diagram

We’ve covered a lot of ground in this guide to encryption, but you may still have some more general questions related to VPNs and protocols that need answering.

Here are the most common questions we hear:

What’s the Safest VPN Protocol?

OpenVPN used with AES-256 cipher is generally considered the best and safest VPN protocol. OpenVPN is open-source and has been publically tested for weaknesses.

OpenVPN offers a great balance between privacy and performance, and is compatible with many popular platforms. Many commercial VPN services use OpenVPN as default.

IKEv2 is a good option for mobile devices as it handles network changes more effectively, automatically restoring dropped connections with ease and speed.

How Do I Change VPN Protocol?

Some VPN services, like ExpressVPN, allow you to change the VPN protocol within the VPN app’s settings menu.

If this is the case, simply open up the settings menu, and select the VPN protocol you wish to use.

Screenshot of ExpressVPN protocol settings menu

If there is no VPN protocol selection within the custom app, you may be able to install alternative protocols using manual configuration.

NordVPN is one example of a VPN service that runs on OpenVPN but allows for manual installation of IKEv2.

If your VPN service supports alternative protocol configuration be sure to carefully follow the instructions given on its website.

Do All VPNs Encrypt Data?

VPNs by their very nature encrypt data, but there are some services out there that claim to be VPNs while providing no encryption. Hola Free VPN is one of these, and you should avoid products like it at all costs.

VPN browser extensions are another product to be a little wary of. Browser extensions are proxy services rather than VPN services, and while some do provide encryption others do not.

The browser extensions that do provide encryption will only protect your browser traffic in the web browser it is running in, so you’ll need to use a full VPN to encrypt all your other apps.

Are All VPNs Safe to Use?

Yes – if you make the right choice.

Not all VPNs are safe to use. Even those that use the best VPN protocols and ciphers may put your personal data at risk by logging your online activities.

You should look for a VPN with a privacy-friendly logging policy, as well as one that supports the safest VPN protocols and ciphers.

To make things easier for you, why not take a look at our highest recommended VPNs – they are the safest ones around.

About the Author


  • JP Jones - CTO @ Top10VPN

    JP Jones

    JP is our CTO and with over 25 years of software engineering and networking experience oversees all technical aspects of our VPN testing process. Read full bio