How modern VPN protocols work

What a VPN is made of

A VPN connection is not one monolithic mechanism but a stack of layers. The bottom — transport (UDP or TCP). On top — acryptographic handshake (TLS 1.3 in modern protocols). On top of that — the tunnel (VLESS, Hysteria, WireGuard packet format). Above — congestion control(speed management). And finally — camouflage (Reality, obfs, xHTTP).

Different protocols pick different combinations. WireGuard: UDP + lightweight crypto (ChaCha20) + its own packet format + simple congestion control + no camouflage. OpenVPN: TCP or UDP + TLS 1.2 + its own tunnel + TCP Cubic + no camouflage. Hysteria 2: UDP + TLS 1.3 via QUIC + its own tunnel + Brutal + QUIC camouflage. VLESS Reality: TCP + TLS 1.3 with fake SNI + simple VLESS tunnel + BBR + maximum camouflage.

QUIC — what it is

QUIC is a transport protocol that Google developed in 2012 and IETF standardized in 2021 (RFC 9000). Core idea: combine in one protocol everything TCP + TLS + HTTP/2 do together. QUIC runs over UDP but still provides delivery guarantees, encryption, and stream multiplexing.

Advantages over TCP+TLS: 3× faster handshake (1 RTT vs 3 RTT); 0-RTT resumption on reconnect; stream multiplexing without head-of-line blocking (in TCP a single packet loss stalls the whole flow; in QUIC — only one stream); congestion-control updates without kernel rebuilds (which TCP needs).

TLS 1.3 vs TLS 1.2

TLS 1.3 was standardized in 2018 and today powers ~80% of HTTPS connections. Key difference from 1.2 — a drastically simplified handshake. 1.2 took 2 round-trips to first byte; 1.3 takes 1, with 0-RTT support for reconnects.

Matters for VPNs for two reasons. First — reconnection speed. On mobile internet connections drop often; TLS 1.3 with 0-RTT restores connections without noticeable pauses. Second — fingerprinting. TLS 1.2 had many parameters to fingerprint (JA3). 1.3 has fewer, and the difference between a regular browser and a VPN client shrinks.

Congestion control — the speed engine

This is where performance lives. Congestion control decides how much data to send into the network per unit of time: too much — packet loss and backoff, too little — idle channel.

TCP Cubic (Linux default since 2006) grows the congestion window along a cubic curve; on loss — halves it. Stable on predictable networks, bad on lossy.

TCP BBR (Google, 2016) doesn't use loss as a signal — instead measures bandwidth × RTT and sends what fits in the pipe without buffer bloat. Beats Cubic everywhere, especially on lossy networks.

Brutal (Hysteria, 2022) is more aggressive than BBR. Fully ignores loss, holds user-specified bandwidth. Best choice for VPNs on unstable networks, less polite to other connections on the wire.

Why WireGuard isn't the ceiling

WireGuard is an excellent 2018-era protocol (its release year). Minimal overhead, simple packet format, modern crypto. But it has architectural limits that bite in 2026.

First — a fixed packet format with no userspace parameters; can't change congestion control without touching the kernel. Second — UDP-only, no TCP fallback. Third — no built-in camouflage; WireGuard traffic is easily fingerprinted. Fourth — static handshake, no 0-RTT resumption.

Hysteria 2 solves all of that: userspace implementation with flexible congestion control, QUIC camouflage, 0-RTT via QUIC, TCP fallback possible. Lunaire doesn't use WireGuard for these reasons — Hysteria 2 gives the same operational simplicity with better performance and stealth.

Camouflage — obfs, Reality, xHTTP

Camouflage is a separate layer that tries to make VPN traffic look like something ordinary. Three main approaches.

obfs (legacy) — random bytes with particular entropy so traffic doesn't look recognizable. Problem — modern classifiers detect obfs by entropy signatures. Hysteria 1 had obfs; Hysteria 2 dropped it.

Reality (2023, Xray) — a new approach. Instead of looking like "nothing", it tries to look like a specific existing site. On handshake the server returns a real Microsoft or Apple certificate and the client appears as a regular visitor to that site.

xHTTP (2024) — a tunnel over HTTP/2 visually indistinguishable from regular HTTP/2 requests. Less stealthy than Reality but works through CDN and Cloudflare.

What to use in 2026

For self-hosted servers: Hysteria 2 as primary + VLESS Reality for UDP-blocked environments. For subscription services — pick ones that carry both protocols at once, e.g. Lunaire. Classic WireGuard-only services in 2026 are gradually becoming legacy.