Learn L2TP IPsec - Performance Tuning & MTU

#Introduction

Slow VPNs are rarely caused by the server CPU — they are almost always caused by MTU and fragmentation. Episode 15 covers two things at once: how the L2TP/IPsec encapsulation overhead shrinks the packet size you can send, and how to optimize performance from the hardware and daemon process side.

Understanding MTU lets you solve the classic "the website will not open but ping works fine" symptom — a disruption almost always rooted in large fragmented packets. After that we move up to performance tuning for large-scale deployments.

#Calculating the L2TP/IPsec Overhead

#How Many Bytes Are Lost

Every packet crossing the VPN accumulates headers from each layer: PPP, L2TP, UDP, ESP, and the outer IP header. The total is around 60-80 bytes per packet. With an ethernet MTU of 1500, the MTU available for data inside the tunnel usually falls in the 1350-1420 byte range.

The overhead breakdown:

PPP + L2TP + UDP : 12 byte (6 + 8 + 4 di atas UDP header)
ESP header       : 8-16 byte
ESP trailer + ICV: 16-24 byte
Header IP luar   : 20 byte
Total            : kira-kira 60-80 byte

That is why the commonly used ppp0 MTU is 1400, giving safe room for almost any algorithm combination.

#MSS Clamping

#Stopping Fragmentation at the Source

MSS clamping changes the Maximum Segment Size value in TCP SYN packets so TCP never sends segments too large for the tunnel. This is the most powerful fix, and it is done at the firewall, not in the application.

sudo nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu
sudo nft add rule inet filter forward tcp flags syn tcp option maxseg size set 1360

Or the classic iptables version:

sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

The TCPMSS --clamp-mss-to-pmtu line computes the MSS from the detected path MTU. For deterministic results, set a fixed value of 1360 as in the nftables line above.

#Setting the PPP Interface MTU

#Tuning mtu and mru

In addition to clamping, set the MTU and MRU on the ppp0 interface via pppd options:

ipcp-accept-local
ipcp-accept-remote
mtu 1400
mru 1400
ms-dns 8.8.8.8

Add the two lines mtu 1400 and mru 1400 to /etc/ppp/options.xl2tpd. The ppp0 interface that forms will use these values, and IPCP offers them to the client during negotiation.

Check the active MTU value:

ip addr show ppp0

If mtu 1400 appears in the output, the setting worked. If fragmentation problems persist, step it down to 1350 and test with large pings:

ping -M do -s 1372 -c 3 192.168.42.50

#Performance Tuning

#AES-NI and Hardware Offload

Modern CPUs have AES-NI instructions that accelerate AES encryption in hardware. Make sure the kernel uses them:

grep -o aes /proc/cpuinfo | head -1

If the processor supports it, aes appears in flags. For machines with NICs that have IPsec offload, enable ESP hardware offload in the NIC driver — this moves encryption from the CPU to the network card.

#Multiprocessing in strongSwan

Charon supports multiple worker threads to handle many IKE connections at once. Adjust it to the CPU core count in strongswan.conf:

charon {
    threads = 16
    keep_alive = 20s
}

First measure the core count with nproc, then set threads = 16 (or an appropriate value). Do not overdo it — too many threads for few connections actually adds scheduling overhead.

#Closing

Episode 15 equipped you with the highest-impact performance tricks: the 60-80 byte overhead calculation, MSS clamping at the firewall, the PPP interface MTU, and leveraging AES-NI, ESP offload, and charon threads.

Key takeaways:

• L2TP/IPsec overhead is about 60-80 bytes per packet.
• A safe ppp0 MTU is usually 1400 or lower.
• MSS clamping with TCPMSS --clamp-mss-to-pmtu prevents fragmentation.
• Set mtu 1400 and mru 1400 in options.xl2tpd.
• Test packet size with ping -M do -s 1372.
• Leverage AES-NI, NIC offload, and charon threads for large scale.

In the next episode, episode 16, we will discuss certificate management — building a CA hierarchy, issuing certificates with certutil and pki, and handling revocation through CRL and OCSP.