Learn L2TP IPsec - Transport Mode vs Tunnel Mode

#Introduction

Now that you have mastered IKE, let us focus on ESP — the protocol that actually protects data. Episode 5 dissects its two working modes: transport mode and tunnel mode. They determine what gets encrypted and how the packet is reassembled, and they are among the first options you encounter in a configuration file.

The question that always comes up: why are there two modes, and when do you use which? The answer depends on who is communicating — two hosts or two gateways. We will answer it by looking at the packet contents in detail, then close with the role of both modes in the L2TP/IPsec architecture.

#Transport Mode: Protecting the Payload

#ESP Transport for Endpoints

In transport mode, ESP only encrypts the payload of the original IP packet — the IP header stays intact and visible on the network. This mode is used for endpoint-to-endpoint communication: two hosts or two daemons talking directly. Because the original header is preserved, there is no additional outer header, so overhead is smaller.

Packet layout in transport mode:

[Original IP header][ESP header][Encrypted payload][ESP trailer]

Transport mode fits when both communication endpoints are the same parties — for example two servers sending data directly, or when L2TP itself provides the tunneling. In the L2TP case, this is why many remote-access implementations use ESP transport mode: IPsec protects the UDP 1701 packets, while L2TP carries the PPP frames.

#Tunnel Mode: Wrapping the Entire Packet

#ESP Tunnel for Gateways

In tunnel mode, ESP encrypts the entire original IP packet — including its header — and wraps it in a new IP packet with an outer header. The outer header contains the gateway addresses, while the inner (original) header is hidden inside the encrypted payload. This mode is the default for site-to-site and gateway-to-gateway remote-access VPNs.

Packet layout in tunnel mode:

[Outer IP header][ESP header][Encrypted original IP packet][ESP trailer]

Because the entire original packet is hidden, tunnel mode protects internal details such as network topology and host addresses behind the gateway. This is what you need when two separate networks are connected over the internet.

#L2TP/IPsec and Mode Selection

#Both Modes Allowed by RFC 3193

RFC 3193 allows ESP in both transport mode and tunnel mode for L2TP/IPsec. What is never allowed is sending L2TP traffic without ESP protection. The practical choice depends on topology:

• Remote access (single LNS): transport mode is common, because L2TP already provides tunneling and IPsec only needs to protect the UDP 1701 packets. This is the type=transport configuration we wrote in episode 3.
• Site-to-site L2TP/IPsec (two LNSes): tunnel mode is common, because traffic must cross two different network gateways.

So the statement that L2TP/IPsec always uses tunnel mode needs to be understood carefully: what is always guaranteed is the presence of ESP, while the mode follows the topology. We will see the concrete details in the type= column below.

#Choosing the Mode in Configuration

#The type Column in Libreswan

In Libreswan, the ESP mode is determined by the type= value. For classic L2TP remote access:

conn L2TP-PSK
    type=transport
    leftprotoport=17/1701
    rightprotoport=17/%any

For site-to-site, switch to tunnel mode and remove the port restrictions:

conn office-to-dc
    type=tunnel
    left=203.0.113.10
    right=198.51.100.20
    leftsubnet=10.10.10.0/24
    rightsubnet=192.168.1.0/24

Check the active SA mode via the encapsulation: transport or tunnel line in the following output:

ipsec status
ipsec statusall | grep -i encapsulation

The ipsec statusall command shows all SA details, including the encapsulation mode and agreed algorithms. Reading it regularly is a good troubleshooting habit.

#Choosing the Mode: Practical Rules

#A Quick Decision Guide

The ESP mode decision is actually quite simple. Ask one question: are both connection endpoints on the same network?

• If yes — two hosts talking directly — use transport mode.
• If no — traffic must cross two different networks via gateways — use tunnel mode.

In the L2TP/IPsec context, this translates to: remote access with a single server uses transport mode, while site-to-site with two gateways uses tunnel mode. This rule applies consistently in both Libreswan and strongSwan.

#Common Mode-Selection Mistakes

Two mistakes occur frequently. First, using tunnel mode for L2TP remote access — it still works, but adds overhead with no benefit because L2TP is already the tunnel. Second, forgetting to match type on both sides — Libreswan will reject the SA or build the SA with an unexpected mode.

If ipsec statusall shows encapsulation: tunnel even though you wrote type=transport, first check whether the configuration file was reloaded:

sudo ipsec setup --restart

The ipsec setup --restart command reloads the entire configuration. A good habit: after changing type=, always restart and re-verify the ipsec statusall output.

#Closing

Episode 5 explained the core difference between transport mode and tunnel mode: transport mode protects the payload with low overhead for endpoint communication, while tunnel mode wraps the entire IP packet for gateways. You also learned that RFC 3193 allows both modes for L2TP/IPsec — as long as ESP is always present — and how to choose via the type= column.

Key takeaways:

• Transport mode only encrypts the payload; the IP header stays visible.
• Tunnel mode wraps the entire IP packet in a new outer header.
• Transport mode fits host-to-host and L2TP remote access.
• Tunnel mode fits site-to-site and gateway VPNs.
• RFC 3193 allows both modes; what is mandatory is the presence of ESP.
• In Libreswan, the mode choice is written in the type= column.

In the next episode, episode 6, we will discuss authentication: PSK, certificates, and XAUTH — the three ways to validate identity in L2TP/IPsec, the strengths and weaknesses of each, and when to use a simple PSK, enterprise X.509, or XAUTH for users.