Learn L2TP IPsec - Authentication: PSK, Certificates & XAUTH

#Introduction

IKE determines how keys are built, but there is a more fundamental question: who is allowed to talk? Episode 6 discusses authentication — the mechanism for proving identity before a tunnel is allowed to stand up. L2TP/IPsec offers three main approaches: Pre-Shared Key, X.509 certificates, and Extended Authentication (XAUTH).

Choosing the authentication method is the most impactful security decision across the entire deployment. Using the same PSK for a thousand users is very different from giving every user a private certificate. Let us dissect the strengths, weaknesses, and implementation of each.

#Pre-Shared Key (PSK)

#Easy, But Limited

PSK is a shared secret key configured at both ends. Its setup is the simplest — just one line in ipsec.secrets and the same value on the client side. PSK is used together with IKEv1 Main Mode to authenticate both parties symmetrically.

Its main weakness: a single PSK is usually shared by all users. If it leaks, every connection can be compromised until the PSK is replaced. That is why PSK is only suitable for small environments with trusted users, and must be rotated periodically.

Its configuration in Libreswan:

203.0.113.10 198.51.100.20 : PSK "rahasia-bersama-yang-panjang"

And it is selected via authby=secret in ipsec.conf:

conn L2TP-PSK
    authby=secret

#X.509 Certificates

#Mutual Authentication with a CA Hierarchy

For enterprises, X.509 certificates are the standard. Every host has a public-private key pair; identity is proven through digital signatures chained to a trusted Certificate Authority (CA). The result is mutual authentication: the server proves its identity to the client, and the client proves its identity to the server — without sharing a single common secret.

Certificates solve the PSK-sharing problem: every user has their own credentials that can be revoked without affecting other users. The details of CA, CRL, and OCSP management are covered thoroughly in episode 16. For now, recognize the shape of the configuration:

conn vpn-cert
    authby=rsasig
    leftcert=server.pem
    rightca=%same
    leftid=@vpn.example.com
    rightid=%fromcert

The leftcert=server.pem column points to the server certificate, while rightca=%same asserts that the client's CA must be the same as the server's CA. Certificates are stored in Libreswan's NSS database, not in plain files.

#Extended Authentication (XAUTH)

#User Authentication on Top of the IPsec SA

XAUTH is a characteristic IKEv1 extra layer that authenticates the user — not just the machine — using a username and password. The flow is layered: Phase 1 builds the IPsec SA, then XAUTH runs as a second authentication that requests the user's credentials, and only after success does the IPsec tunnel actually become active.

XAUTH can source credentials from a local file, RADIUS, or LDAP. In Libreswan, enable it with:

conn vpn-xauth
    authby=secret
    xauthby=file
    xauthfail=hard
    leftxauthserver=yes
    rightxauthclient=yes

User data is stored in /etc/ipsec.d/passwd with this format:

budi:$1$abcdefgh$hashmd5panjang:192.168.42.5
sari:$1$abcdefgh$hashmd5panjang:192.168.42.6

Use openssl passwd -1 to generate the matching MD5 crypt hash. The third column (optional) sets a static Virtual IP that will be covered in episode 17.

#Choosing the Right Method

#A Decision Matrix

There is no single method that is always best — the decision follows scale and threat model:

• Fewer than 10 users, all trusted: PSK is enough; rotate it periodically.
• Enterprise, many users, revocation required: X.509 certificates are mandatory.
• Per-user authentication with the same credentials as other systems: XAUTH with RADIUS or LDAP.
• Strict compliance: a combination of certificates for machines and XAUTH or EAP for users.

Note the interactions as well: XAUTH is an IKEv1 feature; on IKEv2 its role is taken over by EAP. So the IKE version decision in episode 4 also determines which authentication options are available.

There is no hard rule locking your choice — combining methods is actually common. For example, PSK for fast lab deployment, then moving up to certificates when the system opens to many users. What matters is that this decision is documented and approved, because the authentication method is the security component users see the most.

#Closing

Episode 6 completed the three pillars of L2TP/IPsec authentication: simple PSK for small scale, X.509 certificates with mutual authentication for enterprises, and XAUTH for user authentication via a file, RADIUS, or LDAP.

Key takeaways:

• PSK is simple but risky if shared among many users.
• X.509 certificates provide mutual authentication and per-user revocation.
• XAUTH adds user authentication on top of an IKEv1 IPsec SA.
• authby=secret for PSK, authby=rsasig for certificates.
• xauthby=file or xauthby=radius to store XAUTH users.
• Do not combine XAUTH with Aggressive Mode.

In the next episode, episode 7, we will discuss PPP and L2TP configuration — how PPP carries packets, the authentication protocols PAP, CHAP, MS-CHAPv2, and EAP, and the tunnel handshake sequence from SCCRQ to session ICCN.