Learn PKI - TLS Server & Client Certificates (mTLS)

#Introduction

In the previous episode, episode 6, you mastered revocation: generating CRLs, running an OCSP responder, and understanding what happens when a revoked certificate appears in a handshake. The trust foundation is now complete — it is time for the more fun part: installing certificates on real servers and watching TLS work in the real world.

Up to this point, we have only tested certificates in the lab. Episode 7 is the transition episode toward production. You will learn to configure server TLS certificates with the correct chain and SAN, test the configuration result with openssl s_client, then take it up a level: mutual TLS (mTLS) that verifies clients with client certificates.

This episode's roadmap: first we tidy up the certificate files and chain, second we configure Nginx and HAProxy, third we test with openssl s_client, and finally we build mTLS for two-way authentication along with its application scenarios in internal APIs and service mesh.

#Preparing the Correct Server Certificate

#File Structure and Certificate Chain

A server certificate does not stand alone. It sits at the end of a chain of trust that starts at the root CA, passes through one or more intermediate CAs, and ends at the server's leaf certificate. Clients must be able to trace this chain from leaf to a root already in the trust store.

That is why the server should not be given only the leaf file. Best practice is to provide a fullchain: the leaf certificate combined with all intermediate CAs in a single file, ordered from leaf toward root. The private key is stored separately and must be protected with strict permissions.

ls -l /etc/ssl/private/api-server.key
ls -l /etc/ssl/certs/fullchain.pem
cat fullchain.pem

The cat fullchain.pem command above shows the contents of the fullchain. You will see the leaf certificate at the top, followed by the intermediate CA below it. The root CA does not need to be in the fullchain — relying parties already have the root in their trust store, and adding it could actually trigger a warning about redundant certificates.

#SAN and IP Address

Most server configurations fail not because of the cryptography, but because of the Subject Alternative Name (SAN). Modern browsers and TLS libraries ignore the Common Name and only check the SAN to match host names. If the SAN does not contain the name the client uses, the connection is rejected even though the certificate is valid.

SANs can be DNS names, wildcards, or direct IP addresses. For internal services often accessed via IP, make sure the IP address is included. Here is how to add a SAN when creating a CSR.

openssl req -new -key /etc/ssl/private/api-server.key \
  -addext "subjectAltName = DNS:api.internal,DNS:api.internal.local,IP:10.0.5.12" \
  -out api-server.csr

Notice the -addext that injects the SAN directly into the CSR. After the certificate is issued, double-check that its SAN is correct before installing it on the server, because a small mistake here means downtime for all clients.

#Configuring Nginx

Nginx is the most common choice for TLS termination. What matters is pointing ssl_certificate at the fullchain and ssl_certificate_key at the private key. Do not mix them up: pointing at the leaf only will make clients without the intermediate fail to verify the chain.

server {
    listen 443 ssl;
    server_name api.internal;
 
    ssl_certificate /etc/ssl/certs/fullchain.pem;
    ssl_certificate_key /etc/ssl/private/api-server.key;
    ssl_protocols TLSv1.2 TLSv1.3;
}

After writing the configuration, test the syntax and reload. Nginx applies changes without dropping active connections.

nginx -t
nginx -s reload

The nginx -t output will confirm the configuration file is valid, including that the certificate can be read correctly. If there is an error, check the file paths and read permissions on the private key.

#Configuring HAProxy

HAProxy is an alternative widely used as a load balancer and TLS termination. Unlike Nginx, HAProxy accepts the certificate in a single PEM file containing the leaf, intermediate, and private key all at once.

frontend fe_https
    bind :443 ssl crt /etc/haproxy/certs/api.pem
    default_backend be_api

The order of the contents of the api.pem file for HAProxy: private key, then leaf certificate, then intermediate. Combine them in that order and make sure the file permissions only allow the HAProxy process to read it.

#Testing with openssl s_client

Before telling everyone the service is secure, test the handshake first. openssl s_client is the most versatile tool for this purpose. It opens a TLS connection to the server and shows the certificate details and its verification path.

openssl s_client -connect api.internal:443 \
  -servername api.internal -showcerts

The -servername option sends SNI so the server selects the right certificate when there are many virtual hosts. -showcerts displays the entire chain the server sends. Check the Certificate chain section: all certificates must appear, and the Verify return code section must read ok.

To test with a specific trust store, for example only trusting your internal root CA, use -CAfile.

openssl s_client -connect api.internal:443 \
  -servername api.internal -CAfile /etc/ssl/root-ca.crt

If Verify return code is not ok, check the fullchain, SAN, and expiry dates again. openssl s_client is your best friend when debugging TLS — get into the habit of using it before adding layers of automation.

#mTLS: Mutual TLS

#The Client Certificate Concept

Standard TLS only verifies in one direction: the client verifies the server's identity. The server does not care who the client is as long as the handshake succeeds. Mutual TLS (mTLS) flips that logic: the server also requests and verifies a certificate from the client, so both parties recognize each other.

A client certificate is essentially the same as a server certificate, only issued for a client entity such as a service, machine, or human, and usually carries the Extended Key Usage client auth. The server checks that certificate against a trusted CA — which can be the same CA that issued the server certificate, or a separate CA dedicated to clients.

#Scenario: Internal API

The most common scenario is an internal API that only certain services are allowed to call. Without mTLS, anyone who can reach the network could call that API. With mTLS, the server rejects any connection that does not carry a valid client certificate.

In Nginx, this feature is enabled via ssl_verify_client. When set to on, clients without a valid certificate are rejected at the handshake level.

server {
    listen 443 ssl;
    ssl_certificate /etc/ssl/certs/fullchain.pem;
    ssl_certificate_key /etc/ssl/private/api-server.key;
 
    ssl_client_certificate /etc/ssl/certs/client-ca.pem;
    ssl_verify_client on;
}

ssl_client_certificate points to the CA allowed to issue client certificates. It is best to keep this CA separate from the server CA, so revocation or trust for clients can be managed without affecting server certificates. Revoked client certificates will be rejected according to the CRL and OCSP mechanisms we discussed in episode 6.

#Scenario: Service Mesh

Managing client certificates manually for hundreds of services is obviously impractical. This is where service mesh comes in. Technologies such as Istio or Linkerd inject certificate-based identity into every workload automatically through a sidecar, manage rotation, and enforce mTLS policy across the entire mesh.

The difference from manual configuration: you never touch the application code. The sidecar handles the mTLS handshake for the pod, while the application still communicates over localhost. Policy determines the mTLS mode: permissive during migration, and strict once all workloads are ready.

#Enforcing verify_client Across Servers

Besides Nginx, almost all TLS servers have a similar mechanism. In HAProxy, client verification is configured with the verify keyword on the bind line.

frontend fe_internal_api
    bind :8443 ssl crt /etc/haproxy/certs/api.pem \
         verify required ca-file /etc/haproxy/certs/client-ca.pem
    default_backend be_api

The required value on verify makes the handshake fail when the client does not send a valid certificate. This is equivalent to ssl_verify_client on in Nginx. Make sure the policy is also enforced at the application level, because TLS only answers the identity question, not the authorization question.

To test the mTLS configuration, use curl with the --cert and --key options.

curl -s --cacert /etc/ssl/root-ca.crt \
  --cert /etc/ssl/certs/billing-client.pem \
  --key /etc/ssl/private/billing-client.key \
  https://api.internal/health

If you send a request without a client certificate, the connection will be rejected with a bad certificate alert or a TLS error from the server. This is proof that mTLS genuinely enforces identity, not just decoration.

#Closing

Episode 7 takes you from the lab to production. You learned to assemble the fullchain correctly, add a SAN covering names and IPs, configure Nginx and HAProxy, test the handshake with openssl s_client, and build mTLS that verifies client identity at the TLS level.

Key takeaways:

• The server needs a fullchain containing the leaf and intermediate, with a separate private key that has strict permissions.
• The SAN determines the valid host names and IPs; the Common Name is no longer enough for modern browsers.
• openssl s_client is the main tool for verifying handshake, chain, and SAN before production use.
• mTLS verifies client identity through a client certificate issued by a trusted CA.
• ssl_verify_client on in Nginx and verify required in HAProxy force clients to present a certificate.
• Service mesh automates the distribution, rotation, and enforcement of mTLS for many workloads.

Up to this episode, all certificates were still managed manually or semi-manually. In episode 8, we will throw away that old way: you will learn the ACME Protocol from RFC 8555 — the order-to-issuance flow, three challenge types, External Account Binding — then apply it with Let's Encrypt and certbot. See you there!