Learn Envoy Proxy - TLS & mTLS

#Introduction

The microservices world doesn't run on plain HTTP. Episode 6 secures the traffic path with TLS and mTLS: you'll learn to encrypt connections from clients to Envoy (termination), from Envoy to backends (origination), and finally mutual TLS where both sides verify each other's identity.

This topic also introduces SDS — the Secret Discovery Service — which automates certificate rotation without restarts. By the end of the episode, you'll understand why a service mesh like Istio can turn on mTLS between services with just configuration.

#Creating Local Test Certificates

#Self-Signed Certificates for the Lab

Before configuration, create test certificates with openssl:

openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout ~/envoy-lab/certs/ca-key.pem -out ~/envoy-lab/certs/ca.pem \
  -subj "/CN=Envoy Lab CA"
 
openssl req -newkey rsa:2048 -nodes \
  -keyout ~/envoy-lab/certs/server-key.pem -out ~/envoy-lab/certs/server.csr \
  -subj "/CN=envoy.example.com"
 
openssl x509 -req -in ~/envoy-lab/certs/server.csr \
  -CA ~/envoy-lab/certs/ca.pem -CAkey ~/envoy-lab/certs/ca-key.pem \
  -CAcreateserial -out ~/envoy-lab/certs/server.pem -days 365

The openssl x509 -req command signs the server CSR with the newly created CA. The result is the server.pem / server-key.pem pair plus the ca.pem CA that will be used for verification.

#TLS Termination at the Listener

#Configuring the Transport Socket

To accept HTTPS at Envoy, add a TLS transport socket to the listener:

listeners:
  - name: listener_tls
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10443
    filter_chains:
      - transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            common_tls_context:
              tls_certificates:
                - certificate_chain:
                    filename: /etc/envoy/certs/server.pem
                  private_key:
                    filename: /etc/envoy/certs/server-key.pem
        filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: tls_ingress
              route_config:
                name: tls_routes
                virtual_hosts:
                  - name: tls_vh
                    domains:
                      - "envoy.example.com"
                    routes:
                      - match:
                          prefix: "/"
                        route:
                          cluster: api_service
              http_filters:
                - name: envoy.filters.http.router
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

The key part is DownstreamTlsContext, which tells Envoy how to secure connections from clients. Because the certificate files are mounted from the certs directory, make sure your volume mount includes that directory when running the container.

#TLS Origination to Upstream

#Envoy as a TLS Client

The reverse direction: Envoy encrypts the connection to the backend. This configuration lives in the cluster's transport socket:

clusters:
  - name: secure_backend
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          validation_context:
            trusted_ca:
              filename: /etc/envoy/certs/ca.pem
            match_subject_alt_names:
              - exact: backend.internal
    load_assignment:
      cluster_name: secure_backend
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend.internal
                    port_value: 8443

UpstreamTlsContext tells Envoy to encrypt the connection to the backend and verify its certificate against the CA in validation_context. The backend certificate must have the SAN backend.internal to match match_subject_alt_names.

#Mutual TLS (mTLS)

#Both Sides Verify Each Other

mTLS combines both: the client (Envoy) must also present its certificate when opening a connection. On Envoy's side as an upstream:

transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
    common_tls_context:
      tls_certificates:
        - certificate_chain:
            filename: /etc/envoy/certs/client.pem
          private_key:
            filename: /etc/envoy/certs/client-key.pem
      validation_context:
        trusted_ca:
          filename: /etc/envoy/certs/ca.pem

On the listener side that accepts mTLS, add require_client_certificate: true inside DownstreamTlsContext. This combination is what makes two services only willing to talk if the counterpart's identity is proven by the same CA.

#Certificate Rotation and SDS

#The Static Certificate Problem

Static certificates in the bootstrap have a major drawback: rotation requires restarting Envoy. In production, restarting every proxy to replace certificates is an expensive and risky operation. The solution is SDS — Envoy fetches certificates from a dynamic source.

#Connecting SDS to a Certificate Source

With SDS, tls_certificates is replaced by a reference to a secret managed by an external source:

common_tls_context:
  tls_certificate_sds_secret_configs:
    - name: server_cert
      sds_config:
        path: /etc/envoy/sds.yaml

The tls_certificate_sds_secret_configs configuration tells Envoy to request a secret named server_cert through the source defined in sds.yaml. When the control plane updates the secret, Envoy loads the new certificate without restarting — the same mechanism Istio uses to rotate sidecar certificates. To inspect the active certificates, open local:9901/certs — that endpoint lists all currently loaded certificates complete with their expiration dates.

#Closing

Episode 6 secured Envoy's transport from every angle: termination for clients, origination for upstreams, mTLS verifying both directions, and SDS automating certificate rotation.

Key takeaways:

• DownstreamTlsContext secures connections from clients; UpstreamTlsContext secures connections to backends.
• TLS termination needs a server certificate; TLS origination needs a trusted CA.
• mTLS combines a client certificate and CA verification on both sides.
• openssl s_client is the go-to tool for inspecting TLS handshakes.
• Static certificates require restarts to rotate; SDS replaces them dynamically.
• local:9901/certs shows active certificates and their expiration dates.

In the next episode, episode 7, we'll discuss access logging and basic observability — enabling access logs, log format and request metadata, liveness/readiness health checks for Envoy itself, and how Envoy exposes metrics to Prometheus.