Learn Envoy Proxy - Production Hardening & Best Practices

#Introduction

This series ends not with new material but with unification: production hardening and best practices. Episode 22 distills everything you've learned into two checklists — security and operations — plus how to handle Envoy upgrades and fallback strategies when something goes wrong.

This is the episode closest to a platform engineer's daily work. You won't find new configuration here; you'll find the order of priorities and habits that separate the lab from production.

#Security Hardening Checklist

#The Security Checklist

Here's the security checklist you must review before Envoy serves production traffic:

• TLS on every path: termination at listeners, origination to upstreams, certificates from a trusted CA.
• mTLS for internal communication with SAN or SPIFFE verification.
• JWT or ext_authz at the edge for authentication.
• RBAC for attribute-based authorization.
• Rate limiting with failure_mode_deny for public APIs.
• Admin interface not exposed publicly: Envoy's admin should only be reachable from the internal network.

#Securing the Admin Interface

Envoy's admin interface is very powerful — and very dangerous if open to the public:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/server_info
curl -s localhost:9901/stats | grep "admin"

If server_info is reachable from a public network, you must restrict it immediately. Recommended practices:

• Don't bind admin to 0.0.0.0; use 127.0.0.1 or an internal network.
• Never expose port 9901 through a public load balancer.
• Use NetworkPolicy or a firewall to restrict access.
• Consider HTTPS for admin when available.

#Closing Other Surfaces

Additional items often forgotten:

• Make sure failure_mode_allow isn't on without reason for security filters.
• Disable unused features to reduce the attack surface.
• Keep dependencies and images updated with security patches.
• Monitor security metrics: RBAC rejects, JWT failures, and rate limit rejections.

#Operational Checklist and Runbook

#The Operational Checklist

This checklist covers the healthy running side:

• Config managed in Git and validated with envoy --mode validate in CI.
• Health checks for readiness and liveness active in the orchestrator.
• Access logs include trace IDs, request IDs, and response flags.
• Metrics scraped by Prometheus with dashboards for listeners and clusters.
• Tracing active with reasonable sampling.
• Circuit breakers and outlier detection configured on all clusters.
• Backup and restore of the bootstrap for disaster recovery.

#Writing a Runbook

A runbook is a pre-written recovery procedure. One runbook you must have for Envoy: "Envoy is unhealthy". Its minimal contents:

1. Open local:9901/healthcheck/fail or make sure readiness fails.
2. Check local:9901/server_info and startup logs.
3. Look at key metrics: upstream_cx_total, upstream_rq_5xx, listener_downstream_cx_active.
4. If the config looks suspicious, kubectl rollout undo or git revert.
5. Bring traffic back when the health check passes again.

curl -s localhost:9901/server_info
curl -s localhost:9901/config_dump | head -30
curl -s localhost:9901/stats | grep -E "upstream_rq_5xx|downstream_cx_active"
docker logs envoy-prod 2>&1 | tail -50

The four runbook commands above collect diagnostic information in one session. These commands often reveal the root cause before any other step is taken.

#Recovery Drills

A runbook is useless if never tested. Run drills periodically: shut down one control plane, restart one Envoy, or enable fault injection in staging. Practice makes the team confident the runbook truly works when a real incident arrives.

#Upgrading Envoy and Fallback Strategies

#Planning an Upgrade

Upgrading Envoy needs planning because some features change between versions:

1. Read the release notes and list of breaking changes.
2. Test the config in staging with the new version.
3. Validate all features you use (especially WASM and custom filters).
4. Roll the upgrade out gradually to production.
5. Monitor metrics after each stage.

envoy --version | grep "version"
docker images envoyproxy/envoy --format "{{.Tag}}"

The envoy --version command confirms the running binary. Always record the config version alongside the Envoy version — that config-version pair is what matters for rollback.

#Fallback Strategy

Fallback means having a quick exit when an upgrade or change causes problems:

• Config rollback: restore the previous config via GitOps.
• Image rollback: use the previous Envoy image version.
• Staging environment: keep one old-version fleet as a reserve.
• Drain and shift: move traffic to healthy instances with drain_listeners.

curl -s -X POST localhost:9901/drain_listeners
curl -s -X POST localhost:9901/healthcheck/fail

The drain_listeners and healthcheck/fail commands are the safe exit: Envoy stops accepting new connections while finishing in-flight requests, then the pod can be stopped without dropping traffic.

#Building a Compatibility Matrix

Create a compatibility table of config versus Envoy versions: which features are used, at which version each feature became stable, and which config needs to change on upgrade. This matrix turns an upgrade from a stressful event into a scheduled step.

#Reflecting on the Whole Series

#What You've Mastered

Closing episode 22, let's summarize this 23-episode journey:

• From networking prerequisites in episode 0 to production configuration in episode 22.
• Listeners, clusters, routes, and filter chains as the architectural foundation.
• xDS and control planes for dynamic configuration.
• TLS, mTLS, JWT, RBAC, and ext_authz for security.
• Canaries, mirroring, and fault injection for traffic shaping.
• Prometheus, tracing, access logs, and SLOs for observability.
• GitOps, CI/CD, and runbooks for operations.

Every concept that appears in production Envoy — in Istio, Gloo, or a direct deployment — now has meaning for you because you've built it from the ground up.

#Closing

Episode 22 ended the series with production discipline: security and operational checklists, a tested runbook, and upgrade and fallback strategies that make changes less frightening.

Key takeaways:

• Hardening starts with TLS, mTLS, authentication, RBAC, and rate limiting.
• The admin interface must be restricted to the internal network, not public.
• Config is managed in Git, validated in CI, and rolled out gradually.
• A runbook must be written, tested, and used during incidents.
• Envoy upgrades are planned with release notes, staging, and metrics.
• Always have a fallback: config rollback, image rollback, and listener drain.

See you in the next series! You can now read Envoy config in any service mesh with confidence. If you'd like to keep learning, my recommendation is to dive into Istio as the most mature application of Envoy as a data plane, or build your own xDS control plane with go-control-plane to understand the protocol from the inside. Happy building with Envoy!