Learn Envoy Proxy - Performance Tuning & Resource Management

#Introduction

Envoy can handle millions of requests per second if configured correctly — and stall if it isn't. Episode 15 covers performance tuning and resource management: the threading model, worker threads, connection limits, buffer sizes, HTTP/2 pool tuning, and resource optimization in the bootstrap. The philosophy is simple: before adding hardware, understand first where Envoy spends CPU and memory.

#Envoy's Threading Model

#Main Thread, Worker Threads, and File Flushers

Envoy runs with several types of threads:

• Main thread: manages configuration, DNS, the cluster manager, and administrative events.
• Worker threads: process all connections and requests — these work the hardest.
• File flusher threads: write access logs and file buffering.

The number of worker threads is controlled by the --concurrency flag when Envoy runs, or follows the core count automatically if not set.

envoy --concurrency 4 -c /etc/envoy/envoy.yaml
docker run -d --cpus=4 --name envoy-perf \
  -v ~/envoy-lab/configs:/etc/envoy \
  -p 10000:10000 -p 9901:9901 \
  envoyproxy/envoy:v1.31.0

The envoy --concurrency 4 command creates 4 worker threads. A rule of thumb: one worker per core. Concurrency beyond the core count only adds context switching costs.

#Detecting Worker Balance

To make sure load is spread evenly across workers:

curl -s localhost:9901/stats | grep "worker_thread"
curl -s localhost:9901/stats | grep "runtime_override_count"

If one worker handles far more connections than others, there may be connection skew — a problem often seen on TCP listeners with long keepalives. The admin interface metrics are the starting point for diagnosis.

#Connection Limits and Buffer Sizes

#Limiting Connections at the Listener

Watch connection_limit per listener and max_connections per cluster:

listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    connection_limit: 10000
    per_connection_buffer_limit_bytes: 32768
clusters:
  - name: api_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST
    per_connection_buffer_limit_bytes: 32768
    circuit_breakers:
      thresholds:
        - max_connections: 5000
          max_requests: 10000

The value per_connection_buffer_limit_bytes: 32768 caps each connection's read/write buffers at 32 KiB. Buffers that are too large bloat memory; ones that are too small drop throughput because of many syscalls.

#Estimating Connection Memory

Each connection uses read and write buffers. With a 32 KiB limit, 10 thousand active connections need around 640 MB just for buffers. Adjust limits so total buffer memory fits your container budget.

#HTTP/2 Pool Tuning

#Configuring HTTP/2 Protocol Options

For upstream clusters that support HTTP/2, pool configuration heavily determines latency:

clusters:
  - name: api_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST
    http2_protocol_options:
      max_concurrent_streams: 100
      initial_stream_window_size: 1048576
      initial_connection_window_size: 6291456
      connection_keepalive:
        interval: 30s
        timeout: 5s

max_concurrent_streams: 100 controls how many parallel streams each HTTP/2 connection allows. A larger initial_connection_window_size increases throughput for large transfers but adds memory per connection. This tuning is characteristic of HTTP/2 and has no counterpart in HTTP/1.

#Keepalive to Save Connections

connection_keepalive keeps HTTP/2 connections warm with periodic PINGs, so connections don't die at NAT devices. This reduces the frequency of expensive new connection creation.

#HTTP/3 and QUIC

If an edge listener serves modern clients, enable HTTP/3:

listeners:
  - name: listener_http3
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
    filter_chains:
      - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              http3_protocol_options: {}
              stat_prefix: h3_ingress
              route_config:
                name: h3_routes
                virtual_hosts: []
              http_filters:
                - name: envoy.filters.http.router
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

The udp_listener_config and http3_protocol_options sections enable QUIC/HTTP/3 on the listener. HTTP/3 reduces latency on unstable networks — traded off against operational complexity.

#CPU and Memory Optimization

#Bootstrap Tuning

Some bootstrap settings affect overall resource usage:

bootstrap:
  idle_timeout: 300s
  drain_time: 5s
  parent_shutdown_time: 5s
stats_config:
  stats_flush_interval: 30s
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

The value idle_timeout: 300s closes idle connections, freeing buffers. drain_time and parent_shutdown_time govern behavior when Envoy switches versions — important for rolling updates without dropping requests.

#Avoiding Over-Scraping

Scraping Prometheus too often triggers large string allocations. An interval of 15 to 30 seconds is usually enough; for high-cardinality metrics, episode 21 covers more detailed strategies.

#Measuring the Impact of Tuning

#Simple Benchmarking

To measure improvement, run a benchmark before and after tuning:

for i in $(seq 1 200); do
  curl -s -o /dev/null http://localhost:10000/api/ping
done
k6 run --vus 50 --duration 30s loadtest.js

The k6 run command (if installed) gives you p50, p95, and p99 metrics that are more reliable than a curl loop. Compare the numbers before and after changing max_concurrent_streams or buffer sizes to make sure a change truly helps.

#Viewing Envoy's Resources

docker stats envoy-perf
curl -s localhost:9901/stats | grep "^server.memory_allocated"

server.memory_allocated shows the memory Envoy has allocated. Combine it with docker stats to see whether buffer tuning is visible at the OS level.

#Closing

Episode 15 equipped you to optimize Envoy: understanding worker threads, limiting connections and buffers, tuning the HTTP/2 pool, and measuring the impact of every change.

Key takeaways:

• Envoy uses main threads, worker threads, and file flusher threads.
• --concurrency sets the worker count; one worker per core is the rule of thumb.
• per_connection_buffer_limit_bytes controls memory per connection.
• The HTTP/2 pool is tuned via max_concurrent_streams and window sizes.
• HTTP/3 is enabled with udp_listener_config and http3_protocol_options.
• Measure with benchmarks before and after tuning; don't tune blindly.

In the next episode, episode 16, we'll discuss Envoy extensions and WASM filters — Envoy's extension model, how to write a simple WASM filter, and use cases for custom auth, telemetry enrichment, and request transformation.