Learn Envoy Proxy - Rate Limiting & Traffic Control

#Introduction

Uncontrolled traffic is a ticking time bomb. Episode 10 covers rate limiting and traffic control: how Envoy limits request counts, retries failed requests, bounds wait times, injects artificial failures for testing, and protects backends with circuit breaking.

Every feature in this episode answers one operational question: how does the system stay stable when traffic surges, when backends slow down, or when we deliberately want to test system resilience. You'll see that resilience isn't just about hardware — it's about configuration.

#Rate Limit Filter and External Rate Limit Service

#The Built-in Rate Limit Filter

Envoy has a rate limit filter that works together with an external rate limit service:

http_filters:
  - name: envoy.filters.http.ratelimit
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
      domain: api-gateway
      rate_limit_service:
        grpc_service:
          envoy_grpc:
            cluster_name: ratelimit_service
      failure_mode_deny: false
  - name: envoy.filters.http.router
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

The envoy.filters.http.ratelimit filter sends requests to the rate limit service for evaluation. Envoy itself doesn't store counters; the external service decides whether a request is allowed or denied. Each request is labeled with the api-gateway domain so policies can be distinguished per area.

#Configuring Rules at the Route Level

Rate limit rules are set per route or virtual host, as descriptions sent to the external service:

virtual_hosts:
  - name: api_vh
    domains:
      - api.example.com
    routes:
      - match:
          prefix: "/orders"
        route:
          cluster: orders_service
        typed_per_filter_config:
          envoy.filters.http.ratelimit:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimitPerRoute
            vh_rate_limits:
              - actions:
                  - request_headers:
                      header_name: X-Api-Key
                      descriptor_key: api_key
                  - generic_key:
                      descriptor_value: orders

The typed_per_filter_config section describes the request attributes the rate limit service uses to compute counters. In this example, the combination of the X-Api-Key header value and the orders value forms the limit key.

#Running the Rate Limit Service

Envoy's reference service can be run as a container:

docker run -d --name ratelimit -p 8081:8081 \
  -v ~/envoy-lab/configs/ratelimit-config.yaml:/data/ratelimit/config/config.yaml \
  envoyproxy/ratelimit:latest

The docker run command above runs the reference rate limit service with a YAML config defining limits per descriptor. That service configuration is what determines limits like "100 requests per minute per api_key".

#Retry and Timeout

#Configuring Retries at the Route Level

Retries make Envoy resend failed requests to another endpoint:

routes:
  - match:
      prefix: "/api/"
    route:
      cluster: api_backend
      timeout: 5s
      retry_policy:
        retry_on: connect-failure, retriable-status-codes
        num_retries: 3
        retry_host_predicate:
          - name: envoy.retry_host_predicates.previous_hosts
        retriable_status_codes:
          - 503

With retry_policy, Envoy retries up to 3 times for connection failures or 503 statuses, and avoids endpoints it has already tried. timeout: 5s bounds the total request duration including all retry attempts. Remember: avoid retries for non-idempotent requests, because duplication can happen at the backend.

#Fault Injection

#Injecting Artificial Failures

Fault injection lets you simulate failures to test system resilience:

routes:
  - match:
      prefix: "/api/"
    route:
      cluster: api_backend
    typed_per_filter_config:
      envoy.filters.http.fault:
        "@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.Fault
        abort:
          http_status: 503
          percentage:
            numerator: 10
            denominator: HUNDRED
        delay:
          fixed_delay: 2s
          percentage:
            numerator: 20
            denominator: HUNDRED

The fault configuration makes 10 percent of requests fail with a 503 status, and delays 20 percent of requests by 2 seconds. This combination of abort and delay is an ideal chaos engineering tool for testing the retry and timeout config we just made. Fault injection is for testing only — enable it in staging or via a dynamic config that can be switched off quickly.

#Circuit Breaking and Resource Limits

#Protecting Backends from Overload

Circuit breakers limit how many connections and requests can reach a single cluster:

clusters:
  - name: api_backend
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 1000
          max_pending_requests: 1024
          max_requests: 2000
          max_retries: 5
        - priority: HIGH
          max_connections: 200
          max_pending_requests: 256
          max_requests: 500
          max_retries: 3

The circuit_breakers block sets separate limits for DEFAULT and HIGH priorities. When max_requests is reached, new requests are rejected immediately without waiting — this is what protects backends from traffic floods. max_retries bounds active retries, preventing amplification effects. Watch the upstream_rq_pending_overflow metric in the admin interface: if it keeps climbing, the circuit breaker is doing its job protecting the backend.

#Closing

Episode 10 gave you full control over traffic: rate limiting with an external service, tailored retry and timeout, fault injection for testing, and circuit breaking that protects backends from overload.

Key takeaways:

• The rate limit filter delegates decisions to an external service over gRPC.
• Limit rules are set per route via typed_per_filter_config.
• Retries are triggered by conditions like connect-failure and retriable statuses.
• timeout bounds the total request duration including retries.
• Fault injection simulates abort and delay for chaos engineering.
• Circuit breakers limit connections, pending requests, active requests, and retries.

In the next episode, episode 11, we'll discuss observability and telemetry integration — Envoy metrics to Prometheus, distributed tracing with Zipkin, Jaeger, and OpenTelemetry, and access log enrichment for deeper observability.