Learn Envoy Proxy - Clusters & Load Balancing

#Introduction

In episode 4 you learned how to route requests, but everything still went to a single backend. Episode 5 gets to the heart of load distribution: clusters and load balancing. This is where Envoy shows its advantage over simple proxies — the ability to choose among many endpoints with the right algorithm, while keeping each backend healthy.

You'll learn to define a cluster with several endpoints, choose the load balancing policy that fits your needs, then enable active health checks and outlier detection so Envoy automatically avoids problematic backends.

#Defining Clusters and Endpoints

#A Cluster with Many Endpoints

A cluster is the backend abstraction; endpoints are the real instances inside it. Here's an example cluster with two instances:

clusters:
  - name: api_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: api_service
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: api-1.internal
                    port_value: 8080
            - endpoint:
                address:
                  socket_address:
                    address: api-2.internal
                    port_value: 8080

There are two common resolution types:

• STRICT_DNS — every DNS address is resolved, and all results are used.
• LOGICAL_DNS — only one resolved IP is used per connection.

The load_assignment configuration with multiple lb_endpoints is how you declare endpoints statically.

#Viewing Endpoints from the Admin Interface

To see the status of all endpoints managed by Envoy:

curl -s localhost:9901/clusters
curl -s localhost:9901/endpoints

The endpoint local:9901/endpoints shows the list of endpoints per cluster, complete with health status and metadata. This is the admin endpoint you'll check most often when troubleshooting load balancing.

#Load Balancing Policies

#Four Built-in Algorithms

Envoy provides several policies configured through lb_policy:

• ROUND_ROBIN — distributes requests evenly in rotation.
• LEAST_REQUEST — picks the endpoint with the fewest active requests.
• RING_HASH — consistently maps requests to the same endpoint for a given key.
• MAGLEV — a ring hash without cross-change consistency, statistically more even.

clusters:
  - name: cache_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: RING_HASH
    load_assignment:
      cluster_name: cache_service
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: cache-1.internal
                    port_value: 6379
            - endpoint:
                address:
                  socket_address:
                    address: cache-2.internal
                    port_value: 6379

RING_HASH with lb_policy is often used for caches or sessions: requests from the same client always land on the same endpoint, so the cache stays warmer and the hit rate rises.

#Active Health Checks

#Configuring an HTTP Health Check

Envoy can probe backends periodically with HTTP:

clusters:
  - name: api_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    health_checks:
      - timeout: 1s
        interval: 5s
        unhealthy_threshold: 3
        healthy_threshold: 2
        http_health_check:
          path: /healthz
    load_assignment:
      cluster_name: api_service
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: api-1.internal
                    port_value: 8080

The health_checks block makes Envoy send GET /healthz every 5 seconds. Three consecutive failures mark an endpoint unhealthy, and two successes return it to healthy. Unhealthy endpoints are automatically removed from the load balancing rotation.

#Connection Pools and Outlier Detection

#Circuit Breaking and Pools

The connection pool manages Envoy's connections to each endpoint so it doesn't open a new connection for every request:

clusters:
  - name: api_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST
    circuit_breakers:
      thresholds:
        - max_connections: 1000
          max_pending_requests: 1024
          max_requests: 2000

The circuit_breakers values limit the maximum load to a cluster: connections, pending requests, and active requests. We'll go deeper in episode 10, but from now on, get used to seeing this block as the cluster's main guardrail.

#Outlier Detection

Unlike proactive health checks, outlier detection is reactive — it ejects endpoints that start to slow down or error:

outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s

With consecutive_5xx: 5, an endpoint that produces 5 consecutive 5xx errors is ejected temporarily. This catches problems that periodic health checks miss.

#Closing

Episode 5 explained how Envoy distributes traffic: clusters as the backend abstraction, endpoints as real instances, four load balancing algorithms, active health checks, connection pools, and outlier detection that keeps backends healthy.

Key takeaways:

• A cluster is a collection of endpoints; load_assignment declares the backend addresses.
• ROUND_ROBIN for even load, LEAST_REQUEST for variable durations.
• RING_HASH and MAGLEV for key-based consistent hashing.
• Active health checks scan backends periodically with HTTP probes.
• Circuit breakers limit the maximum load to a cluster.
• Outlier detection temporarily ejects endpoints that start to error or slow down.

In the next episode, episode 6, we'll discuss TLS and mTLS — TLS termination at Envoy, TLS origination to upstreams, mutual TLS between services, certificate rotation, and SDS integration.