Learn Istio - Performance Tuning & Scalability

#Introduction

A small mesh feels light. When workloads swell to hundreds or thousands, symptoms start to appear: istiod gets overwhelmed, sidecars use too much memory, and xDS configuration flows more slowly. Episode 14 covers how to measure, scale, and tune Istio so it stays responsive at large scale.

#Scaling the Control Plane: istiod

#Horizontal Scaling and HPA

istiod can be scaled horizontally, but the default configuration is a single replica. For large scale, enable a HorizontalPodAutoscaler:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: istiod
  namespace: istio-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: istiod
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

averageUtilization: 80 adds replicas when istiod CPU passes 80 percent. Do not just add replicas without data — monitor the metrics first (the last section of this episode).

#Managing xDS Churn

Istiod spends CPU mainly on xDS churn: every time something changes (a service is created, a Pod restarts, config changes), istiod has to compute and send new configuration to affected sidecars. Ways to reduce churn:

• Avoid unnecessary Deployment updates — every Pod restart produces a discovery event.
• Limit sidecar scope with the Sidecar CRD (episode 12).
• Raise the publish interval if needed via an env var on istiod, for example PILOT_PUSH_THROTTLE.

Remember the rule of thumb: the more configuration that must be recomputed, the heavier istiod becomes. Reduce the number of listeners and clusters per sidecar with scoping.

#Sidecar Resource Sizing

#Realistic Requests and Limits

Envoy consumes resources based on the number of listeners, clusters, and traffic. The default sizing is often too small or too large. Start from these numbers, then measure with real metrics:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/proxyCPU: "100m"
        sidecar.istio.io/proxyCPULimit: "2000m"
        sidecar.istio.io/proxyMemory: "128Mi"
        sidecar.istio.io/proxyMemoryLimit: "1024Mi"
    spec:
      containers:
      - name: productpage
        image: istio/examples-bookinfo-productpage-v1:1.19.0

sidecar.istio.io/proxyCPU: "100m" is an example annotation for setting the sidecar CPU request. Verify usage with:

kubectl top pod productpage-abc123 --containers
kubectl exec productpage-abc123 -c istio-proxy -- curl localhost:15000/stats | grep -E "server.memory|server.live"

kubectl top pod --containers shows real usage per container. Set requests from the median of normal usage and limits with a safe margin.

#Reduce Scope to Save Memory

Envoy's biggest memory source is the number of listeners and clusters in its configuration. A Sidecar CRD that limits egress (episode 12) can cut memory significantly in large meshes — often this is the best investment before raising limits.

#The TCP vs HTTP Path and Connection Pools

HTTP traffic is processed with L7 filters; raw TCP traffic is not. The HTTP path carries extra cost (routing, telemetry, tracing) that TCP does not. When performance is a priority and you do not need L7 features, consider TCP protocol on the Service and Gateway for those services.

The connection pool manages the queue to upstreams:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: tuning-pool
spec:
  host: productpage
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
        connectTimeout: 3s
      http:
        http2MaxRequests: 2000
        maxRequestsPerConnection: 100
        idleTimeout: 10m

idleTimeout: 10m closes idle connections to free resources. Measure first, then tune: a pool that is too small adds latency, one that is too large wastes backend resources.

#Control Plane Observability

The control plane also has to be measured. Istiod exposes metrics on port 15014:

kubectl exec -n istio-system deploy/istiod -- curl localhost:15014/metrics | grep -E "pilot_|istio_xds" | head -20

Watch metrics like pilot_proxy_convergence_time (configuration convergence time) and istio_xds_push_errors. If convergence swells, check churn and mesh size. These metrics are also used for alerting in episode 20.

#Summary

Episode 14 kept the mesh fast: HPA for istiod with xDS churn monitoring, measurement-based sidecar resource sizing, scope limiting to save memory, connection pool tuning, and control plane metrics for measuring convergence.

Key takeaways:

• HPA for istiod helps, but reduce churn first.
• xDS churn is istiod's main load; limit scope and avoid unnecessary restarts.
• Sidecar sizing must come from real measurements, not guesses.
• The sidecar.istio.io/proxy* annotations control sidecar resources.
• The Sidecar CRD is the most effective memory saver.
• HTTP carries L7 cost; choose TCP when you do not need L7 features.
• istiod metrics (convergence, push errors) determine control plane health.

In the next episode, episode 15, we will go beyond the built-in CRDs: advanced extensibility with EnvoyFilter and WASM — when to safely use EnvoyFilter, building Proxy-WASM filters for custom telemetry, and examples of a header enricher and an authz hook.