Learn Istio - Resilience Patterns (Retries, Timeouts, Circuit Breakers, Fault Injection)

#Introduction

On real networks, failures are normal: connections drop, Pods restart, backends slow down. A resilient application is not one that never fails, but one that recovers quickly when failures happen. Episode 7 covers four resilience patterns you can install without changing application code: retries, timeouts, circuit breakers, and fault injection.

#Retries and Retry Budgets

#Retries in VirtualService

Retries make Envoy repeat a failed request before giving up:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: productpage-retry
spec:
  hosts:
  - productpage
  http:
  - route:
    - destination:
        host: productpage
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: connect-failure,refused-stream,5xx

attempts: 3 allows up to three retries, perTryTimeout limits the duration of a single attempt, and retryOn determines which conditions are worth retrying — 5xx includes 500 and 503, but not errors that are not server responses.

#Retry Budgets

Retries can also be capped globally per service with a RetryBudget (available in newer Istio versions):

apiVersion: networking.istio.io/v1
kind: RetryBudget
metadata:
  name: productpage-budget
spec:
  match:
  - hosts:
    - productpage
  retryBudget:
    percent: 20
    minRetriesPerSecond: 10

retryBudget.percent: 20 means additional retries are capped at 20 percent of normal traffic — preventing retry storms when many requests fail at once, something that can make an outage worse.

#Timeouts

A timeout limits the total time a single request may take:

spec:
  hosts:
  - productpage
  http:
  - route:
    - destination:
        host: productpage
    timeout: 5s

timeout: 5s stops a request that runs longer than five seconds and returns a 504 error to the client. Combining a timeout with perTryTimeout keeps latency under control: if the backend is slow, you fail fast instead of hanging.

#Outlier Detection and Circuit Breakers

The circuit breaker in Istio is built from two parts: the connection pool and outlier detection. Both live in a DestinationRule:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: productpage-circuit
spec:
  host: productpage
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 1m
      maxEjectionPercent: 100

How it works is simple: the connection pool limits how many connections and requests may target a Pod. If a Pod starts producing 5xx errors, consecutive5xxErrors: 5 causes that Pod to be ejected from the pool for baseEjectionTime. Traffic is diverted to healthy Pods — this is the Envoy-style circuit breaker.

#Fault Injection for Chaos Testing

Fault injection is not about making a system fail, but about proving that the system can recover. The two main types: fault.delay (delays responses) and fault.abort (returns an error):

spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        x-zone:
          exact: chaos
    fault:
      delay:
        percentage:
          value: 100
        fixedDelay: 5s
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1

Requests with the x-zone: chaos header are delayed five seconds before reaching v2. This way you can observe how clients and downstream services handle high latency without damaging production.

To test total failure, use abort:

spec:
  http:
  - match:
    - headers:
        x-zone:
          exact: chaos
    fault:
      abort:
        percentage:
          value: 50
        httpStatus: 503
    route:
    - destination:
        host: reviews
        subset: v1

fault.abort.httpStatus: 503 makes 50 percent of tagged requests fail with 503 — exactly the scenario retries and circuit breakers are expected to handle. Verify the effect through metrics:

kubectl get virtualservice reviews-routing -o yaml
curl -H "x-zone: chaos" http://reviews:9080/reviews

#Summary

Episode 7 made the mesh tough: retries with retry budgets, per-request timeouts, circuit breakers from the combination of connection pools and outlier detection, and fault injection to prove the system can recover from bad scenarios.

Key takeaways:

• Retries help with transient requests, but must be balanced with a retry budget.
• Timeouts prevent hanging requests and make the system fail fast.
• The connection pool limits load; outlier detection removes sick Pods.
• Their combination is the Envoy circuit breaker.
• Fault injection is for chaos testing, not for damaging production.
• Always test with metrics and verify the real configuration in Envoy.
• Do not retry non-idempotent requests like payments.

In the next episode, episode 8, we will look at the mesh from the measurement side: observability with metrics, logs, and tracing — the telemetry v2 pipeline, Prometheus metrics, distributed tracing with Jaeger, and Kiali and Grafana dashboards.