Learn gRPC - Resilience, Retry, and Fault Tolerance

#Introduction

Failure is a certainty — servers crash, networks drop, dependencies slow down. The question isn't "will it fail", but "will the system keep serving when it does". Episode 15 builds gRPC resilience: retries for transient errors, circuit breakers for repeated failures, timeouts and failover for slow dependencies, and graceful shutdown so version rollouts don't cut off in-flight requests.

The principle: a resilient system isn't one that never fails, but one that fails gracefully and recovers quickly.

#Retry Policy via Service Config

#Standard Retry Configuration

gRPC has built-in retries configured through a service config JSON, not code. Remember the config from episode 7 — now we use it in full:

{
  "methodConfig": [
    {
      "name": [{ "service": "catalog.v1.CatalogService" }],
      "retryPolicy": {
        "maxAttempts": 4,
        "initialBackoff": "0.1s",
        "maxBackoff": "2s",
        "backoffMultiplier": 2.0,
        "retryableStatusCodes": ["UNAVAILABLE", "RESOURCE_EXHAUSTED"]
      }
    }
  ]
}

"retryableStatusCodes": ["UNAVAILABLE"] determines which errors are safe to retry. Never include INVALID_ARGUMENT — retrying a logically wrong request only wastes resources. Exponential backoff (0.1s, 0.2s, 0.4s) gives dependencies time to recover.

#Applying It on the Client

conn, _ := grpc.NewClient("localhost:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithDefaultServiceConfig(retryConfig),
)

grpc.WithDefaultServiceConfig(retryConfig) ships the retry policy along with the channel. Retries are transparent to application code — gRPC handles the attempts.

#Circuit Breaker

#Three-State Logic

Retries add load while a service is dying — making things worse. A circuit breaker prevents that: three states — closed (normal), open (tripped), and half-open (trial). When the error rate crosses the threshold, the breaker opens and all calls are rejected fast with UNAVAILABLE without touching the server; after a pause, half-open tests a single call before closing again.

if breaker.IsOpen() {
    return nil, status.Error(codes.Unavailable, "circuit terbuka")
}
err := callBackend()
breaker.Record(err)

breaker.Record(err) updates the error statistics. Production-ready implementations are available in libraries like sony/gobreaker or hystrix-go, plus native circuit breakers in service meshes.

#Timeout and Failover Strategies

#Layered Timeouts

Don't rely on a single deadline. Apply layered timeouts — short for internal calls, longer for aggregation:

ctx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
defer cancel()
res, err := client.GetProduct(ctx, req)
if status.Code(err) == codes.DeadlineExceeded {
    return nil, fallbackFromCache(ctx)
}

context.WithTimeout(ctx, 500*time.Millisecond) gives each dependency at most half a second. A fallback pattern — to a cache, for instance — keeps the user experience good while the primary source is slow.

#Failover with Multi-Target

A client can try backup addresses when the primary target fails. The gRPC resolver supports comma-separated address lists:

targets := "catalog-svc:50051,catalog-backup:50051"
conn, _ := grpc.NewClient(targets,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithDefaultServiceConfig(`{"loadBalancingConfig":[{"pick_first":{}}]}`),
)

With pick_first, the client uses the first healthy address from the list. If the primary dies, the connection moves to the backup — the basis of high availability without an intermediary.

#Graceful Shutdown and Probes

#Graceful Shutdown

When a container is stopped, in-flight requests must finish rather than be cut off forcibly. In Go, GracefulStop handles this:

quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
<-quit
 
s.GracefulStop()

signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT) captures the shutdown signals from Kubernetes. s.GracefulStop() stops accepting new requests but completes the ones in progress — then s.Stop() can be called as a safety net for timeouts.

#Health Checks and Readiness Probes

Combine with episode 9 so the orchestrator knows when it's safe to route traffic:

readinessProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:50051"]
  initialDelaySeconds: 5
livenessProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:50051"]
  periodSeconds: 10

The readinessProbe holds traffic while the server is still preparing; livenessProbe requests a restart when the server freezes. During graceful shutdown, readiness should be turned off so the pod doesn't receive new traffic.

#Closing

Key takeaways:

• Retry policy in service config handles transient errors with exponential backoff.
• Retry only for statuses like UNAVAILABLE, never for INVALID_ARGUMENT.
• Circuit breakers prevent retries from worsening cascading failures.
• Layered timeouts with fallbacks keep the user experience intact when dependencies are slow.
• Multi-target resolvers give automatic failover to backup addresses.
• Graceful shutdown plus readiness and liveness probes deliver near-zero downtime.

In episode 16 next, we cover gRPC-Web, Envoy, and gateway integrations — the gRPC-Web concept for browser interoperability, integration with the Envoy proxy and API gateways, and combining gRPC with a REST gateway for hybrid APIs. The resilient service is now opened up to consumers outside the internal network.