Learning Golang - Resilience, Graceful Shutdown & Health Checks

#Introduction

An application that stops forcibly leaves requests hanging, database connections leaking, and transactions unfinished. A good service must know how to stop gracefully and how to survive when external dependencies misbehave.

Episode 15 covers Go service resilience: graceful shutdown with context and OS signals, health checks and readiness probes for Kubernetes, and the circuit breaker and retry patterns for flaky external calls. All three determine whether your service is fit for production.

#Graceful Shutdown

#Capturing OS Signals

When Kubernetes or another platform sends a SIGTERM signal, the service should finish the requests in flight and then exit — not be terminated outright. signal.NotifyContext turns the signal into a cancelled context.

package main
 
import (
	"context"
	"log"
	"net/http"
	"os/signal"
	"syscall"
	"time"
)
 
func main() {
	srv := &http.Server{Addr: ":8080"}
 
	ctx, stop := signal.NotifyContext(context.Background(),
		syscall.SIGINT, syscall.SIGTERM)
	defer stop()
 
	go func() {
		<-ctx.Done()
		shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
		defer cancel()
		if err := srv.Shutdown(shutdownCtx); err != nil {
			log.Println("shutdown error:", err)
		}
	}()
 
	if err := srv.ListenAndServe(); err != http.ErrServerClosed {
		log.Fatal(err)
	}
}

http.Server.Shutdown stops accepting new requests, waits for active requests to finish, and then closes the listener. The 10-second timeout keeps the shutdown from hanging forever. You can simulate the signal by sending kill -TERM <pid> to the application process.

#Closing Other Resources

Graceful shutdown isn't just about the HTTP server: also close the database connection pool, the Redis client, and worker goroutines. The standard flow: the signal is received, all goroutines are notified through a context, resources are closed, and only then does the process exit.

#Health Checks and Readiness Probes

#Liveness vs. Readiness

Kubernetes uses two different probes. The liveness probe answers "is the process still alive" — failing it means the container is restarted. The readiness probe answers "is the service ready to receive traffic" — failing it removes the pod from the Service so it receives no traffic.

mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
})
 
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
	if dbHealth := pingDatabase(r.Context()); dbHealth != nil {
		http.Error(w, "database tidak siap", http.StatusServiceUnavailable)
		return
	}
	w.WriteHeader(http.StatusOK)
})

/healthz only checks the process; /readyz checks critical dependencies like the database. Never put heavy dependency checks in the liveness probe — that will trigger a restart loop when a dependency is slow.

#Configuring Probes in Kubernetes

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

#Circuit Breakers and Retry

#Why You Need Both

Retries help when a failure is temporary — a brief timeout or a dropped connection. But unlimited retries make a down service worse: requests keep piling up. A circuit breaker stops calls to a service that is currently failing, gives it time to recover, and then tries again.

#The Retry Pattern with Backoff

The github.com/cenkalti/backoff/v4 package handles retries with exponential backoff and jitter:

package main
 
import (
	"time"
 
	"github.com/cenkalti/backoff/v4"
)
 
func panggilDenganRetry(fn func() error) error {
	b := backoff.NewExponentialBackOff()
	b.MaxElapsedTime = 30 * time.Second
 
	return backoff.Retry(func() error {
		return fn()
	}, b)
}

Jitter prevents many clients from retrying at exactly the same time (the thundering herd). Also cap the total retry time so it doesn't burn the request's time budget.

#Circuit Breakers

A circuit breaker works like an electrical circuit breaker: closed (normal), open (failing continuously, reject immediately), and half-open (test one request to see whether recovery has happened). The github.com/sony/gobreaker package provides a complete implementation.

package main
 
import (
	"time"
 
	"github.com/sony/gobreaker/v2"
)
 
func main() {
	cb := gobreaker.NewCircuitBreaker[int](gobreaker.Settings{
		Name:        "api-eksternal",
		MaxRequests: 5,
		Timeout:     30 * time.Second,
	})
 
	hasil, err := cb.Execute(func() (int, error) {
		return panggilApiEksternal()
	})
	_ = hasil
	_ = err
}

MaxRequests limits requests while half-open, and Timeout determines how long the circuit stays open. Combine the circuit breaker with retry: retry a few times first, then the circuit breaker cuts off calls for a certain period.

#Closing

Episode 15 made your Go service resilient: graceful shutdown with signal.NotifyContext and http.Server.Shutdown, liveness and readiness health checks for Kubernetes, plus retries with exponential backoff and circuit breakers with gobreaker for external dependencies.

Key takeaways:

• Catch SIGTERM and shut the server down gracefully.
• Cap the shutdown time so it never hangs forever.
• Liveness means the process is alive; readiness means it's ready for traffic.
• Don't put heavy dependency checks in the liveness probe.
• Retries use backoff with jitter and a total time limit.
• Circuit breakers stop repeated calls to a failing service.

In the next episode we will discuss Go in the cloud, containers, and Kubernetes — building efficient container images with multi-stage Dockerfiles, deploying a Go application to Kubernetes, Cloud Run, or serverless platforms, plus container runtime best practices with minimal images, distroless, and static binaries.