Learning Rust - Resilience, Graceful Shutdown, and Health Checks

#Introduction

A healthy service is not just one that runs fast — it is one that can fail gracefully. Episode 15 discusses resilience: shutting down the server without cutting off in-flight requests, telling the orchestrator about its health, and absorbing upstream failures with retry, timeout, and circuit breaker.

These concepts are increasingly important because almost every modern deployment runs on Kubernetes or a cloud platform that stops instances at any moment. Resilience is the difference between a smooth restart and an incident.

#Graceful Shutdown

#Why Shut Down Cleanly

When an instance is stopped, in-flight requests should be completed, database connections closed, and background tasks given time to finish their work. Graceful shutdown gives the application that window before the process actually ends.

#Signal Handling with Tokio

Tokio provides tokio::signal::ctrl_c and an API for Unix signals. Combine it with axum::serve, which supports graceful shutdown:

cat > src/main.rs <<'EOF'
use axum::{routing::get, Router};
use std::time::Duration;
 
#[tokio::main]
async fn main() {
    let app = Router::new().route("/", get(|| async { "resilien" }));
 
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000")
        .await
        .unwrap();
 
    println!("menunggu sinyal shutdown...");
    axum::serve(listener, app)
        .with_graceful_shutdown(shutdown_signal())
        .await
        .unwrap();
}
 
async fn shutdown_signal() {
    tokio::signal::ctrl_c().await.unwrap();
    println!("menerima sinyal, menutup dengan bersih");
    tokio::time::sleep(Duration::from_millis(500)).await;
}
EOF
cargo run

with_graceful_shutdown waits for all active connections to finish before closing the listener. shutdown_signal waits for Ctrl-C. In the cloud, the SIGTERM signal from the orchestrator triggers the same flow — the platform grants a grace window and then force-kills.

#Background Tasks Shut Down Too

Do not forget background tasks: use JoinSet or TaskTracker (episode 12) and cap the overall wait time. General rule: aim for the shutdown to finish within the Kubernetes grace window (usually 30 seconds) so no request is cut off.

#Health Checks and Readiness Probes

#Liveness vs Readiness

Kubernetes uses two probes: liveness determines whether the pod should be restarted (is the application still alive), while readiness determines whether the pod receives traffic (is the application ready to serve). They are often two different HTTP endpoints.

cat > src/main.rs <<'EOF'
use axum::{routing::get, Json, Router};
use serde::Serialize;
 
#[derive(Serialize)]
struct Health {
    status: &'static str,
}
 
#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/healthz", get(|| async { Json(Health { status: "ok" }) }))
        .route("/readyz", get(|| async { Json(Health { status: "ok" }) }));
 
    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080")
        .await
        .unwrap();
    axum::serve(listener, app).await.unwrap();
}
EOF
cargo run

/healthz returns 200 as long as the process is alive. /readyz returns 200 only if the dependencies are ready — for example, the database can be connected. When a dependency is down, readyz returns an error status so Kubernetes stops traffic while liveness stays on.

#Declaring Probes in Kubernetes

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 15

readinessProbe stops traffic when /readyz fails, while livenessProbe restarts the pod when /healthz fails. This configuration is written in the deployment spec. kubectl apply applies the changes; kubectl get pods shows the readiness status.

#Retry, Timeout, and Circuit Breaker

#Timeout with Tower

Timeout protects you from slow upstreams: requests are cancelled after a deadline. tower-http provides the layer (cargo add tower-http --features timeout):

use axum::{routing::get, Router};
use std::time::Duration;
use tower_http::timeout::TimeoutLayer;
 
#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/", get(|| async { "cepat" }))
        .layer(TimeoutLayer::new(Duration::from_secs(10)));
 
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000")
        .await
        .unwrap();
    axum::serve(listener, app).await.unwrap();
}
EOF
cargo run

TimeoutLayer::new(Duration::from_secs(10)) limits the time of every request. Without a timeout, one slow upstream can hang many connections and cascade across the entire service.

#Retry for Transient Failures

Retry repeats calls that failed for temporary reasons: timeouts, 5xx, or dropped connections. The tower ecosystem provides tower::retry and tower::reconnect. For HTTP calls, crates such as reqwest-middleware and reqwest-retry add a retry policy with jitter.

#Circuit Breaker

A circuit breaker prevents requests from being forwarded to an upstream that is down: when failures cross a threshold, the circuit "opens" and requests are rejected directly for a cooldown period — giving the upstream time to recover. tower::limit and crates such as failsafe or boomerang (cargo add failsafe) provide implementations:

use std::time::Duration;
use failsafe::{CircuitBreaker, Config};
 
fn main() {
    let cb = Config::new()
        .with_failure_rate_threshold(0.5)
        .with_volume_threshold(10)
        .build();
 
    match cb.call(|| Err::<(), ()>(())) {
        Ok(()) => println!("sirkuit tertutup, panggilan ok"),
        Err(failsafe::Error::Failure(_)) => println!("panggilan gagal"),
        Err(failsafe::Error::Open) => println!("sirkuit terbuka, tolak request"),
    }
}
EOF
cargo run

with_failure_rate_threshold(0.5) opens the circuit when more than half of the calls fail. The combination of timeout, retry, and circuit breaker is a layered defense: timeout bounds the duration, retry absorbs temporary failures, and the circuit breaker stops futile calls when the upstream is truly down.

#Closing

Key takeaways:

• Graceful shutdown completes in-flight requests before the process ends.
• axum::serve with with_graceful_shutdown closes the server cleanly.
• Liveness determines restarts; readiness determines traffic acceptance.
• readyz and healthz are the probe conventions in the Kubernetes ecosystem.
• Timeout bounds duration; retry absorbs transient failures.
• A circuit breaker stops calls to an upstream that is down.

In the next episode 16 we will discuss Rust in the cloud, containers, and WebAssembly — building static binaries with musl and distroless images, deploying to Kubernetes, Cloud Run, and serverless, as well as using WebAssembly with wasm-pack, wasm-bindgen, and yew. Your work starts to spread everywhere.