Learning Caddy - Failover & Circuit Breaking

#Introduction

Backends will fail. Servers restart, processes hang, memory fills up — it's part of production life. The question isn't whether failure happens, but how your system responds to it. Episode 16 covers failover and circuit breaking mechanisms in Caddy.

You'll learn the two types of health checks (passive and active), safe retry policies, and the circuit breaker pattern that prevents requests from overloading a sick backend. Finally, we'll look at how to structure high availability with multiple Caddy instances.

The end goal: when one component dies, users feel nothing.

#Passive Health Checks

#Detecting Failure from Requests

A passive health check doesn't probe backends. Instead, Caddy judges from actual request responses:

app.example.com {
    reverse_proxy {
        to localhost:8080 localhost:8081 localhost:8082
        max_fails 2
        fail_duration 15s
    }
}

• max_fails 2 — two consecutive failures mark a backend unhealthy.
• fail_duration 15s — the window in which failures are counted.

A failed backend is removed from the pool automatically and retried after a while. This is a lightweight first line of defense.

#Automatic Recovery

A removed backend is re-added after a certain period. Caddy retests that backend with real requests — if it succeeds, the backend returns to the pool. No manual intervention needed.

#Active Health Checks

#Background Probing

Active health checks probe backends periodically, even without user requests:

app.example.com {
    reverse_proxy {
        to localhost:8080 localhost:8081
        health_uri /healthz
        health_interval 10s
        health_timeout 5s
        health_fails 3
        health_passes 2
        health_expected_status 200
    }
}

Full options:

• health_uri /healthz — the endpoint that gets probed.
• health_interval 10s — the probe interval.
• health_timeout 5s — the probe response time limit.
• health_fails 3 — three consecutive failed probes = unhealthy.
• health_passes 2 — two consecutive successful probes = healthy again.
• health_expected_status 200 — the status code considered a success.

health_expected_status 200 ensures a probe is considered successful only if the backend returns 200 — not merely that a connection opened.

#Custom Health Endpoints

A good health endpoint should reflect the health of the application, not just the process. An application can be alive but unable to process requests (for example, when the database is down). Make sure /healthz in your application checks important dependencies.

#Retry Policies

#When Retries Are Safe

Caddy can retry a request against another backend when the first backend fails:

app.example.com {
    reverse_proxy {
        to localhost:8080 localhost:8081
        try_duration 10s
        try_interval 250ms
    }
}

try_duration 10s gives Caddy 10 seconds to try other backends; try_interval 250ms sets the pause between attempts.

#Idempotent Methods Only

Automatic retries should be limited to idempotent requests — ones safe to repeat without double effects:

• Safe: GET, HEAD, PUT, DELETE.
• Dangerous: POST — could create duplicate transactions.

By default Caddy retries requests it considers idempotent or those already partially sent. For sensitive POSTs, it's better to let them fail than create duplicates.

#The Circuit Breaker Pattern

#Open, Closed, and Half-Open States

A circuit breaker protects a sick backend from a flood of requests:

• Closed: requests flow normally.
• Open: after failures pass the threshold, all requests are rejected immediately.
• Half-open: after a recovery period, one test request is allowed; if it succeeds, it returns to closed.

In Caddy, this behavior emerges from a combination of health checks and failure limits:

app.example.com {
    reverse_proxy {
        to localhost:8080 localhost:8081
        max_fails 1
        fail_duration 10s
        try_duration 5s
    }
}

A single failure immediately marks the backend, requests are routed to another backend, and the failed backend gets time to recover — the essence of circuit breaking without extra plugins.

#Fallback Responses

When all backends fail, give a clear response:

app.example.com {
    reverse_proxy {
        to localhost:8080 localhost:8081
    }
    handle_errors {
        @down {
            expression {http_error.status_code} >= 502
        }
        handle @down {
            respond "Service is under maintenance" 503
        }
    }
}

expression {http_error.status_code} >= 502 catches gateway errors and shows a maintenance page — users get a clear message, not a blank screen.

#High Availability Setup

#Multiple Caddy Instances

A single Caddy instance is a single point of failure. For high availability:

• Run two or more Caddy instances on different servers.
• Put an external load balancer (haproxy, cloud LB, or DNS) in front.
• Share certificate storage so all instances use the same certificates.

Combined with keepalived + VRRP for a virtual IP, or DNS-based failover, traffic keeps flowing when one instance dies.

#Shared Storage

ACME certificates must be shared across instances. Without shared storage, each instance requests its own certificate and renewals can clash. Options:

• Shared filesystem (NFS) — the same storage location.
• Distributed storage (Redis, S3) via a plugin.
• One instance manages issuance; the others read the results.

#Conclusion

Episode 16 built service resilience: passive health checks with max_fails, active health checks with health_uri and fail/success thresholds, safe retry policies for idempotent methods, the circuit breaker pattern, and high availability with multiple instances and shared storage.

Key takeaways:

• Passive health checks judge from real requests; active ones probe periodically.
• health_expected_status 200 ensures probes assess application health.
• Retries are safe for idempotent methods; be careful with POST.
• A circuit breaker prevents a sick backend from being flooded.
• Prepare a fallback response when all backends are down.
• HA needs multiple instances, a load balancer, and shared storage.

In the next episode, episode 17, we move into basic authentication — the basicauth directive, hashing passwords with caddy hash-password, protecting an entire site or specific paths, multiple users, and use cases like admin panels and staging environments. Basic security awaits.