Learn 9router - Recovery & Operational Resilience

#Introduction

In episode 17 you learned how 9router places routes across many regions and provides provider redundancy to serve global users. But redundancy is only half the story. The more important part is what happens when a provider truly falls — does your gateway rebuild itself, or does it fall along with the provider.

Episode 18 covers recovery and operational resilience. The roadmap has three layers: failover strategies for provider disruptions, backup routes and degraded mode behavior, then structured incident response when routing fails. The end goal is simple — the gateway keeps answering even when your favorite provider isn't having its best day, and you don't panic when that happens.

#Failover Strategies for Provider Disruptions

Provider disruptions don't always come as a total outage. Sometimes it's a string of 5xx errors, sometimes prolonged timeouts, sometimes a flood of rate limits. Automatic failover is the first line of defense: detect the failure, switch the target, then retry the request against a backup provider.

9router's fallback configuration is deterministic — the order executes in declaration order:

routes:
  - name: chat-main
    match:
      intent: general_chat
    target:
      model: gpt-4o
      provider: openai-prod
      fallback:
        - model: claude-3-5-sonnet
          provider: anthropic-prod
        - model: gpt-4o-mini
          provider: azure-openai-prod

If gpt-4o fails, the request moves to claude-3-5-sonnet, then to gpt-4o-mini. One thing to note: fallback isn't for every error. 4xx errors like an invalid API key or a rejected prompt won't trigger failover — the request is genuinely wrong and will never succeed on another provider. Failover is only for transient errors: 5xx, timeouts, and rate limits.

To monitor each route's condition directly, use the status command 9router status routes:

9router status routes
9router status route chat-main --show-latency

The output shows provider health scores, the number of failover requests, and the last error. That's a quick picture for deciding whether manual intervention is needed.

#Backup Routes: Route-level Fallback

The failover above works within one route. A backup route is the higher level: the entire route definition — including match and policy — has its own twin. This matters when the failure isn't a single model, but an entire routing path, for example a misapplied rate limit policy or an erroring intent classifier.

routes:
  - name: chat-primary
    match:
      intent: general_chat
    target:
      model: gpt-4o
      provider: openai-prod
    policy:
      rate_limit: 1000
 
  - name: chat-backup
    match:
      intent: general_chat
    target:
      model: claude-3-5-haiku
      provider: anthropic-prod
    policy:
      rate_limit: 200

With this pattern, the backup route carries its own complete configuration. The backup can be given a smaller capacity limit because it usually shares provider quota with other routes. When chat-primary enters an error state, 9router directs traffic to chat-backup until the primary route recovers — and users don't notice that the provider changed.

#Degraded Mode: Serving a Little Rather Than Not at All

Sometimes no backup is strong enough to carry the whole load. That's where degraded mode comes in: the gateway keeps serving, but with deliberately reduced capacity. The principle is that availability matters more than quality — a shorter response is still better than no response at all.

routes:
  - name: chat-primary
    match:
      intent: general_chat
    target:
      model: gpt-4o
      provider: openai-prod
    degraded:
      enabled: true
      fallback_model: gpt-4o-mini
      max_tokens: 256
      cache_ttl: 10m
      response_on_failure: "Layanan sedang sibuk, silakan coba beberapa saat lagi."

When this mode is active, 9router swaps in a cheaper, faster model, cuts max_tokens to shorten response time, and enables a cache with a 10-minute TTL so identical requests don't need to call the provider. Only if all paths are exhausted — provider down and cache empty — is the substitute response sent. The mode status can be checked anytime with 9router status degraded. Degraded mode isn't a design failure; it's a conscious choice to pick lower quality over a totally dead system.

#Circuit Breakers: Stopping Failure Amplification

Imagine a provider goes down, and millions of requests keep calling it and waiting on timeouts. That's not just wasteful — it's failure amplification that can take down other healthy parts of the system. A circuit breaker prevents this by cutting the path temporarily once a failure threshold is exceeded.

circuit_breaker:
  enabled: true
  failure_threshold: 5
  recovery_timeout: 30s
  half_open_max: 1

A circuit breaker works like a three-position switch:

• Closed: normal. Every failure increments a counter.
• Open: after failure_threshold is reached, the path closes. All requests are immediately diverted to the fallback without waiting on timeouts, so no request hangs.
• Half-open: after recovery_timeout passes, one test request is sent. If it succeeds, the switch returns to closed. If it fails, it returns to open.

With half_open_max: 1, only one trial request is allowed in — preventing an immediate flood of requests when a provider is just starting to recover.

#Incident Response for Routing Failures

Automation handles a lot, but not everything. Incidents still need humans, and what separates mature teams is how they handle them. The right sequence:

1. Detection — an alert tells you the fallback rate has spiked. This alerting is what we'll build in episode 20.
2. Triage — determine whether this is a routing incident (wrong 9router configuration) or a provider incident (the vendor is down).
3. Mitigation — if the provider is down, disable the primary route so retries don't pile up, then let the fallback carry the load.
4. Communication — inform internal stakeholders and, if necessary, users.
5. Postmortem — document the root cause and prevention steps so it doesn't recur.

For manual mitigation, 9router provides explicit commands that leave a trail:

9router route disable chat-primary --reason "provider outage"
9router route enable chat-primary --after 15m
9router incident report 42 --severity major

--reason and --after ensure manual actions are recorded in the audit and the route isn't forgotten to be re-enabled. Incidents are numbered so they can be tracked, linked to alerts, and reviewed in postmortems.

#Conclusion

Episode 18 arms your gateway against failures: deterministic failover between providers, backup routes with their own capacity, degraded mode that preserves availability, circuit breakers that stop failure amplification, and structured incident response.

Key takeaways:

• Failover must have a deterministic fallback order and only trigger on transient errors, not 4xx.
• A backup route is a route-level fallback with its own policy and capacity — not just a model swap.
• Degraded mode chooses to serve less rather than not serve at all.
• Circuit breakers prevent a down provider from piling up retries and taking down healthy systems.
• Documented incident response turns chaos into executable steps.

In episode 19 we shift from handling failures to preventing them from reaching production: CI/CD and deployment automation for 9router configuration — validation in the pipeline, automated deployment of route changes, and staged rollout of new policies. See you there!