Learn 9router - Advanced Routing Patterns

#Introduction

Episode 7 closed observability: you can now see routes, models, and providers in the form of metrics, logs, and traces. That data isn't an end goal — it's the fuel for smarter routing patterns.

Episode 8 takes you up a level to advanced routing patterns. We'll break down traffic splitting between models or tool pipelines, canary routes for gradual releases, shadowing to validate decisions without risk, A/B testing on route policies, and dynamic routing that reacts to runtime signals. After this episode, your configuration is no longer static — it can migrate itself in a controlled way.

#Traffic Splitting

Traffic splitting divides identical requests to several targets with certain percentages. This is the basic primitive for every other pattern: canary, shadowing, and A/B are all built on the ability to carve up traffic. In 9router, splitting is declared directly on the route.

routes:
  - name: chat-split
    match:
      intent: chat
    split:
      - weight: 80
        target: gpt-4o-mini
      - weight: 20
        target: claude-3-5-sonnet

The key to splitting is consistency: requests from the same user should always go to the same target so the experience doesn't feel erratic. 9router supports consistent hashing based on a key — for example user_id — so a specific user is always served by the same model while the configuration doesn't change. To review the active split weights, 9router routes show --name chat-split displays the current distribution.

#Canary Routes

Canary is a release strategy: introduce a change to a small share of traffic first, observe, then expand when it's safe. For AI gateways, a canary is usually a new model or provider tested on 5–10 percent of traffic before replacing the old target.

routes:
  - name: chat-canary
    match:
      intent: chat
    split:
      - weight: 95
        target: gpt-4o-mini
      - weight: 5
        target: gpt-4o
    canary:
      enabled: true
      promote_on:
        success_rate_above: 0.99
        p95_latency_below_ms: 2000

The canary section holds the promotion conditions: if the success rate stays above 99 percent and p95 latency below 2 seconds during the observation period, 9router can automatically raise the canary weight toward 100 percent. If the conditions fail, the weight drops back to zero — automatic rollback without a deploy.

#Shadowing

Shadowing sends a copy of a request to a shadow target without changing the original response. The caller still receives the answer from the primary target, while 9router compares the shadow target's results behind the scenes. This is the safest way to validate a new model or provider with real production traffic.

routes:
  - name: chat-shadow
    match:
      intent: chat
    target: gpt-4o-mini
    shadow:
      - target: gpt-4o
        sample: 1.0
        capture: [response, latency, tokens]

Note: the shadow target never affects users. Shadow data is used to compare answer quality, cost, and latency before a promotion decision. Remember that shadowing doubles the provider cost for the shadowed traffic, so limit the sample or the duration.

#A/B Testing on Route Policies

A/B testing brings experimentation into the policy domain, not just models. You want to know whether a new policy — say a more aggressive intent matcher — produces better decisions than the old policy. 9router lets two policy variants run side by side with deterministic per-user splitting.

experiments:
  - name: strict-vs-loose-filter
    key: user_id
    variants:
      - name: control
        weight: 50
        config_file: policies/strict.yaml
      - name: treatment
        weight: 50
        config_file: policies/loose.yaml

The results of each variant are labeled with the experiment name in metrics and logs — remember episode 7. You then compare success rate, latency, and cost between variants from the already-separated observability data. To monitor progress, 9router experiments status shows the request count and summary metrics of each variant. Once one variant proves superior, promote it to the default policy and end the experiment.

#Dynamic Routing Based on Runtime Signals

The most dynamic pattern: routing that changes with runtime conditions, not static configuration. 9router supports decisions based on signals computed at runtime, such as provider queue load, recently observed latency, token cost, or provider health status. This lets the gateway pick the best path for every moment.

routes:
  - name: chat-dynamic
    match:
      intent: chat
    dynamic:
      - target: openai/gpt-4o-mini
        when: provider_healthy(openai) and p95_latency(openai) < 1500
      - target: anthropic/claude-3-5-sonnet
        when: not provider_healthy(openai)
      - target: azure/openai-eu
        when: region == "eu"

When the OpenAI provider is healthy and fast, traffic flows there; when the provider goes down, the route shifts to Anthropic; and requests from Europe are always directed to the Azure instance in the EU region. Evaluation happens per request using real-time signals, so the gateway can respond to disruptions in seconds, not minutes.

#Conclusion

Advanced routing patterns transform 9router from a static request forwarder into a system that learns and adapts. Traffic splitting is the foundation, canary provides gradual releases with automatic promotion, shadowing validates without risk, A/B testing makes policy decisions data-driven, and dynamic routing responds directly to runtime conditions. Combining these patterns makes your gateway more agile without losing control.

Key takeaways:

• Traffic splitting is the basic primitive; key it by user identity if you want a consistent experience.
• Canary routes promote changes gradually and can auto-rollback when target conditions fail.
• Shadowing validates new models with production traffic without changing user responses.
• A/B testing on policies requires experiment labels in metrics and logs for a clean comparison.
• Dynamic routing evaluates real-time signals per request; decision quality follows signal quality.

In episode 9 we deal with many customers at once: Multi-tenant & Customer-specific Routing — tenant-aware route rules, per-customer model preferences, isolation of sensitive workloads, up to varying SLAs. See you there!