Learn 9router - Observability at Scale

#Introduction

In episode 19 9router configuration flows through automated pipelines and every change can be rolled back quickly. But automation without vision is like driving with your eyes closed: you might go fast — until the crash. The observability introduced in episode 7 covered the basics of metrics, logs, and tracing for AI requests. Episode 20 raises it to production scale.

This episode's roadmap: first define the metrics that must be monitored, second build dashboards for routes, models, and tools, third detect anomalies on routing decisions, then design alerting that isn't noisy but hits its target.

#Metrics You Must Monitor

Good observability starts with choosing the right metrics — not recording everything. For an AI routing gateway, metrics divide into three layers.

The route layer. This is the health of the gateway's main function: request volume, success rate, fallback rate, and classifier intent confidence. A high fallback rate is an early alarm — it could mean the primary model is in trouble or rule matching is drifting off course.

The model layer. Per-provider performance: latency p50/p95/p99, token counts, cost per request, and rate limit frequency. This is where the cost and latency trade-off from episode 15 can be measured, not estimated.

The tool layer. Every tool invocation has its own metrics: call latency, error rate, and cache hit ratio. A slow tool slows the entire agentic flow even if the model is perfect.

All these metrics are exposed by 9router through the /metrics endpoint and can be checked quickly with 9router metrics check. Here are example queries to measure them in PromQL:

sum(rate(router_requests_total{route="chat-main"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(router_latency_seconds_bucket[5m])))
sum(rate(router_fallback_total[5m])) by (route)
sum(rate(router_cost_usd_total[5m])) by (model)

Get used to looking at these three metrics together: volume, error, and fallback. Volume rising is normal at peak hours; volume rising together with fallback rising is a problem.

#Route, Model, and Tool Dashboards

A dashboard is where metrics become readable. In Grafana, an effective layout for an AI gateway usually has three rows. The first row "Routing Overview": request rate per route, success rate, and fallback rate. The second row "Model Performance": latency histogram per provider and cost estimates. The third row "Tool Health": error rate and latency per tool.

Mandatory panels include:

• Request volume per route — see the traffic distribution and identify unusual routes.
• Latency p95 per provider — p95 is more honest than the average; averages hide spikes.
• Fallback rate — this metric deserves its own panel, not hidden inside another metric.
• Cost per model — prevents the monthly bill from being a surprise.

A p95 latency per provider panel can be written like this:

histogram_quantile(
  0.95,
  sum by (le, provider) (
    rate(router_latency_seconds_bucket[5m])
  )
)

Add route and provider variables to the dashboard so it can be filtered without editing queries. A good dashboard is one anyone can read — it shouldn't require one specific person to interpret the graphs.

#Detecting Anomalies on Routing Decisions

Even healthy metrics can hide dangerous anomalies. Anomalies in routing don't always take the form of errors; sometimes they're subtle shifts:

• Fallback rate silently spikes from 1 percent to 15 percent with no error alert.
• Classifier confidence drops — intents get miscategorized often, and traffic scatters to the wrong routes.
• Drastic traffic redistribution — one route loses volume within minutes with no deploy change.

Anomaly detection works by comparing the current state against a baseline. Several approaches you can use:

1. Static thresholds — simple, for example a fallback rate above 20 percent.
2. Percentage of change — comparing the last 5 minutes against the same period last week.
3. Baseline prediction — using historical data to predict the expected value, then computing the deviation.

Example query that compares against a one-week-old baseline:

(
  sum(rate(router_fallback_total[5m]))
  /
  clamp_min(sum(rate(router_requests_total[5m])), 1)
)
/
(
  sum(rate(router_fallback_total[1h] offset 1w))
  /
  clamp_min(sum(rate(router_requests_total[1h] offset 1w)), 1)
)
> 2

This query is true when the current fallback rate is twice as high as the same period last week. Techniques like this catch anomalies that slip past simple thresholds.

#Alerting for Failed and Degraded Routes

Alerting is the part easiest to over-generalize and easiest to get wrong. The main rule: every alert must be actionable, and every alert must rarely fire. Alerts that are too sensitive will be ignored — and eventually defeat their purpose.

Example of healthy alert rules:

groups:
  - name: 9router-routes
    rules:
      - alert: RouteHighErrorRate
        expr: router_success_rate < 0.95
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Error rate route tinggi dalam 5 menit"
 
      - alert: FallbackRateElevated
        expr: fallback_rate_ratio > 0.2
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "Fallback rate tinggi, periksa provider utama"
 
      - alert: DegradedModeActive
        expr: router_degraded_mode > 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Gateway masuk degraded mode"

Notice the for on each rule — an alert only fires if the condition holds for that duration, so the pager isn't burned by momentary spikes. Severity is wisely differentiated: page for conditions that need a human now (degraded mode, high error rate), ticket for conditions worth investigating but not urgent (fallback rate).

Finally, tie every alert to a runbook via an annotation. An on-call woken at midnight shouldn't have to guess the steps; that runbook is the topic of episode 21.

#Conclusion

Episode 20 completes your production eyes: layered metrics for routes, models, and tools; dashboards anyone can read; anomaly detection techniques that compare conditions to a baseline; and alerting that rarely fires but always hits its target.

Key takeaways:

• Fallback rate is the most important metric that's rarely monitored — give it a dedicated dashboard.
• Dashboards must be readable by anyone, not just their creators.
• Anomalies are often subtle shifts, not errors; compare against a baseline to catch them.
• Alerts should rarely fire and always be actionable; for prevents the pager from burning.
• Every alert must reference a runbook — on-call shouldn't guess.

In episode 21 we build the operational foundation: operational readiness and runbooks — writing routing incident procedures, setting ownership and support boundaries, and documenting route and policy standards. See you there!