Learn 9router - Performance Optimization

#Introduction

So far your gateway is secure (episodes 12–13) and accountable (episode 14). But security and audit mean nothing if every request takes 3 seconds when the model only needs 300 milliseconds. In an AI gateway, latency isn't just about comfort — it's service quality, cost per token, and user experience.

Episode 15 polishes performance on three fronts: routing latency and throughput optimization, the cost/latency trade-off when choosing models, and response caching and embedding reuse. The goal is clear: faster requests, cheaper bills, and providers not called twice for the same thing.

#Measuring Latency and Throughput

Before optimizing, you must be able to measure. Latency in an AI gateway breaks down into components you can optimize one by one: gateway handling time (intent detection, policy evaluation, model selection), queue time to the provider, and model response time.

metrics:
  enabled: true
  latencies:
    - name: gateway_overhead
    - name: provider_ttft
    - name: provider_total
    - name: total_request
  throughput:
    requests_per_second: true
    tokens_per_second: true
  percentiles: [p50, p95, p99]

Note two important measures: TTFT (time to first token) determines when the user starts seeing the answer flow, and throughput is measured per second in both requests and tokens. Mis-focused optimization — chasing the average while forgetting p95/p99 — leaves some users with a bad experience. Enable measurement with 9router metrics enable before changing anything.

9router metrics latency --percentile p95 --window 1h
9router metrics throughput --window 1h

#Optimizing Routing Overhead and Throughput

The easiest layer to optimize is the gateway's own overhead. Every guardrail and policy evaluation takes time; too many layers running heavy classification add tens of milliseconds before the request reaches a provider. Prioritize cheap guards on the hot path, and move heavy ones to async paths when possible.

runtime:
  max_connections_per_provider: 128
  connection_idle_timeout: 30s
  request_timeout: 60s
  stream_first_token_timeout: 5s
  http2: true

Connection pooling is the next big win. Opening a TLS connection to a provider for every request is expensive; with a pool, connections are reused. http2: true lets many streams share one connection, and stream_first_token_timeout ensures a provider stuck mid-stream doesn't hang your user.

#Model Selection: Cost and Latency Trade-off

Models aren't free goods. gpt-4o is smarter but more expensive and slower; gpt-4o-mini is cheap and fast but less capable. The art of routing is placing requests on the right model — strong enough for the task, as small as possible in cost. 9router provides model assessment based on the request profile:

routes:
  - name: chat-general
    match:
      intent: general_chat
      complexity: low
    model: gpt-4o-mini
    provider: openai-prod
  - name: chat-complex
    match:
      intent: general_chat
      complexity: high
    model: gpt-4o
    provider: openai-prod
    budget:
      max_tokens: 2048
      max_cost_per_request: 0.05

The chat-complex route sets budget as a safety net: if the selected model exceeds the token or cost limit, the gateway can switch to a smaller model or reject with a clear message. The cost/latency trade-off thus becomes a policy, not an ad hoc decision.

9router models compare --route chat-general
9router route test --prompt "Jelaskan apa itu routing" --all

The models compare command shows the estimated cost and latency of each candidate model so you can choose with data, not guesses.

#Response Caching

Many AI requests repeat: FAQs, templates, nearly identical answers. Calling the provider for a question already answered yesterday is a waste of cost and latency. Response caching stores model responses and serves them directly from cache when the question is identical.

cache:
  response_cache:
    enabled: true
    ttl: 3600s
    max_size: 512MB
    include_metadata:
      - model
      - temperature
    exempt_routes: [chat-complex]

The cache key is formed from the prompt content plus important metadata (model, temperature), so an answer for gpt-4o doesn't leak into gpt-4o-mini. Routes like chat-complex are exempt because their answers are personal and change too often. A cache hit isn't just cost savings — it slashes total latency dramatically because there's no trip to the provider at all.

{
  "event": "cache_hit",
  "request_id": "req_88bc12",
  "cache_key_hash": "3f9a21e7",
  "ttl_remaining_s": 1840
}

#Semantic Cache and Embedding Reuse

The response cache is still rigid: a question with the same meaning but different wording won't hit the cache. The solution is the semantic cache — comparing the embedding similarity of a new question against already-answered questions, then reusing the matching answer when the similarity is high enough.

cache:
  semantic_cache:
    enabled: true
    embedding_model: text-embedding-3-small
    similarity_threshold: 0.97
    store: redis
    ttl: 7200s
embeddings:
  reuse: true
  store: redis
  namespace: embeddings-app

Embeddings for the same question are often computed repeatedly by many features. embeddings.reuse ensures an embedding is computed once then reused — whether by the semantic cache, RAG retrieval, or intent classification. In episode 5 you met routing to embedding models; now those embeddings become an asset that's cached and shared across routes.

9router cache hit-rate
9router cache invalidate --route chat-general

Be careful with invalidate: it's clearly needed after content changes, but don't use it excessively because it throws away all cache value. Monitor the hit-rate and adjust TTL per route — a cache that almost never hits just adds overhead, not savings.

#Conclusion

Episode 15 makes your gateway measurably faster and cheaper: latency and throughput metrics focused on p95/p99, routing overhead and connection pooling that cut wait times, model tiering with cost/latency budgets as policy, and response cache plus semantic cache and embedding reuse that stop repeated provider calls.

Key takeaways:

• Measure first with layered metrics (gateway overhead, TTFT, total) and watch p95/p99, not just averages.
• Connection pooling, HTTP/2, and proper timeouts have more impact than micro-optimizations elsewhere.
• Put the cost/latency trade-off in routing configuration, not in guesses — small models suffice for simple tasks.
• Response cache handles identical questions; semantic cache handles questions with the same meaning.
• Cache and reuse embeddings to avoid recomputation, but monitor hit-rate so the cache doesn't become a burden.

A fast, cheap gateway is now in your hands. In episode 16 we widen its logic: Route Extensions & Custom Actions — custom routing hooks, plugin-based decision logic, domain-specific actions, and reusable route modules. See you there!