Learn Multigress - SLOs, Runbooks & Operational Readiness

#Introduction

A gateway that works without a clear target is hard to evaluate. Episode 21 covers SLOs, runbooks, and operational readiness: defining SLIs for ingress and routing, writing runbooks to handle failures and route changes, and building a healthy on-call pattern.

All the metrics from episodes 7 and 20 are now assembled into measurable service promises understood by the whole team.

#Defining SLIs for Ingress and Routing

#Choosing Availability and Latency SLIs

An SLI starts from a simple question: how does the team know the service is doing well or badly? For a gateway, the answer is usually two numbers: availability and latency.

sum(rate(multigress_http_requests_total{status!~"5.."}[28d]))
/ sum(rate(multigress_http_requests_total[28d]))

The query above computes the proportion of requests that didn't return a 5xx status over 28 days. status!~"5.." excludes all server errors from the numerator.

#Error Budget and SLO Targets

An SLO sets the target number for an SLI, and the error budget is the remainder you can afford to lose. For a 99.9 percent monthly availability SLO, the error budget is only about 43 minutes. Monitor the burn rate with alerts so the team knows when the budget is eroding fast.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: multigress-slo
  namespace: monitoring
spec:
  groups:
    - name: gateway-slo
      rules:
        - alert: ErrorBudgetBurn
          expr: |
            (1 - (sum(rate(multigress_http_requests_total{status=~"5.."}[1h]))
              / sum(rate(multigress_http_requests_total[1h]))))
            < 0.99
          for: 15m
          labels:
            severity: page

The ErrorBudgetBurn alert fires when one-hour availability drops below 99 percent for 15 minutes. A high burn rate means the monthly budget is eroding quickly and must be handled immediately.

#Runbooks for Failures and Route Changes

#Runbook Structure

A runbook removes guessing during an incident. A consistent structure makes it usable by anyone, even in the middle of the night:

• A summary of the condition and visible symptoms.
• Diagnosis steps to confirm the cause.
• Mitigation steps to restore the service.
• Rollback procedures to restore configuration.
• A postmortem to record the root cause.

#Example Runbook: Sudden Rate Limit

One common incident is a sudden spike of 429s. A runbook for this case contains two short diagnosis steps.

kubectl get backendtrafficpolicy -A
kubectl logs -n multigress-system -l app=multigress-gateway --tail=200

The kubectl get backendtrafficpolicy -A command checks the active policies, then the recent logs hint at whether the 429s come from the rate limit or from the application. The runbook closes with a mitigation step: temporarily raising rps or adding replicas.

#On-Call Patterns for Gateway Operations

#Rotation, Handover, and Escalation

A healthy on-call has clear rotation and written handovers. Before a shift ends, the operator records what's in progress, what looks suspicious, and what the next shift should check.

#Alert Routing by Severity

Not every alert should wake someone up. Separate critical alerts that page the on-call from warnings that just go to a chat channel.

routes:
  - matchers: ["severity=critical"]
    receiver: pagerduty-oncall
  - matchers: ["severity=warning"]
    receiver: slack-gateway

The severity=critical matcher routes to PagerDuty, while severity=warning is enough for Slack. Disciplined routing keeps on-call from burning out on noise.

#Closing

Episode 21 made gateway operations measurable and learnable: SLIs define health, SLOs set targets with error budgets, runbooks prepare incident steps, and on-call keeps responses healthy.

The key takeaways:

• An SLI is a metric; an SLO is an agreed target.
• An error budget sets how long a service is allowed to miss its target.
• Burn rate alerts warn when the budget erodes quickly.
• Runbooks turn incidents from guessing into procedure.
• A healthy on-call needs clear rotation, handover, and escalation.
• Alert routing separates what pages people from what merely logs.

In the next episode 22, the final episode, we'll discuss production hardening & best practices — security hardening, traffic governance and configuration hygiene, and upgrade paths with compatibility checks. All the series lessons will be summarized into a production checklist.