Learn Multigress - Observability at Scale

#Introduction

In episode 7 we learned the basics of observability. Episode 20 raises the level: observability at scale — monitoring route performance and latency accurately, linking traces from ingress to backend, and building dashboards for traffic SLIs that teams actually use.

At a large scale, the volume of data is overwhelming. The challenge isn't collecting metrics; it's turning them into signals you can make decisions on.

#Monitoring Route Performance and Latency

#Key Gateway Metrics

A few metrics deserve to be the first rows of your dashboard: total requests, error rate by status, and latency. Latency is most useful as a histogram because it enables percentile calculations.

histogram_quantile(0.95,
  sum(rate(multigress_http_request_duration_seconds_bucket[5m])) by (le, route))

The histogram_quantile(0.95, ...) query computes the P95 of the request duration histogram per route. This number is far more useful than an average, because it reveals latency anomalies that affect a subset of users.

#Latency-Based Alerts

Worsening latency must trigger an alert, not just show up on a dashboard. Create a PrometheusRule that computes the P95 continuously.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: multigress-latency
  namespace: monitoring
spec:
  groups:
    - name: gateway-latency
      rules:
        - alert: GatewayHighP95
          expr: |
            histogram_quantile(0.95,
              sum(rate(multigress_http_request_duration_seconds_bucket[5m])) by (le, route))
            > 1
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "P95 latency tinggi pada {{ $labels.route }}"

If the P95 exceeds 1 second for 10 minutes, the GatewayHighP95 alert fires with the route label as context. This alert becomes the main ingredient for SLIs in episode 21.

#Trace Correlation Through Ingress and Backend

#Enabling Tracing in Multigress

Metrics tell you there's a problem; traces explain where. Multigress can send traces to an OTLP backend like Tempo or Jaeger.

helm upgrade multigress multigress/multigress \
  --namespace multigress-system \
  --set tracing.enabled=true \
  --set tracing.exporter=otlp \
  --set tracing.otlpEndpoint=tempo.tracing.svc.cluster.local:4317

The configuration above sends traces to Tempo over gRPC on port 4317. From here on, every request passing through the gateway can be followed all the way to the backend service.

#Propagating Trace Context

For a trace to continue to the backend, the trace context must be propagated through headers like traceparent. The gateway injects the header when a request arrives, and other services pass it along.

With proper propagation, you can see from a single request's trace: the time at the gateway, the time in the application, all the way to the time in the database. That segmentation is what shortens latency incident investigations.

#Building Dashboards for Traffic SLIs

#Grafana Dashboard Panels

A good dashboard shows SLIs directly, not a hundred raw metrics. Start with error rate and latency per route, then add details as needed.

- title: Error rate per route
  targets:
    - expr: |
        sum(rate(multigress_http_requests_total{status=~"5.."}[5m])) by (route)
        / sum(rate(multigress_http_requests_total[5m])) by (route)

The Error rate per route panel shows the proportion of 5xx status per route. One panel like this is more meaningful than twenty panels rarely looked at.

#Dashboards as a Team Agreement

The best dashboard is the one used during incidents. Include panels for traffic, errors, and latency on the same page, so an incident's timeline reads without switching tabs.

#Closing

Episode 20 built reliable observability: histogram metrics for accurate latency, traces that link gateway and backend, and dashboards that present SLIs as a team agreement.

The key takeaways:

• Histograms enable accurate latency percentile calculations.
• P95-based alerts detect degradation before users complain loudly.
• OTLP tracing follows a request from the gateway to the database.
• Trace header propagation keeps traces connected.
• A good dashboard shows SLIs, not raw metrics.
• Store dashboards as code so they can be reviewed.

In the next episode 21 we'll discuss SLOs, runbooks & operational readiness — defining SLIs for ingress and routing, writing runbooks for failures and route changes, and on-call patterns for gateway operations. The metrics and dashboards from this episode will be assembled into measurable targets.