Learn Chi - Observability & Monitoring

#Introduction

An application that runs without visibility is an application that can't be operated. Episode 18 connects the chi router to the three pillars of observability: metrics, traces, and logs. Together they answer the questions that always come up in production — where is it slow, how many errors, and how high is the traffic. Because chi is pure net/http, the entire Go observability ecosystem attaches without adapters. We start with Prometheus, then OpenTelemetry, and close with health checks.

#Prometheus Metrics

#Metrics Middleware

Expose HTTP metrics through middleware:

go get github.com/prometheus/client_golang/prometheus/promhttp

func prometheusMetrics(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        start := time.Now()
        next.ServeHTTP(w, req)
        httpDuration.WithLabelValues(
            req.URL.Path, req.Method,
        ).Observe(time.Since(start).Seconds())
    })
}

httpDuration.WithLabelValues(path, method).Observe(...) records duration per route and method. This histogram becomes the basis for latency percentile calculations in dashboards.

#Exposing the Endpoint

r := chi.NewRouter()
r.Use(prometheusMetrics)
 
r.Get("/metrics", func(w http.ResponseWriter, req *http.Request) {
    promhttp.Handler().ServeHTTP(w, req)
})

promhttp.Handler().ServeHTTP(w, req) serves the Prometheus output on the /metrics path. Make sure this endpoint is protected or only reachable internally.

#Collecting with Prometheus

scrape_configs:
  - job_name: chi-service
    static_configs:
      - targets: ["localhost:8080"]

targets: ["localhost:8080"] tells Prometheus to pull metrics from the application on every scrape interval.

#OpenTelemetry

#Distributed Tracing

OpenTelemetry captures request traces across services:

go get go.opentelemetry.io/otel

func otelMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        ctx, span := tracer.Start(req.Context(),
            req.Method+" "+req.URL.Path)
        defer span.End()
 
        span.SetAttributes(
            attribute.String("http.method", req.Method),
            attribute.String("http.target", req.URL.Path),
        )
        next.ServeHTTP(w, req.WithContext(ctx))
    })
}

tracer.Start(req.Context(), name) opens a new span for every request. Spans carry method and path as attributes — linking latency to its cause. When a request is forwarded to another service, propagation headers are carried by http.Client, so cross-service traces join in a single trace tree on backends like Jaeger or Tempo.

#Structured Logs

#One Log Stream

Combine slog with metrics for full context:

func requestLogger(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        start := time.Now()
        next.ServeHTTP(w, req)
        slog.Info("http_request",
            "method", req.Method,
            "path", req.URL.Path,
            "latency", time.Since(start).Milliseconds(),
        )
    })
}

slog.Info("http_request", "method", req.Method, "path", req.URL.Path, "latency", ...) writes one JSON line per request. Latency in milliseconds is ready to be sent to a log aggregator like Loki. Add the trace ID as an attribute ("trace_id", traceID) so logs and OpenTelemetry spans read as one search thread when a request misbehaves.

#Health Checks with Heartbeat

#Built-in Middleware

r.Use(middleware.Heartbeat("/healthz"))

middleware.Heartbeat("/healthz") replies OK with status 200 on the /healthz path without involving other handlers. Kubernetes probes and load balancers use this endpoint to decide when to send requests.

#Readiness vs Liveness

• Liveness (/healthz): is the server alive? If it fails, the pod is restarted.
• Readiness (/readyz): is it ready to accept traffic? If it fails, traffic is paused temporarily.

For readiness, add a dependency check such as a database with pool.Ping(req.Context()) — the server isn't considered ready until the database connection is healthy, and the load balancer won't send traffic while a dependency is failing.

#Dashboards and Alerting

#Metrics to Monitor

The three main metrics always monitored:

• Latency: p95 and p99 percentiles from the duration histogram.
• Error rate: percentage of 5xx responses out of total requests.
• Throughput: requests per second (RPS).

#Alerting Rules

groups:
  - name: chi-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{code=~"5.."}[5m]) > 0.05
        labels:
          severity: critical
        annotations:
          summary: "Error rate melebihi 5 persen"

rate(http_requests_total{code=~"5.."}[5m]) > 0.05 triggers an alert when 5xx errors exceed 5 percent over 5 minutes. A Grafana dashboard displays latency and throughput trends from the same metrics.

#Conclusion

Key takeaways:

• A Prometheus metrics middleware records duration per route.
• The /metrics endpoint is served with promhttp.
• OpenTelemetry spans connect traces across services.
• slog writes structured JSON logs ready for aggregation.
• middleware.Heartbeat provides an instant health check.
• Dashboards and alerting revolve around latency, error rate, and throughput.

In the next episode 19 we polish performance: performance and troubleshooting — efficient route patterns, http.Server tuning, connection pooling, caching with Redis, and solutions for common problems such as route conflicts, panic handlers, context deadlocks, and goroutine leaks.