Learn Gin - Observability & Monitoring

#Introduction

An unobservable application is a puzzle when an incident occurs. This episode 18 dissects observability & monitoring for a Gin application: exposing Prometheus metrics with prometheus/client_golang, tracing requests across services with OpenTelemetry, structuring logs with slog, and providing health check endpoints for orchestrators like Kubernetes.

Observability has three pillars: metrics to ask "how many and how slow", traces to ask "through which path", and logs to ask "what are the details". They complement each other. Without all three, you can only guess when production has a problem.

This episode completes the foundation from episode 11 (logging) and episode 12 (context). The health endpoints you build here will be reused when building the production architecture in episode 21.

#Prometheus Metrics with promhttp

#Installing the Prometheus Client

Prometheus pulls metrics from an HTTP endpoint that exposes the text exposition format. Install the official client first:

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

#Exposing the /metrics Endpoint

Once installed, register the promhttp handler on a Gin route:

r.GET("/metrics", gin.WrapH(promhttp.Handler()))

promhttp.Handler() serves the default metrics — Go runtime memory usage, goroutines, and garbage collector. gin.WrapH adapts a standard http.Handler so it can be installed as a Gin handler. The Prometheus server then pulls this endpoint periodically according to its scrape interval.

#Custom Metrics with Counter and Histogram

#Registering Metric Vectors

Default metrics only tell the runtime's story. To monitor application behavior, register custom metrics — a Counter for request counts and a Histogram for latency:

var (
    httpRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total request HTTP yang masuk.",
    }, []string{"method", "path", "status"})
 
    httpRequestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Durasi pemrosesan request HTTP.",
        Buckets: prometheus.DefBuckets,
    }, []string{"method", "path"})
)

promauto.NewCounterVec and promauto.NewHistogramVec register metric vectors on the global Prometheus registry. Labels like method, path, and status let metrics be broken down by dimension when queried in Grafana.

#Metric Recording Middleware

Don't record metrics one by one in handlers. Install a single middleware that records all requests:

func metricsMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
 
        c.Next()
 
        status := strconv.Itoa(c.Writer.Status())
        method := c.Request.Method
        path := c.FullPath()
        httpRequestsTotal.WithLabelValues(method, path, status).Inc()
        httpRequestDuration.WithLabelValues(method, path).Observe(time.Since(start).Seconds())
    }
}

c.FullPath() returns the route pattern like /users/:id, not the actual parameter values. This matters: labels with random parameter values would blow up cardinality and slow Prometheus down. Record Inc() and Observe() after c.Next() so the final status is captured.

#OpenTelemetry Tracing with otelgin

#Automatic Instrumentation

Metrics answer "where is it slow", traces answer "why". OpenTelemetry creates a span for every request and propagates the trace context across all services. Use the official middleware for Gin:

go get go.opentelemetry.io/otel/sdk/trace
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

tp := initTracerProvider()
otel.SetTracerProvider(tp)
 
r := gin.New()
r.Use(otelgin.Middleware("belajar-gin"))

otelgin.Middleware("belajar-gin") creates a span for every request and names the service belajar-gin on tracing backends like Jaeger or Tempo. Because the trace context flows through c.Request.Context(), database and HTTP calls inside handlers are automatically connected in one trace.

#Structured Logs with slog

#Logs Correlated with Traces

Logs add details that neither metrics nor traces can provide. Correlate logs with traces through shared attributes:

slog.Info("user created",
    "user_id", userID,
    "request_id", requestID,
    "duration_ms", duration,
)

slog.Info("user created", "user_id", userID, ...) writes structured logs that can be filtered by query. In production, direct JSON logs to an aggregator like Loki or CloudWatch rather than raw stdout. With the same request_id attribute in both logs and traces, an incident can be traced from a metric down to a log line.

#Health Checks for Kubernetes

#Liveness and Readiness Endpoints

Kubernetes distinguishes two probes: liveness indicates the process is alive, readiness indicates it's ready to accept traffic. Create both as separate endpoints:

r.GET("/healthz", func(c *gin.Context) {
    c.JSON(200, gin.H{"status": "ok"})
})
 
r.GET("/readyz", func(c *gin.Context) {
    if err := db.Ping(c.Request.Context()); err != nil {
        c.JSON(503, gin.H{"status": "not ready"})
        return
    }
    c.JSON(200, gin.H{"status": "ready"})
})

/healthz always returns 200 as long as the process is running. /readyz checks dependencies like the database; if it fails, it returns 503 and Kubernetes stops sending traffic without killing the pod. Don't put health checks behind authentication — probes don't carry credentials.

#Closing

Key takeaways:

• promhttp.Handler() at /metrics exposes Prometheus metrics.
• Counters and Histograms with bounded labels measure requests and latency.
• c.FullPath() keeps label cardinality low.
• otelgin.Middleware adds automatic spans for every request.
• slog with context attributes makes logs easy to correlate.
• /healthz and /readyz for liveness and readiness in Kubernetes.

In the next episode, episode 19, we'll dissect performance & troubleshooting — gin.SetMode ReleaseMode, reducing allocations, connection pooling and Redis caching, handling panics, and common troubleshooting like context deadlocks and routing conflicts.