Learning Golang - Observability, Logging & Tracing

#Introduction

When an application runs in production, you can't attach a debugger. The only way to understand its behavior is observability: logs, metrics, and traces you can query. Go has a mature observability ecosystem that integrates with its official toolchain.

Episode 14 covers three pillars: structured logging with zap, logrus, or zerolog; distributed tracing with OpenTelemetry; and structured metrics with the Prometheus client. By the end of the episode, you'll be able to answer the question "what is happening with my service" quantitatively.

#Structured Logging

#Why Not fmt.Println

Plain text logs are hard for machines to analyze. Structured logs store key-value pairs so they can be filtered and aggregated by Elasticsearch, Loki, or CloudWatch. zerolog is a popular choice because it is extremely fast and oriented toward zero allocation.

go get github.com/rs/zerolog/log

package main
 
import "github.com/rs/zerolog/log"
 
func main() {
	log.Info().
		Str("service", "api").
		Int("port", 8080).
		Msg("server mulai berjalan")
}

The output is JSON: every field becomes a searchable key. zap from Uber and logrus offer similar features with a different API style. Consistency with one library across the whole team matters more than the choice of library itself.

#Log Levels

Set the level through an environment variable: debug for development, info in staging, and warn in production. Avoid overly detailed debug logs in production because of the disk and CPU cost.

log.Logger = log.With().Timestamp().Logger()
log.Warn().Str("event", "retry").Int("attempt", 2).Msg("coba ulang")

#Distributed Tracing with OpenTelemetry

#The Concepts of Trace and Span

In a microservices architecture, one request passes through many services. A trace connects all of those operations, and a span is a single unit of work within it. OpenTelemetry is the open standard for producing and collecting this telemetry.

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp

#Instrumenting HTTP Handlers

An example of HTTP instrumentation with OpenTelemetry middleware:

package main
 
import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/trace"
)
 
func main() {
	tracer := otel.Tracer("service-api")
 
	_ = trace.SpanFromContext
	_, span := tracer.Start(ctx, "proses-pengguna")
	defer span.End()
}

A span carries attributes such as the operation name and status. Every span for a single request shares the same trace ID, so visualization tools like Jaeger or Grafana Tempo can show the request's complete journey across services.

#Context Propagation

For a trace to continue across services, headers such as traceparent must be forwarded. OpenTelemetry handles this propagation automatically through HTTP middleware. On the client side, net/http is instrumented with otelhttp so every outgoing request automatically carries the context.

#Metrics with the Prometheus Client

#Counter, Gauge, and Histogram

The Prometheus client for Go provides three main metric types: Counter for values that only increase, like request counts; Gauge for values that go up and down, like active connections; and Histogram for the distribution of durations.

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

#Recording and Exporting Metrics

package main
 
import (
	"net/http"
 
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)
 
var jumlahRequest = prometheus.NewCounterVec(
	prometheus.CounterOpts{
		Name: "http_requests_total",
		Help: "Total request HTTP per route",
	},
	[]string{"route"},
)
 
func main() {
	prometheus.MustRegister(jumlahRequest)
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":8080", nil)
}

The /metrics endpoint is exported in Prometheus format and collected by a scraper. Grafana then visualizes these metrics into dashboards. To inspect the endpoint locally, curl localhost:8080/metrics displays all metrics in Prometheus text format.

#The RED and USE Methods

When choosing metrics, use the RED framework (Rate, Errors, Duration) for services: how many requests per second, how many error, and how long they take. The USE framework (Utilization, Saturation, Errors) applies to resources such as CPU and memory. Both ensure you measure the right things.

#Closing

Episode 14 made your Go application observable: structured logging with zerolog, zap, or logrus in JSON format; distributed tracing with OpenTelemetry and context propagation; and Counter, Gauge, and Histogram metrics with the Prometheus client exported through /metrics.

Key takeaways:

• Structured logs are JSON so machines can aggregate them.
• Log levels are controlled by the environment: debug, info, warn, error.
• Traces connect operations across services through spans.
• OpenTelemetry is the open standard for tracing.
• Prometheus exports Counter, Gauge, and Histogram metrics.
• Use the RED and USE methods to choose the right metrics.

In the next episode we will discuss resilience, graceful shutdown, and health checks — shutting down a service gracefully with context and OS signals, providing health checks and readiness probes for Kubernetes, plus circuit breakers and retry policies for unstable external calls.