Learn gRPC - Observability, Tracing & Monitoring

#Introduction

A server you can't observe is a black box: you know there's a problem, but not where. Episode 14 opens that box. We build observability for gRPC on three pillars: metrics (aggregate numbers like QPS and latency), logging (individual events), and tracing (one request's journey across many services).

You'll use Prometheus for metrics, OpenTelemetry for tracing, and grpcurl and ghz for hands-on debugging. After this episode, every RPC can answer the question: where is it slow, and why?

#Monitoring with Prometheus

#gRPC Metrics

gRPC supports standardized metrics. In Go, plug in the automatic interceptor from go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc or a metrics package:

import (
    "github.com/grpc-ecosystem/go-grpc-prometheus"
)
 
grpcPrometheus.Register(s)

grpcPrometheus.Register(s) monitors every RPC and produces metrics like grpc_server_started_total, grpc_server_handled_total, and grpc_server_handling_seconds — labeled with method and status code.

#Expose a Metrics Endpoint

Prometheus pulls metrics through an HTTP endpoint. Run a small HTTP server on a separate port:

http.Handle("/metrics", promhttp.Handler())
go http.ListenAndServe(":9090", nil)

With promhttp.Handler() at /metrics, Prometheus can collect the data. Add a scrape config to prometheus.yml:

scrape_configs:
  - job_name: grpc-catalog
    static_configs:
      - targets: ["catalog:9090"]

The catalog:9090 target above points Prometheus at the gRPC server's metrics endpoint. The metrics are then visualized in Grafana with a ready-made gRPC dashboard.

#OpenTelemetry and Distributed Tracing

#Setting Up a Tracer

OpenTelemetry is the cross-vendor observability standard. Install it as a gRPC interceptor so tracing is automatic without touching your handlers:

import (
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)
 
s := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

grpc.StatsHandler(otelgrpc.NewServerHandler()) wraps the server with a tracing stats handler: every RPC automatically becomes a span with duration, method, and status. On the client side, install otelgrpc.NewClientHandler() the same way.

#Span Context Propagation

The magic of tracing happens in propagation: the client's span becomes the parent of the server's span. gRPC carries this context through the standard traceparent metadata:

conn, _ := grpc.NewClient("localhost:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)
res, err := client.GetProduct(ctx, &pb.ProductId{Id: "p-001"})

When GetProduct calls another service, the same span is forwarded. The result: one trace spanning from A to B to C — and in Jaeger or Tempo you can see which request contributes the most latency.

#Debugging gRPC with Dedicated Tools

#grpcurl for Direct Inspection

When there's an anomaly, direct debugging beats waiting for a dashboard:

grpcurl -plaintext localhost:50051 list
grpcurl -plaintext -v localhost:50051 \
  catalog.v1.CatalogService/GetProduct

The -v flag reveals metadata headers and status details — including grpc-status and grpc-message, which explain why a call failed.

#ghz to Confirm Behavior

When you suspect a performance regression, run a quick benchmark:

ghz --insecure -n 1000 -c 20 \
  localhost:50051 catalog.v1.CatalogService/GetProduct

ghz --insecure -n 1000 -c 20 gives immediate numbers on throughput and latency — a perfect counterpart to the historical dashboards in Grafana.

#Reading Dashboards Correctly

Combine the three signals: error rate rising together with p99 latency means the server is overwhelmed; error rate up but latency normal points to a problem with the input; error rate stable but latency up means a bottleneck in a dependency. These three signals are what separate targeted debugging from guesswork.

#Closing

Key takeaways:

• Prometheus gRPC metrics (starting with grpc_server_*) provide QPS, latency, and error numbers.
• Expose /metrics and scrape it with Prometheus, then visualize in Grafana.
• OpenTelemetry produces spans automatically through a StatsHandler without touching handlers.
• traceparent propagation makes a single trace span across services.
• grpcurl -v unpacks metadata and status for direct debugging.
• Combine error rate, p99, and dependency duration for targeted diagnosis.

In episode 15 next, we cover resilience, retry, and fault tolerance — retry policy via service config, circuit breakers, timeout and failover strategies, and graceful shutdown, health checks, and readiness probes. The system that's now observable will be built to keep standing when the components beneath it fall.