Learn Envoy Proxy - Observability & Telemetry Integration

#Introduction

Episode 7 introduced access logs and basic metrics. Episode 11 brings observability to the ecosystem level: metrics to Prometheus for aggregation, distributed tracing with Zipkin, Jaeger, and OpenTelemetry to follow a single request through many services, and access log enrichment so logs carry enough context to correlate.

This is a bridge episode: after it, you don't just have raw data from Envoy — you have a system that can answer "where is this request slow?" and "which service has the most errors?".

#Envoy Metrics to Prometheus

#Statistics Configuration and Flushing

Envoy counts internal metrics and exposes them through the admin interface. For production, make sure statistics are active and either sent to a statsd server or scraped by Prometheus:

stats_config:
  stats_tags:
    - tag_name: cluster
      regex: "^cluster\\.(.+?)\\.(upstream|membership)\\."
    - tag_name: listener
      regex: "^listener\\.(.+?)\\."
stats_flush_interval: 30s

The stats_config block defines tags extracted from metric names. With the regex above, the metric cluster.orders_service.upstream_rq_total automatically gets the label cluster with the value orders_service, so Prometheus can aggregate per cluster.

#Scraping with Prometheus

scrape_configs:
  - job_name: envoy
    metrics_path: /stats/prometheus
    static_configs:
      - targets:
          - envoy-1:9901
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

The target envoy-1:9901 points to each Envoy's admin interface. The instance relabel gives each instance a unique label so metrics from many Envoys can be distinguished.

#Key Metrics to Monitor

Some metrics always worth looking at:

• envoy_cluster_upstream_rq_total and envoy_cluster_upstream_rq_time for volume and latency.
• envoy_cluster_upstream_rq_5xx for per-cluster error rate.
• envoy_listener_downstream_cx_active for active connections per listener.
• envoy_cluster_membership_healthy for the number of healthy endpoints.

curl -s localhost:9901/stats/prometheus | grep "envoy_cluster_upstream_rq_time"

The curl localhost:9901/stats/prometheus command pulls metrics directly in Prometheus format. In Grafana, you can chart latency with a query like histogram_quantile(0.99, sum(rate(envoy_cluster_upstream_rq_time_bucket[5m])) by (le, cluster)).

#Distributed Tracing

#Configuring a Tracer in the Bootstrap

Distributed tracing lets a single request be tracked across many proxies and services. Tracer configuration lives in the bootstrap:

tracing:
  http:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.Tracing
      http:
        name: envoy.tracers.opentelemetry
        typed_config:
          "@type": type.googleapis.com/envoy.tracers.opentelemetry.v3.OpenTelemetryConfig
          grpc_service:
            envoy_grpc:
              cluster_name: otel_collector
          service_name: envoy-gateway

The tracing block connects Envoy to an OpenTelemetry collector over gRPC. Every request passing through Envoy produces a span with the service_name envoy-gateway, forming part of the same trace.

#Zipkin and Jaeger

For Jaeger or Zipkin, swap the tracer:

tracing:
  http:
    name: envoy.tracers.zipkin
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.Tracing
      http:
        name: envoy.tracers.zipkin
        typed_config:
          "@type": type.googleapis.com/envoy.tracers.zipkin.v3.ZipkinConfig
          collector_cluster_name: zipkin_collector
          collector_endpoint: /api/v2/spans
          collector_endpoint_version: HTTP_JSON

The collector_cluster_name concept is the same for all tracers: Envoy sends spans to the cluster pointing at the collector. Since the Otel tracer is now the most common, many teams switch straight to the OpenTelemetry Collector as their single destination.

#Header Propagation

Tracing works because headers are propagated between services. Envoy reads and forwards headers like traceparent (W3C) and x-b3-traceid (Zipkin). Make sure your backend application forwards these headers too; otherwise the trace will break at the first service.

#Access Log Enrichment

#Adding Trace Context to Access Logs

Access logs can be enriched with trace IDs so they're easy to correlate with tracing data:

access_log:
  - name: envoy.access_loggers.file
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      path: /dev/stdout
      format: "%START_TIME% %DOWNSTREAM_REMOTE_ADDRESS% %REQ(X-REQUEST-ID)% traceid=%REQ(TRACEPARENT)% %RESPONSE_CODE% %DURATION%ms %UPSTREAM_CLUSTER%\n"

The format above adds %REQ(TRACEPARENT)% and %REQ(X-REQUEST-ID)% to every log line. With this context, you can take one trace ID and find all related log lines across every service.

#Request ID for Correlation

Envoy can also automatically add x-request-id if it doesn't exist:

http_filters:
  - name: envoy.filters.http.router
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
      start_child_span: true

The value start_child_span: true makes the router create a child span per request, enriching the trace on Envoy's side. The same x-request-id header can be used to join access logs across all hops.

#Verifying the Observability Flow

To make sure everything is connected:

curl -s -H "Host: api.example.com" http://localhost:10000/api/orders
docker logs jaeger 2>&1 | tail -5

After sending the request, open the Jaeger UI on port 16686 and search for traces with the service envoy-gateway. The docker logs jaeger command is a quick way to see whether spans have reached the collector.

#Closing

Episode 11 connected Envoy with the observability ecosystem: label-rich Prometheus metrics, distributed tracing through OpenTelemetry, Zipkin, and Jaeger, and access logs carrying trace and request ID context.

Key takeaways:

• stats_config with regex tags makes Envoy metrics easy to aggregate per cluster.
• Key metrics: upstream_rq_time, upstream_rq_5xx, downstream_cx_active, membership_healthy.
• Tracers are configured in the bootstrap; Otel is now the main choice.
• Trace headers like traceparent must be propagated by the application for complete traces.
• Access logs can be enriched with trace IDs and request IDs for correlation.
• x-request-id lets a single request be tracked across all Envoy hops.

In the next episode, episode 12, we'll discuss secure service-to-service communication — Envoy as a secure gateway for microservices, TLS contexts and certificate validation, and authorization policies with the RBAC filter.