Learn Traefik - Distributed Tracing

#Introduction

Metrics tell you there is a slow request; access logs tell you which one. But where exactly is time lost? The answer is in distributed tracing: the technique of following a single request through every service it touches, recording each stop as a span, and assembling them into a complete trace.

Traefik is the perfect point to start and propagate a trace: every request enters through Traefik, so a trace created here covers the entire request journey. Episode 25 covers configuring Traefik tracing to various backends and how to read the results.

#Supported Tracing Backends

#Five Options

• Jaeger: open source, full-featured, the most common choice with Traefik.
• Zipkin: Jaeger's predecessor, still widely used.
• Datadog APM: integrated with Datadog observability.
• Elastic APM: the pair for the Elastic stack.
• OpenTelemetry: the modern standard that exports to many backends.

Traefik v3 supports all of them; the choice is determined by the observability stack you already have. For this episode we use Jaeger because it is easy to run and feature-rich.

#Jaeger Integration

#Running Jaeger and Configuring Traefik

Jaeger can be run in a single all-in-one container for development. Run docker run with the jaegertracing/all-in-one image:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Then point Traefik to Jaeger via static config:

tracing:
  jaeger:
    samplingServerURL: "http://jaeger:5778/sampling"
    samplingType: const
    samplingParam: 1.0
    localAgentHostPort: "jaeger:6831"
    traceContextHeaderName: "uber-trace-id"
    propagation: jaeger

• samplingType: const with samplingParam: 1.0: samples all traces. For production, lower it to 0.1 (10 percent) to keep storage costs under control.
• localAgentHostPort: the Jaeger agent address. Traefik sends spans to this agent.
• propagation: the trace context propagation mechanism.

#Agent vs. Collector

Jaeger has two intake paths: the agent (UDP, port 6831, lightweight, installed near the application) and the collector (HTTP, port 14268, for large batches). Traefik supports both. For large clusters, send directly to the collector so UDP does not become a bottleneck.

#Trace Context Propagation

#Connecting Traces Between Services

For one request to form one complete trace, the trace ID must spread to every service it passes through. Traefik reads the propagation header from the client, adds its own span, then forwards that header to the backend:

uber-trace-id: 1234abc:5678def:0:1

If the backend is also instrumented (e.g. with an OpenTelemetry SDK), it reads the same header, creates its own span under the same trace, and forwards it again to the next service. The result appears in the Jaeger UI as one interconnected waterfall, not separate pieces.

#Analyzing Traces

#Waterfall and Latency Breakdown

The Jaeger UI (port 16686) displays traces as a waterfall: one row per span, with duration on the time axis. From here you can see:

• Request path: from Traefik, to service A, then to service B.
• Latency breakdown: which span consumes the most time.
• Error tracking: spans with an error status are marked red.
• Bottleneck identification: services that are consistently slow across all traces.

The recommended analysis steps:

1. Select the traefik service and look for the slowest traces.
2. Examine the waterfall — which span is dominant?
3. Drill down into the suspicious service span.
4. Compare with Prometheus metrics to see whether the problem is systemic.

#OpenTelemetry and Other Backends

#The Modern Standard

For a modern stack, use OpenTelemetry which exports to Jaeger, Tempo, or Datadog simultaneously:

tracing:
  openTelemetry:
    grpc:
      endpoint: "otel-collector:4317"
      insecure: true

This pattern sends spans to an OpenTelemetry Collector, which then forwards them to the backend of choice. With one configuration, you can switch backends without touching Traefik.

#Closing

Key takeaways:

• Tracing records a request's journey as spans in a single trace.
• Backends: Jaeger, Zipkin, Datadog APM, Elastic APM, OpenTelemetry.
• Jaeger all-in-one for dev; a UDP agent or HTTP collector for production.
• Trace context propagation through headers connects spans between services.
• The waterfall in the Jaeger UI shows latency breakdown and errors.
• Low sampling for production; raise it only when investigating.

In episode 26 next we enter the production phase: high availability & clustering — running multiple stateless Traefik instances, configuration synchronization, sharing acme.json for Let's Encrypt, load balancing in front of Traefik, and /ping health checks for external load balancers.