Learn Observability with the LGTM Stack - Distributed Tracing with Tempo - Introduction

#Introduction

The third pillar of observability is traces — the map of a request's journey across many services. Without traces, finding which service slows down a request is pure guesswork. Grafana Tempo is the backend that stores and queries that trace data.

Tempo is designed with the same philosophy as Loki: cheap, scalable, and based on object storage. This episode covers Tempo's architecture, fundamental tracing concepts, sampling strategies, and a comparison of Tempo with Jaeger.

#Grafana Tempo Overview

#Design and Advantages

• Scalable distributed tracing backend: every component can scale horizontally.
• Object storage based: traces are stored in S3, GCS, or Azure Blob — not on local disk.
• Native OTLP support: Tempo receives traces directly in the OTLP protocol, without adapters.
• Cost-effective trace storage: without full indexing, costs are far lower than Jaeger.
• TraceQL query language: a dedicated query language for finding traces by span attributes — covered in episode 15.

curl -s http://localhost:3200/ready

The ready output from curl -s http://localhost:3200/ready means Tempo is ready to receive traces — a habit you've been using since episode 4.

#Tempo Architecture

Like its siblings Mimir and Loki, Tempo splits the trace path into components:

• Distributor: receives traces from collectors, validates them, and sends them to ingesters.
• Ingester: stores recent traces in memory and writes blocks to object storage.
• Querier: answers queries by TraceID or TraceQL queries.
• Query-frontend: manages the queue and splits large queries.
• Compactor: merges small blocks into larger ones.
• Metrics-generator (optional): generates RED metrics from traces — covered in episode 24.

otlp -> distributor -> ingester -> object storage
query -> query-frontend -> querier <- blok

The otlp -> distributor -> ingester -> object storage flow is the main storage path. The metrics-generator reads the same traces to derive metrics.

#Tracing Concepts Deep Dive

#Span and Trace

• Trace: the complete journey of a request, identified by a TraceID.
• Span: one unit of work within a trace, like "database query" or "HTTP call", with a duration and attributes.
• Span context: the span identity metadata forwarded between services.

span:
  name: call.payment
  trace_id: 4bf92f3577b34da6
  span_id: 00f067aa0ba902b7
  parent_span_id: 1d2e3f4a5b6c7d8e
  duration_ms: 145
  attributes:
    http.status_code: 500

The parent_span_id field is what forms the parent-child relationships between spans within one trace.

#Context Propagation

A trace only forms when context is forwarded between services. The standard used is W3C Trace Context via the traceparent header — you'll practice this hands-on in episode 14.

#Sampling Strategy

Storing every trace is unrealistic at high traffic. There are two main approaches:

• Head-based sampling: decides whether to sample when the trace starts, consistent for the whole trace.
• Tail-based sampling: decides after the trace finishes, allowing smarter rules.

sampler:
  type: traceidratio
  ratio: 0.1

The value ratio: 0.1 means 10 percent of traces are stored. The fuller sampling concept is in episode 24.

#Tempo vs Jaeger

#Architecture Comparison

• Architecture: Tempo adopts the Mimir-style component pattern; Jaeger uses a gRPC architecture with Cassandra/Elasticsearch.
• Storage: Tempo stores trace blocks in object storage without content indexing; Jaeger relies on an external database with indexing.
• Query: Tempo has TraceQL for attribute-based queries; Jaeger searches by service, operation, and tag.
• Scalability: both are scalable, but Tempo's operational cost is lower because of object storage.

tempo: object storage + TraceQL, cheap, Grafana-integrated
jaeger: external database, mature, broad ecosystem

The tempo: object storage + TraceQL pattern is the choice in this series because of native Grafana integration and storage cost.

#Closing

In episode 13 you understood Tempo as an object-storage-based trace backend with native OTLP support, learned its architectural components from distributor to metrics-generator, understood the concepts of span, trace, context propagation, and sampling, and compared Tempo with Jaeger.

The key takeaways:

• Tempo stores traces in object storage, not local disk.
• Traces are built from spans with parent-child relationships.
• OTLP is Tempo's primary ingress protocol.
• Sampling reduces trace volume and cost.
• Tempo wins on cost and Grafana integration.

In the next episode 14 we'll discuss OpenTelemetry instrumentation for traces — auto-instrumentation per language, creating manual spans, W3C context propagation, and best practices for span naming and attributes. It's time to make your applications produce real traces.