Learn Observability with the LGTM Stack - Scaling Tempo for Production

#Introduction

Traces differ from metrics and logs: every TraceID is unique, and finding a trace without its TraceID requires expensive search. This challenge is why scaling Tempo needs its own strategy — from deployment to sampling.

This episode covers Tempo deployment, ingestion strategies, query optimization with bloom filters and caching, sampling strategies, and metrics derived from traces via the metrics-generator.

#Tempo Deployment

#Single Binary vs Microservices

• Single binary: all components in one process — enough for small and medium scale.
• Microservices: distributor, ingester, querier, and others are separate — for large scale.

single binary -> microservices

The single binary -> microservices pattern follows the same path as Loki — start simple, upgrade as needed.

#Storage and Retention

• Backend storage selection: choose S3, GCS, or Azure Blob according to your environment.
• Retention configuration: set how long traces are stored — consider cost and compliance.

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-blocks
    block:
      blocklist_poll: 5m

The block backend: s3 moves trace storage to object storage — the same pattern as Mimir in episode 22.

#Ingestion Strategies

#Rate Limiting and Backpressure

• Rate limiting: limit the trace volume per tenant to keep the cluster healthy.
• Load balancing: distribute the ingestion load across many distributors.
• Batching strategies: send traces in batches from the Collector.
• Backpressure handling: signal collectors when ingesters are busy.

overrides:
  per_tenant_override_config:
    ingester:
      max_local_traces_per_user: 100000

The value max_local_traces_per_user: 100000 limits traces per tenant at the ingester — a basic safeguard when traffic spikes.

#Query Optimization

#TraceID Lookup and Search

• TraceID lookup: direct access by TraceID — fast, using the bloom index.
• Search performance: TraceQL searches require healthy bloom filters.
• Caching configuration: cache search results for repeated queries.
• Bloom filters: a data structure that speeds up span searches within blocks.

storage:
  trace:
    block:
      bloom_filter_false_positive: 0.05
      index_downsample_bytes: 1048576

The value bloom_filter_false_positive: 0.05 balances search speed against index size.

#Sampling Strategy

#Four Main Approaches

• Probabilistic sampling: selecting randomly based on probability — the simplest.
• Rate limiting sampling: limiting the number of traces per second.
• Tail-based sampling: deciding after a trace completes, allowing smart rules.
• Adaptive sampling: adjusting based on traffic and priorities.

tail_sampling:
  policies:
    - name: keep-errors
      type: status_code
      status_code:
        status_codes:
          - ERROR

The tail_sampling policy ensures error traces are always stored even at high traffic — a valuable property when debugging.

#Sampling Trade-Offs

Sampling reduces cost but narrows coverage. A general rule: store all error traces, sample normal traces at a ratio, and consider service priorities.

#Metrics from Traces

#Metrics-Generator

The metrics-generator component derives metrics from passing traces:

• Span metrics generator: creates per-service RED metrics.
• Service graphs: maps relationships between services.
• RED metrics from traces: rate, errors, and duration from trace data.
• Exemplar generation: generates exemplars that link metrics to traces.

trace -> metrics-generator -> RED metrics -> Mimir -> dashboard

The trace -> metrics-generator -> RED metrics pattern explains how services not instrumented for metrics can still produce metrics through trace data.

#Closing

In episode 24 you understood Tempo deployment from single binary to microservices, ingestion strategies with rate limiting and backpressure, query optimization with bloom filters and caching, the four sampling approaches, and metrics derived from traces.

The key takeaways:

• Traces without a TraceID are expensive to find; use bloom filters.
• Sampling reduces cost; store all error traces.
• Rate limiting protects ingesters from surges.
• The metrics-generator derives RED metrics from traces.
• Service graphs map relationships between services.

In the next episode 25 we'll discuss multi-tenancy and access control — tenancy models, multi-tenancy in Mimir, Loki, and Tempo, RBAC in Grafana, and authentication methods from basic auth to SAML. When many teams share one stack, isolation and access control become the key to security.