Learn Observability with the LGTM Stack - Correlation - Unifying the Three Pillars

#Introduction

The three pillars of observability are just a pile of separate data if they aren't connected to each other. The main advantage of the LGTM Stack lies precisely in correlation — the ability to move seamlessly from metrics to traces to logs within a single investigation flow.

This episode covers correlation between pillars: from metrics to traces via exemplars, from logs to traces via TraceID, from traces back to logs, and the unified debugging flow that's the primary habit in production.

#Metrics to Trace

#Exemplars in Mimir

An exemplar is the mechanism that links metrics to real traces. An exemplar stores a TraceID and a duration as an example inside a histogram metric:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

From a histogram panel in Grafana, you can click a point and select the TraceID listed as an exemplar. That click opens the relevant trace directly — the first bridge between metrics and traces.

#Dashboard Drill-Down and Span Metrics

• Trace ID injection: the application includes the TraceID when recording duration metrics.
• Dashboard drill-down: metric panels are equipped with links to trace panels.
• Span metrics from Tempo: the metrics-generator derives RED metrics from traces — details in episode 24.

metric panel -> click point -> exemplar TraceID -> trace waterfall

The metric panel -> click point -> exemplar TraceID pattern is the most valuable interaction in observability dashboards.

#Log to Trace

#TraceID in Logs

Every structured log carries a TraceID (see episode 12). In Grafana, a TraceID appearing in a log panel automatically becomes a clickable link to the related trace.

{
  "level": "error",
  "service": "payment",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "msg": "payment gateway timeout"
}

The trace_id field is what Grafana recognizes to create automatic log → trace correlation without manual configuration.

#Trace Logs Panel and Automatic Correlation

The Trace logs panel in Grafana displays logs that match the active span. With settings on the Tempo data source — such as derived fields — Grafana automatically links log lines to traces and vice versa.

#Trace to Log

#Derived Fields and Log Queries from Traces

To open logs from a trace, Grafana uses derived fields: regex patterns that extract a TraceID or other values from span attributes, then trigger a LogQL query to Loki.

derivedFields:
  - name: TraceID
    matcher: traceid=(\\w+)
    datasourceUid: loki-uid
    url: $${__value.raw}

The derivedFields configuration above makes Grafana take the TraceID value from span attributes and directly open a log query to Loki.

#Contextual Log Viewing

With derived fields, from a trace waterfall you can one-click display all the logs for that TraceID — without manually copying the TraceID.

#Unified Workflow

#Alert → Metric → Trace → Log

This is the golden triangle introduced in episode 2, now in real working form:

alert -> metrik -> trace -> log -> root cause

The alert -> metrik -> trace -> log pattern is followed during incidents: receive the alert, look at metrics to confirm, open the trace to find the problematic span, then follow the logs for error details.

#Designing Dashboards for Correlation

• Provide drill-down links between panels.
• Enable exemplars on histogram panels.
• Make sure all logs carry a TraceID.
• Use derived fields on the Tempo data source.

#Closing

In episode 18 you understood correlating metrics to traces via exemplars, logs to traces via clickable TraceIDs, traces back to logs via derived fields, and the unified debugging flow of alert → metric → trace → log that becomes the production working pattern.

The key takeaways:

• Exemplars link metrics to real traces.
• A TraceID in logs opens a trace with a single click.
• Derived fields extract TraceIDs from span attributes.
• Golden triangle: alert, metric, trace, then log.
• Correlation requires configuration preparation, not automatic magic.

In the next episode 19 we'll discuss alerting strategy and AlertManager — the principles of actionable alerts, Grafana Alerting, routing and grouping, recording rules, and alert maintenance best practices. The correlated metrics, logs, and traces will start protecting your system proactively.