Learn Observability with the LGTM Stack - Three Pillars of Observability - Concepts & Philosophy

#Introduction

The entire stack you'll build in this series serves three types of telemetry data known as the three pillars of observability: metrics, logs, and traces. Each answers a different question, and none can replace the others.

This episode dissects the philosophy and concepts behind each pillar, from metric types, log structure, to trace anatomy. You'll also learn how all three correlate with each other — this is what distinguishes true observability from a mere collection of tools. Understand this episode well, because every term here will be used over and over.

#Metrics — What Is Happening

#Metric Types

Metrics are numeric data collected at regular intervals, stored as time-series. Each series is represented by a metric name and a set of key-value labels, for example http_requests_total{method="GET", status="200"}. Three main types you must know:

• Counter: a value that only increases monotonically, for counting occurrences, for example the number of requests.
• Gauge: a value that can go up and down, for example the number of active connections or CPU usage.
• Histogram and Summary: distributions of observations, for example request latency, used to compute percentiles.

Metrics are collected and aggregated, but you must be wary of cardinality — too many label combinations will multiply the number of series and burden storage. This principle matters when you design labels in episode 6.

#The USE and RED Methods

Two popular frameworks for choosing the right metrics:

• USE method (Utilization, Saturation, Errors): focuses on resources. Utilization is how full a resource is, saturation is excess capacity, and errors are resource failures. Good for infrastructure monitoring.
• RED method (Rate, Errors, Duration): focuses on services. Rate is the number of requests per second, Errors is the number of failed requests, and Duration is the latency distribution. Good for microservices.

rate:     http_requests_total
errors:   http_requests_total{status=~"5.."}
duration: http_request_duration_seconds

The example above will become real PromQL queries in episode 7. Remember the pattern {status=~"5.."} as a label filter representing 5xx status codes.

#Logs — What Happened

Logs are records of events that occur in a system, usually one line per occurrence complete with a timestamp. The log philosophy is simple: it is the most detailed source of truth about exactly what happened.

#Log Structure and Levels

• Structured vs unstructured logs: structured logs are in key-value or JSON form so machines can parse them easily; unstructured logs are just free text. Always prioritize structured logs — the details are covered in episode 12.
• Log levels: DEBUG, INFO, WARN, ERROR, and FATAL are used to group levels of severity. Wrong levels (too verbose in production) will waste storage costs.

#Contextual and Correlation

Good logs carry context: service name, request ID, user ID, and especially TraceID. With a TraceID, a single log line can be linked directly to the trace that contains it. This is the main bridge between logs and traces.

{
  "level": "error",
  "ts": "2026-08-10T09:12:33Z",
  "service": "checkout",
  "trace_id": "b3c9e1a2f4d50781",
  "msg": "payment gateway timeout"
}

Note the "trace_id" pattern above — fields like this are what Grafana uses to automatically correlate logs with traces in episode 18.

#Traces — Why It Happened

Traces map the journey of a request across many services. This is the pillar that most distinguishes observability from traditional monitoring.

#Span, Trace, and Relationships

• Span: a single named unit of work with a duration, for example "database query" or "payment API call". Each span carries a start time, duration, attributes, and status.
• Trace: a collection of spans that form one complete request, linked by the same TraceID.
• Parent-child relationships: a span can be the parent of another span, forming a tree — for example an HTTP request span becomes the parent of a database query span.
• Critical path analysis: identifying the longest span in a trace to find the bottleneck.

#Context Propagation

For a trace to form across services, each service must forward the context — TraceID, SpanID, and sampling flags — to the next service via HTTP headers. The most widely used standard is W3C Trace Context with the traceparent header.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The traceparent header format has four parts separated by hyphens: version, TraceID, SpanID, and flags. You'll practice propagation details hands-on in episode 14.

#Correlation Between Pillars

The power of observability emerges when all three pillars are combined. Some common correlation mechanisms:

• Exemplars: certain metrics carry a reference to an example trace, so metric dashboards can open the relevant trace directly.
• TraceID in logs: every log carries a TraceID that links it to a trace.
• Unified querying: from Grafana, you can move from metrics to logs to traces without switching tools.
• Context switching workflows: the standard investigation flow — start from an alert, look at metrics, dive into traces, then follow the related logs.

This flow is dubbed the golden triangle of observability. The sequence alert → metrics → traces → logs will be the main debugging pattern in episodes 18 and 21.

alert -> metrics -> traces -> logs

You'll see this alert -> metrics -> traces -> logs diagram repeatedly. Memorize the direction, because almost every incident investigation starts here.

#Closing

In episode 2 you understood the philosophy and concepts of the three pillars of observability: metrics answering "what is happening" with counter, gauge, histogram, and summary types, logs answering "what exactly happened" in structured form, and traces answering "why" through spans and context propagation.

The key takeaways:

• Metrics use the USE method for resources and RED for services.
• Structured logs with a TraceID are the bridge to traces.
• Traces are built from spans with parent-child relationships.
• The W3C context propagation standard unifies traces across services.
• Correlate pillars through exemplars, TraceID, and unified querying.
• The golden triangle flow: alert, metrics, traces, then logs.

In the next episode 3 we'll discuss OpenTelemetry — the vendor-agnostic observability standard, from the history of the OpenTracing and OpenCensus merger, the API and SDK components, to why it became the foundation of modern instrumentation. Make sure your understanding of the three pillars is solid before moving on.