Learn Observability with the LGTM Stack - TraceQL - The Tempo Query Language

#Introduction

Finding traces by TraceID is useful, but how do you find all traces that failed in payment in the last hour? The answer is TraceQL — Tempo's query language that allows finding traces by span attributes and relationships between spans.

This episode builds your TraceQL skills from the foundations: span selection, the difference between intrinsic fields and attributes, the most commonly used query patterns, the available operators, up to advanced queries that exploit trace structure.

#TraceQL Fundamentals

#Span Selection

TraceQL selects spans using curly braces with attribute conditions:

{ span.http.status_code >= 500 }
{ resource.service.name = "checkout" }
{ span.http.route = "/api/orders" }

The { span.http.status_code >= 500 } query selects all spans with an HTTP status of 500 or above — the first pattern when investigating errors.

#Intrinsic Fields vs Attributes

Tempo distinguishes two kinds of fields:

• Intrinsic fields: built-in span fields like name, duration, status, and kind.
• Attributes: arbitrary key-value pairs, distinguished by scope — span. for span attributes and resource. for resource attributes.

{ name = "orders.process" }
{ duration > 500ms }
{ status = error }

The example { duration > 500ms } shows that duration can be filtered directly as an intrinsic field.

#Query Patterns

#Common Trace Searches

Some patterns most often used in production:

{ resource.service.name = "payment" && status = error }
{ resource.service.name = "orders" && duration > 2s }
{ span.http.route = "/api/checkout" } && { span.http.status_code = "500" }

The first query finds traces from the payment service that failed; the second finds slow requests in the orders service.

#Duration and Error Filters

For latency and failure investigations, combine intrinsics and attributes:

{ resource.service.name = "checkout" } && { duration > 1s }
{ status = error } | count() > 3

The expression { status = error } | count() > 3 finds traces with more than three error spans — an indication of cascading failures.

#TraceQL Operators

#Available Operators

• Comparison operators: =, !=, >, <, >=, <=.
• Logical operators: && (and), || (or), ! (negation).
• Pipeline operators: | to process results between conditions.
• Aggregate functions: count(), avg(), max(), min() on sets of spans.

{ resource.service.name = "checkout" && span.http.status_code = "500" }
{ resource.service.name = "payment" } && { name = "call.retry" }

The && operator inside one pair of braces combines conditions on the same span; between two pairs of braces it combines conditions on different spans.

#Advanced Queries

#Multi-Span and Parent-Child

TraceQL's power emerges when you combine conditions across spans:

{ resource.service.name = "gateway" } >> { resource.service.name = "payment" }
{ resource.service.name = "orders" } > { resource.service.name = "payment" }

The >> operator selects traces where the first span is an ancestor of the second, while > selects a direct parent-child relationship.

#Metrics from Traces

With nested {} operators, TraceQL can produce time-series metrics:

{ resource.service.name = "checkout" } | rate()
{ resource.service.name = "checkout" && status = error } | count()
{ resource.service.name = "payment" } | quantile_over_time(duration, .95)

The { resource.service.name = "checkout" } | rate() query computes the number of traces per second — exactly the pattern used by RED dashboards in episode 24.

#Closing

In episode 15 you mastered TraceQL: span selection with attribute conditions, the difference between intrinsic fields and attributes, query patterns for finding errors and slow requests, comparison, logic, pipeline, and aggregation operators, and advanced multi-span queries and metrics from traces.

The key takeaways:

• TraceQL selects spans with conditions inside curly braces.
• Intrinsic fields like name, duration, and status.
• && combines conditions, >> and > select span relationships.
• Pipelines and aggregations turn traces into metrics.
• Start queries from a service then widen conditions gradually.

In the next episode 16 we'll discuss the OpenTelemetry Collector deep dive — the receiver, processor, exporter, and extension architecture, agent and gateway deployment modes, popular receivers like OTLP and Prometheus, and configuration best practices. The Collector becomes the central point that manages all telemetry data.