Learn Observability with the LGTM Stack - LogQL - The Loki Query Language

#Introduction

Loki stores logs, and to read them you need a query language: LogQL. Unlike most query languages, LogQL works in two layers — selecting log streams based on labels, then filtering and parsing the line content within those streams.

This episode builds your LogQL skills from scratch: stream selectors, line filters, parser expressions, log metrics, and optimization techniques. With LogQL, you can turn raw log collections into answers — how many errors in an hour, or what's the average latency per service.

#LogQL Basics

#Stream Selectors and Label Matchers

Every LogQL query starts by selecting a log stream using labels:

{job="checkout"}
{service="payment", level="error"}
{job=~"checkout|cart"}

The {job="checkout"} notation is the same concept as Prometheus label selectors — a combination of =, !=, =~, and !~ matchers for labels.

#Line Filters

After selecting the stream, you can filter lines by their content:

{job="checkout"} |= "error"
{job="checkout"} |~ "timeout|refused"
{job="checkout"} != "debug"

The |=, |~, !=, and !~ operators filter lines without parsing. {job="checkout"} |= "error" returns only lines containing the word error.

#Parser Expressions

To read fields from structured logs, use parsers:

#JSON and Logfmt

{job="checkout"} |= "error" | json
{job="checkout"} | json level="lvl" | lvl="error"
{job="checkout"} | logfmt

The json parser extracts all JSON fields into temporary query labels. You can also remap them like json level="lvl" to keep field names consistent.

#Pattern Parser

For semi-structured logs, use the pattern parser:

{job="api"} | pattern "<method> <path> <status> <duration>"

The pattern parser uses positional templates to extract fields. This pattern is very efficient for server access logs.

#Log Queries and Metric Queries

#Two Query Types

LogQL has two kinds of results:

• Log query: returns log lines, for example {job="checkout"} |= "error".
• Metric query: returns time series computed from logs, wrapped in aggregation functions.

#Stream Aggregation

The most commonly used log metric functions:

count_over_time({job="checkout"} |= "error" [5m])
rate({job="checkout"} |= "error" [5m])
bytes_over_time({job="checkout"} [5m])

The count_over_time({job="checkout"} |= "error" [5m]) query counts the number of error lines in 5 minutes — exactly like the RED errors pattern.

#Unwrap and Advanced Aggregation

To compute numeric values from log fields, use unwrap:

avg_over_time({job="api"} | json | unwrap duration [5m])
sum(rate({job="api"} | json | unwrap duration [5m]))

The combination | json | unwrap duration extracts the duration field then computes its statistics — similar to a histogram from log data.

#Advanced LogQL

#Label Formatting and Binary Operations

LogQL can rewrite labels with label_format and combine two queries with binary operators:

{job="checkout"} | label_format env=service_name
sum(rate({service="payment"} [5m])) / sum(rate({service="orders"} [5m]))

The results of binary operations can be combined with Grafana template variables for dynamic dashboards.

#Performance Optimization

A few habits to keep LogQL queries fast:

• Use the right labels in the stream selector — the less data it traverses, the faster.
• Limit the time range — don't query a full week without need.
• Filter lines before parsing — |= "error" is cheaper than parsing then filtering.
• Take advantage of caching aggregation results for queries used repeatedly.

#Closing

In episode 10 you mastered LogQL: selecting streams with stream selectors, filtering lines with filter operators, extracting fields with the json, logfmt, and pattern parsers, computing metrics from logs, and applying performance optimization techniques.

The key takeaways:

• A LogQL query always starts with a label stream selector.
• Line filters are cheaper than parsing.
• The json, logfmt, and pattern parsers extract fields from log content.
• Metric queries turn logs into time series.
• Limit labels, time ranges, and data before parsing.

In the next episode 11 we'll discuss collecting logs with Grafana Alloy — the replacement for Grafana Agent, its River-based component architecture, collection methods from files and Docker, parsing and enrichment processes, up to sending logs to Loki with remote write. Your LogQL queries will soon have a data source that keeps flowing.