Learn Observability with the LGTM Stack - PromQL - The Prometheus Query Language

#Introduction

All the metrics in Mimir are meaningless without a language to read them. PromQL (Prometheus Query Language) is the query language that lets you select time series, compute rates, aggregate across labels, and calculate latency percentiles.

This episode builds your PromQL understanding from scratch: vector concepts, label selectors, operators, common query patterns, important functions, and performance best practices. This is the skill you'll use most often in Grafana, dashboards, and alert rules.

#PromQL Basics

#Instant Vector vs Range Vector

PromQL works with two main data types:

• Instant vector: a set of time series with a single value at one point in time — for example: http_requests_total.
• Range vector: a set of time series with values over a time range — for example: http_requests_total[5m].

Functions like rate and increase need a range vector as input, while dashboard queries need an instant vector as output.

#Selectors and Operators

Selectors pick series by name and label:

http_requests_total{status="500"}
http_requests_total{status=~"5.."}
http_requests_total{job!="cron"}

Operators in PromQL come in three types:

• Arithmetic: +, -, *, /, %.
• Comparison: ==, !=, >, <, >=, <=.
• Logical: and, or, unless.

sum | avg | max | min | count

Aggregators like sum combine several series into one based on labels.

#Common Query Patterns

#Rate and Increase

rate computes the average counter increase per second — the most common output for RED metrics. Meanwhile increase computes the total increase over a time range.

rate(http_requests_total[5m])
increase(http_requests_total[5m])

The rate(http_requests_total[5m]) query tells you how many requests per second on average in the last 5 minutes. This is the basic pattern for RED method dashboards.

#Latency Percentiles with histogram_quantile

Histograms store the latency distribution in buckets. To compute the 95th percentile:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

The histogram_quantile(0.95, ...) function interpolates the percentile value from histogram buckets. You must understand this concept because almost all latency SLOs in episode 20 use percentiles.

#Prediction and Other Functions

• predict_linear(http_requests_total[1h], 3600): predicts the value 1 hour ahead for capacity planning.
• irate(http_requests_total[5m]): the instant rate, sensitive to spikes.
• delta(counter[5m]): the change in a gauge value.
• avg_over_time(metric[5m]): the average value over a time range.

#Advanced PromQL

#Subqueries

Subqueries allow a query to contain a time range computed from another query's results, useful for nested aggregation. Here's an example computing the per-minute average rate:

avg_over_time(rate(http_requests_total[1m])[15m:1m])

#Recording Rules

Recording rules store expensive query results as new named metrics, so dashboards just read precomputed results. Rules are written in YAML form:

groups:
  - name: checkout.rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

The job:http_requests:rate5m rule will produce a derived metric that can be queried directly without recomputing the rate every time. Metrics resulting from recording rules usually follow the level:metric:operation convention.

#Alerting Rules and Label Manipulation

Alerting rules are rules that evaluate certain conditions and produce alerts — full details in episode 19. For label manipulation, PromQL provides the label_replace and label_join functions to rewrite or combine labels.

#Best Practices

• Performance optimization: start queries with a limiting label selector, use recording rules for heavy queries, and avoid nested subqueries without need.
• Avoid high cardinality: never add high-cardinality labels to metrics; it slows queries and consumes memory.
• Choose the right time range: overly long range vectors slow queries; keep them sufficient for the dashboard's needs.
• Design alert queries carefully: use for and stable conditions so alerts don't flap — detailed in episode 19.

#Closing

In episode 7 you mastered the PromQL basics: the difference between instant vectors and range vectors, operators and aggregation, common patterns like rate, increase, and histogram_quantile, advanced functions like predict_linear, and best practices for query performance.

The key takeaways:

• Instant vectors for current values, range vectors for time ranges.
• rate is the main pattern for RED metrics.
• histogram_quantile computes percentiles from histogram buckets.
• Recording rules store expensive queries as derived metrics.
• Label selectors limit data from the start of a query.
• Avoid high cardinality for performance and cost.

In the next episode 8 we'll discuss instrumenting applications for metrics — using the Prometheus and OpenTelemetry SDK libraries to create counters, gauges, and histograms, manual and auto-instrumentation patterns, and exposing metrics via a metrics endpoint and OTLP. It's time to make your own applications speak.