Learn Observability with the LGTM Stack - Alerting Strategy & Alert Manager

#Introduction

Observability isn't just about seeing problems — it's about detecting problems before users complain. This is alerting's role: connecting telemetry queries to notifications that trigger action.

This episode covers a healthy alerting strategy, the Grafana Alerting mechanism, the routing and grouping concepts in the AlertManager style, recording rules for expensive queries, and best practices so your team doesn't drown in alert fatigue.

#Alerting Principles

#Actionable Alerts

A good alert answers three questions: what's wrong, how severe is it, and what should be done. An alert without a runbook is just noise.

• Actionable: every alert must trigger a clear action.
• Alert fatigue prevention: reduce noise; alerts that are often wrong just get ignored.
• Severity levels: use critical, warning, and info with clear thresholds.
• On-call best practices: don't wake up on-call for conditions that can wait.

what's wrong? how severe? what should be done?

The three questions what's wrong? how severe? what should be done? are the first filter before creating a new alert.

#Grafana Alerting

#Creating Alert Rules

An alert rule in Grafana is a query evaluated periodically:

sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))

The query above computes the ratio of 5xx requests. The alert rule combines it with a condition, a for period, and a destination folder.

#Multi-Dimensional Alerting

Grafana supports multi-dimensional alerts: one rule produces many alert instances, one per label combination. For example: the error rate rule above will create one alert per service that breaches the threshold.

#Evaluation and State Management

• Alert evaluation: the query is evaluated at each interval, for example every 1 minute.
• Pending: the condition is met but hasn't passed for.
• Firing: the condition has held for the for period.
• Resolved: the condition returns to normal.

conditions:
  - evaluator:
      type: greater_than
      params: [0.05]
for: 10m

The value for: 10m prevents alert flapping — the alert only fires when the error rate stays high for 10 consecutive minutes.

#AlertManager

#Routing and Grouping

Routing determines where alerts are sent, grouping combines similar alerts into one notification:

route:
  group_by: ["service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: on-call-pagerduty
    - match:
        severity: warning
      receiver: slack-alerts

The rule match: severity: critical sends critical alerts to the PagerDuty on-call, while warnings are enough for Slack.

#Inhibition, Silencing, and Templating

• Inhibition rules: suppress lower-severity alerts when a critical alert is active.
• Silencing: disable notifications for certain conditions during maintenance.
• Notification channels: email, Slack, PagerDuty, webhook, and others.
• Templating: customize notification content with alert data.

#Recording Rules

#Pre-Computing Expensive Queries

Recording rules compute heavy queries in advance and store them as new metrics:

groups:
  - name: error_rate.rules
    rules:
      - record: job:http_errors:ratio5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) /
              sum(rate(http_requests_total[5m])) by (job)

The metric job:http_errors:ratio5m can be used by alerts and dashboards without recomputing the heavy query — exactly the concept you saw in episode 7.

#Alert Best Practices

• Consistent naming: service:condition:severity, for example checkout:error-rate:critical.
• Runbook links: every alert carries a link to a runbook.
• Contextual information: include actual values, thresholds, and duration.
• Alert testing: test every new alert with simulations.
• Alert review: audit regularly to remove alerts that are no longer useful.

#Closing

In episode 19 you understood the principles of actionable alerts, the Grafana Alerting mechanism with evaluation and state management, routing and grouping in the AlertManager style, recording rules for expensive queries, and best practices for maintaining alerts.

The key takeaways:

• Every alert must be actionable and carry a runbook.
• An alert rule is a query with a condition, threshold, and for period.
• Routing determines the destination, grouping combines similar alerts.
• Recording rules preload heavy queries as derived metrics.
• Regular audits prevent alert fatigue.

In the next episode 20 we'll discuss SLI, SLO, SLA, and error budgets — defining reliability indicators, setting targets, creating SLO dashboards and burn rate alerts, and managing error budgets as decision-making input. Alerting without reliability targets is just an alarm without direction.