Learn RabbitMQ - Alerting & Health Checks

#Introduction

In episode 24 you built observability: metrics flow to Prometheus and appear in Grafana. But a beautiful dashboard won't wake you at 3 a.m. when queues pile up. In this episode, observability transforms into actionability — through health checks and alerting.

Health checks answer two simple but important questions: "is the broker still alive?" (liveness) and "is the broker ready to receive traffic?" (readiness). Meanwhile, alerting ensures abnormal conditions reach the team immediately, through the right channels — Slack, email, or PagerDuty.

This episode sets up alerting systematically: what must be alerted, what the thresholds are, and through which notifications. By the end, you'll have a basic runbook for responding to the first signs of broker failure.

#Health Checks

#Liveness and Readiness Checks

A liveness check ensures the node process is alive. The easiest way: check whether the AMQP port responds, or ask for the node status:

nc -zv localhost 5672
rabbitmqctl await_startup

A readiness check ensures the node is ready to receive traffic — queues are accessible, no alarms are active, and the node isn't partitioned. The right readiness check uses the internal health check API:

rabbitmqctl eval 'rabbit_health_checks:check_local_alarms().'
rabbitmqctl eval 'rabbit_health_checks:check_if_running().'

The check_local_alarms() evaluation fails if the node is in a memory/disk alarm — that's the "not ready" signal orchestrators like Kubernetes can use (episode 29).

#Custom Health Check Endpoint

For specific needs, write your own health check script that verifies: the node isn't alarming, consumer connections are active for critical queues, and queue depth is below a threshold. Run this script on a schedule and send the results to your monitoring system.

#Alerting Strategies

#Critical Alerts You Must Have

Some conditions must trigger an alert:

• Memory alarm — the node is blocking publishers; an immediate action sign.
• Disk alarm — the disk is almost full; the node stops processing messages.
• Queue length — queue depth passes a threshold for a certain duration.
• Consumer down — no active consumer for critical queues.
• Cluster partition — nodes are separated; split-brain risk.
• Dead letter queue — DLQ depth keeps rising (a reminder from episode 11).

#Writing Prometheus Alert Rules

An example queue depth alert rule in prometheus.yml:

groups:
  - name: rabbitmq
    rules:
      - alert: RabbitQueueDepthHigh
        expr: rabbitmq_queue_messages > 10000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Queue {{ $labels.queue }} melebihi 10 ribu pesan"

The rule above fires if queue messages exceed 10,000 for 5 minutes — long enough to ignore momentary spikes.

#Alerting Tool Integration

#Notifications to Slack and Email

Grafana and Alertmanager can forward alerts to many channels. In Alertmanager, set up Slack and email receivers:

receivers:
  - name: ops
    slack_configs:
      - channel: "#infra-alert"
        api_url: "https://hooks.slack.com/services/XXXX"
    email_configs:
      - to: "ops@example.com"

Both receivers above make sure notifications reach the ops team's Slack channel and a backup email.

#PagerDuty and Custom Webhooks

For critical incidents demanding immediate response, connect to PagerDuty or OpsGenie — on-call alerts get escalated automatically. For internal systems, a custom webhook is enough: Alertmanager sends a JSON POST to your team's endpoint, which can forward it to an internal notification app or bot.

#Conclusion

In episode 25 you created liveness and readiness checks, used the internal health check API, set up alerting for memory, disk, queue length, consumer down, partition, and DLQ, and integrated notifications into Slack, email, PagerDuty, and webhooks.

Key takeaways:

• Liveness checks that the process is alive; readiness checks that it can accept traffic.
• check_local_alarms is a reliable built-in readiness check.
• Memory and disk alerts must be responded to immediately.
• Queue length alerts with a for duration avoid false positives.
• Consumer down and cluster partition alerts catch critical failures.
• PagerDuty for on-call escalation; Slack for general information.
• Too many alerts numb the team — choose carefully.

In the next episode we will connect federation and shovel — replicating exchanges and queues between clusters with Federation, forwarding messages point-to-point with Shovel, comparing the two, and designing multi-datacenter patterns and reliable message transfer for disaster recovery. This is the bridge between brokers!