Learn Apache Kafka - Alerting & Health Checks

#Introduction

Monitoring gives you data; alerting turns data into action. Without good alerts, metrics just pile up on dashboards — real problems are only detected when users report them. Episode 23 covers how to build an effective alerting system for Kafka: health checks, critical alerts, and integration with notification channels.

The key to good alerting isn't quantity, but selectivity. Alerts should signal something that genuinely requires action, avoid alert fatigue, and provide enough context for a fast response.

#Health Check Strategies

#Broker Liveness

The basic check: is the broker responding? Use the CLI to test the connection and API:

bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 --version-check

kafka-broker-api-versions.sh --version-check confirms the broker responds with its API versions. For periodic checks, run this command on a cron schedule and alert if it fails — it verifies TCP connectivity, the protocol, and handlers all at once.

#Cluster Health Metrics

Beyond per-broker liveness, monitor aggregate health:

• ActiveControllerCount = 1: more than one or zero signals a problem.
• UnderReplicatedPartitions = 0: any value above zero needs investigation.
• OfflinePartitions = 0: a partition without a leader is a critical condition.
• Write availability: try producing a test record and verify it's stored.

#Topic Availability and Consumer Health

• Topic availability: make sure critical topics have all partitions with leaders. Also monitor min.insync.replicas so acks=all doesn't fail.
• Consumer health: a consumer group missing from the description or growing lag signals a troubled consumer. Monitor group status and consumption rate.

#Critical Alerts

#Under-Replicated and Offline Partitions

Recommended thresholds:

• UnderReplicatedPartitions > 0 for more than a few minutes: warning; replication hasn't recovered.
• OfflinePartitions > 0: critical; data isn't available.
• ActiveControllerCount != 1: critical.

A common Prometheus rule:

- alert: KafkaUnderReplicatedPartitions
  expr: kafka_server_replicamanager_underreplicatedpartitions > 0
  for: 5m
 
- alert: KafkaOfflinePartitions
  expr: kafka_controller_kafkacontroller_offlinepartitionscount > 0
  for: 1m

for: 5m gives transition tolerance so alerts don't flood during normal rebalance. These rules only fire when the condition persists past the duration.

#Controller Flapping and Lag Spikes

• Controller flapping: ActiveControllerCount changing rapidly signals unstable brokers — a critical alert because every controller switch delays cluster operations.
• Consumer lag spike: lag suddenly jumps, usually from a consumer crash or rebalance. Alert with thresholds relative to the baseline: for example lag exceeding 2x the average, or passing an absolute threshold for critical topics.
• Disk space: remaining space below 20 percent — warning; below 10 percent — critical.
• Network partition: detect via inter-broker request error metrics or ISR inconsistency.

#Alert Integration

#PagerDuty and OpsGenie

For critical events, on-call integration: Prometheus Alertmanager sends alerts to PagerDuty or OpsGenie, which escalate to the on-call team. Set clear severities: critical (pages on-call) versus warning (email/Slack), so only real problems wake people up.

#Slack Notifications

Non-critical channels or aggregate notifications go to Slack:

receivers:
  - name: "slack-ops"
    slack_configs:
      - channel: "#kafka-alerts"
        send_resolved: true

slack_configs routes notifications to a specific channel with send_resolved: true so teams know the problem is resolved. Include alert context — which broker, what metric, since when — in the message.

#Custom Webhooks and Remediation

For special needs, Alertmanager can call a custom webhook. This opens up remediation automation: a webhook triggers a script that investigates the troubled broker, collects a thread dump, or executes standard repair actions. Start with safe automated investigation, then expand to remediation once the action is proven reliable.

#Closing

In this episode 23 you've understood broker and cluster health check strategies, critical alerts for under-replicated partitions, offline partitions, controller flapping, and lag spikes, and alert integration into PagerDuty, Slack, and custom webhooks.

The key takeaways:

• Health checks test liveness and cluster health periodically.
• OfflinePartitions and controller flapping are critical alerts.
• Use the for duration to avoid alert fatigue during normal rebalance.
• PagerDuty for critical, Slack for warnings; separate severities.
• Lag spikes and disk space need clear thresholds.
• Every alert must have a runbook and enough context.

In the next episode 24 we'll dissect replication and high availability — leader and follower replicas, In-Sync Replicas, the high watermark, unclean leader election, min.insync.replicas, and rack awareness for failure domain isolation.