Learn Apache Kafka - Monitoring & Observability

#Introduction

An unmonitored Kafka cluster is a time bomb: slowing replication, piling consumer lag, or a full disk can wreck production without warning. Monitoring and observability give you the visibility to spot problems before they become outages.

Episode 22 covers how Kafka exposes metrics via JMX, key metrics at the broker, producer, consumer, and topic levels, integration with Prometheus and Grafana, and specialized tools like Burrow for reliably monitoring consumer lag.

#Kafka Metrics

#JMX Metrics

Kafka brokers and clients expose metrics through JMX (Java Management Extensions). With -Dcom.sun.management.jmxremote at startup, metrics are available through the JMX port. For modern collection, these metrics are translated into the Prometheus format by the JMX exporter.

The JMX exporter configuration is fairly simple: a YAML file registers the metric patterns you want to expose, then the Java agent attaches to the broker process:

lowercaseOutputName: true
rules:
  - pattern: "kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent"
    name: kafka_request_handler_avg_idle_percent

kafka.server:type=KafkaRequestHandlerPool is an MBean pattern; this rule maps the JMX metric to a Prometheus name. Other patterns for brokers, producers, and consumers can be added as needed.

#Broker Metrics

Key broker-side metrics:

• UnderReplicatedPartitions: the number of partitions whose replicas are lagging behind the leader. Should be near zero.
• OfflinePartitions: partitions without a leader — a critical condition that stops writes and reads.
• ActiveControllerCount: must be exactly 1 in a cluster; more than 1 signals conflict.
• Request latency for Produce and Fetch, broken down by percentile.

#Producer and Consumer Metrics

• Producer: record-send-rate, record-error-rate, request-latency-avg, and buffer-available-bytes (running out of buffer signals an overwhelmed producer).
• Consumer: records-lag-max (the largest lag across all partitions), fetch-rate, and poll-interval-avg — a poll exceeding max.poll.interval.ms triggers a rebalance.

#Topic Metrics

Per-topic metrics can be seen with the CLI description or from topic-labeled JMX metrics: bytes-in, bytes-out, and message rate per topic. These help identify hot topics dominating cluster load.

#Key Metrics to Monitor

#Under-Replicated and Offline Partitions

The first two metrics to put on your dashboard:

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

kafka-topics.sh --describe --under-replicated-partitions lists all under-replicated partitions. If the number keeps rising, a broker may be slow at writing or the network is disturbed; combining it with the OfflinePartitions metric separates replication problems from availability problems.

#Consumer Lag

Consumer lag is the difference between the log-end-offset (the newest data in a partition) and the current-offset (consumption position). Growing lag means consumers can't keep up with the production rate:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-consumers

kafka-consumer-groups.sh --describe --group order-consumers shows per partition: current-offset, log-end-offset, and lag. Lag itself isn't always bad — what's dangerous is lag that keeps growing.

#Disk, Network, and Request Rate

• Disk usage and I/O: a full disk is the most common outage cause; monitor space and I/O wait.
• Network throughput: bytes-in/out per broker for bottleneck detection.
• Request handler utilization: the percentage of time broker handlers work; a high value means the broker is saturated.

#Monitoring Tools

#JMX Exporter and Prometheus

The standard open-source observability flow:

java -jar jmx_prometheus_javaagent.jar 7071:/etc/kafka/jmx-exporter-config.yml \
  -Dcom.sun.management.jmxremote=true \
  -jar kafka-server-start.jar config/server.properties

jmx_prometheus_javaagent.jar opens port 7071 with metrics in the Prometheus format, which Prometheus scrapes periodically and Grafana visualizes. Many Kafka dashboard templates are available — start from one, then adjust.

#Grafana Dashboards

Grafana combines Prometheus metrics into dashboards: replication health, throughput, lag, and broker resources in a single screen. Create alerts in Grafana for key metrics — alerting details are covered in episode 23.

#Specialized Tool: Burrow

LinkedIn's Burrow monitors consumer lag more reliably than client metrics: it reads directly from Kafka, computes per-partition lag in a stateless way, and applies a sliding window-based evaluation to judge consumer status. Main advantages: it doesn't depend on client reports (which can be misleading when a client is dead) and detects consumers that are dead but still committing.

#Closing

In this episode 22 you've understood JMX metric exposure, broker, producer, consumer, and topic metrics, key metrics like under-replicated partitions and consumer lag, Prometheus and Grafana integration, and lag monitoring tools like Burrow.

The key takeaways:

• All Kafka metrics originate from JMX and can be exported to Prometheus.
• UnderReplicatedPartitions and OfflinePartitions are the first health metrics.
• Consumer lag must be monitored by trend, not just value.
• A full disk is the most common outage cause — monitor space and I/O.
• Burrow computes lag statelessly and is more reliable than client metrics.
• Grafana combines everything in one dashboard for fast response.

In the next episode 23 we'll build alerting and health checks — health check strategies for brokers and clusters, critical alerts like under-replicated partitions and lag spikes, and integrating alerts into PagerDuty, Slack, and custom webhooks.