Learn Apache Kafka - Operational Excellence & Production Readiness

#Introduction

Taking Kafka to production isn't the end of the journey — it's the beginning of an ongoing operational routine. Production clusters are run, upgraded, monitored, handled when something goes wrong, and cost-optimized. Episode 34 brings all these practices together into an operational excellence guide.

You'll learn production checklists, operational procedures, performance baselines, common problems along with their troubleshooting, and cost optimization strategies. This episode is the closest to the daily reality of a platform engineer.

#Production Checklist

#Deployment Checklist

Before real traffic arrives, verify:

• Capacity planning is done (episode 20): number of brokers, partitions, storage, network.
• Security hardening is applied (episode 33): SASL, TLS, ACLs, network segmentation.
• Monitoring and alerting is active (episodes 22-23): key metrics and alerts have runbooks.
• Backup and DR are tested (episode 25): restore and failover drills have been run.
• Documentation and runbooks are complete: who's responsible and what the steps are.
• The team is trained: every member knows the basic procedures and escalation paths.

#Runbooks and Training

Runbooks turn procedures into action during crises. For every scenario (broker down, full disk, lag spike), write: symptoms, investigation, mitigation, and prevention. Train the team with game-days or chaos tests (episode 32) so runbooks actually work.

#Operational Procedures

#Change Management and Upgrades

Production changes must go through a process: documentation, review, a time window, and a rollback plan.

Broker upgrade procedure:

1. Back up the configuration and record versions.
2. Upgrade one broker, verify it joins and replicates normally.
3. Continue the rolling upgrade of the next brokers.
4. When done, verify versions across the cluster and observe.

bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

kafka-broker-api-versions.sh shows API versions; use it before and after the upgrade to verify all brokers are on the desired version.

#Broker Maintenance and Topic Management

• Broker maintenance: for host maintenance, shut down the broker in a controlled way (graceful shutdown), let partitions move, then bring it back up and run preferred leader election.
• Partition reassignment: do it when new brokers are added or workloads are unbalanced (episode 20).
• Topic management: define and review topics via code (episode 31); clean up unused topics.
• Consumer group management: monitor and stop unused groups so offsets don't pile up.

#Performance Baselines

#Establishing Baselines

Before problems occur, record the normal state:

• Throughput benchmarks: records/sec and MB/sec per broker and per topic.
• Latency percentiles: p50, p99, p999 for produce and fetch.
• Resource utilization: CPU, memory, disk I/O, network per broker.
• Consumer lag targets: normal vs abnormal lag per group.

Baselines let you distinguish real problems from normal variation. Alerts comparing current values to a baseline (for example 2x the average) are far more useful than absolute thresholds.

#Common Production Problems

#Rebalance Storms and Lag Spikes

• Rebalance storms: repeated rebalances caused by consumers leaving and rejoining a group (for example max.poll.interval.ms too short or GC pauses). Stabilize with the CooperativeStickyAssignor (episode 21) and consumer tuning.
• Lag spikes: a consumer crashed or slowed down. Investigate poll intervals, resources, and errors in consumer logs.

#Full Disks, Network Partitions, and Memory

• Full disks: the most common outage cause. Monitor space and set retention; add storage or move data (episode 20).
• Network partitions: replicas go out of sync; check ISR and inter-broker request errors.
• Memory leaks and GC pauses: a continuously growing heap or long pauses indicate problems in the client application or broker JVM; use thread and heap dumps for analysis.

#Troubleshooting

#Log Analysis and Metrics

Start with broker logs and JMX metrics. Look for repeating error patterns, request timeouts, and sudden metric changes. Broker logs mention partitions, principals, and the failing phase — often enough for an initial diagnosis.

#Thread and Heap Dumps

For mysterious performance problems:

jstack <broker-pid> > /tmp/broker-thread.txt

jstack <broker-pid> captures a broker thread snapshot — GC pauses or stuck threads are clearly visible. Heap dumps (jmap) reveal memory leaks. Analyze offline with tools like Eclipse MAT or VisualVM.

#Network and Client Debugging

• Check connections with ss and inter-node latency.
• Test client connections with console tools to separate client and broker problems.
• Use --verbose on Kafka CLIs and increase client logging during investigation.

#Cost Optimization

• Right-sizing brokers: match instances to real load; don't over-provision without data.
• Tiered storage (episode 11): move old data to cheap object storage.
• Compression optimization (episode 21): zstd reduces storage and bandwidth.
• Retention tuning: shorten retention for data that doesn't need long storage.
• Cloud: use reserved instances for stable loads and spot instances for development; shut down non-production clusters outside working hours.

#Closing

In this episode 34 you've understood production checklists, operational procedures like upgrades and maintenance, performance baselines, common problems with troubleshooting, and cost optimization strategies.

The key takeaways:

• Verify the production checklist before real traffic arrives.
• Upgrade brokers in a rolling fashion with verification at each step.
• Performance baselines distinguish real problems from normal variation.
• Rebalance storms and full disks are the most common production problems.
• Use logs, metrics, thread, and heap dumps for structured diagnosis.
• Cost optimization: right-sizing, tiered storage, compression, and retention.

In episode 35 — the final episode — we'll discuss the latest features and the future of Kafka: KRaft, tiered storage, recent KIPs like KIP-848, trends like Kafka as a database and event meshes, and a summary of the whole series' best practices. See you in the finale!