Learn Apache Kafka - Scaling Kafka Clusters

#Introduction

A healthy Kafka cluster today isn't necessarily healthy six months from now — traffic grows, new topics appear, and brokers start nearing disk or CPU limits. The question isn't whether you need to scale, but how to do it without downtime and without disrupting applications.

Episode 20 covers horizontal scaling by adding brokers, moving partitions and replicas through reassignment, the impact of adding partitions on ordering and consumers, correct capacity planning, and Cruise Control for automating rebalancing and detecting anomalies.

#Horizontal Scaling

#Adding Brokers

Adding a broker to a KRaft-mode cluster is straightforward: prepare a new node, format the storage, and start it. The new broker automatically joins and starts accepting new partitions. However — important to understand — a new broker does not automatically take over existing partitions; balance is only achieved after reassignment.

#Partition Reassignment

Move partitions between brokers with kafka-reassign-partitions.sh:

bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json --execute

The reassignment.json file contains the list of partitions and their destination broker list:

{
  "version": 1,
  "partitions": [
    {"topic": "orders", "partition": 0, "replicas": [1, 2, 3]}
  ]
}

"replicas": [1, 2, 3] determines which brokers hold the replicas. The reassignment process copies data first (mirror), then moves leadership — safe and without data loss.

#Replica Reassignment and Load Balancing

Reassignment is also used to balance replicas (not just leaders) — for example ensuring two replicas of a topic aren't on the same broker (rack awareness). After the new broker catches up, run preferred leader election so leader load spreads evenly, and repeat kafka-reassign-partitions.sh --verify to make sure it's complete.

#Partition Scaling

#Increasing the Partition Count

Sometimes a topic needs more partitions because a consumer group needs more parallelism. Use --alter:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic orders --partitions 12

kafka-topics.sh --alter --partitions 12 expands a topic from 6 to 12 partitions. This operation can only increase, never decrease.

#Impact on Keys and Consumers

Adding partitions has side effects you must understand:

• Key hashing changes: murmur2(key) % numPartitions recomputes the key destination, so records with the same key can move partitions — per-key ordering is no longer fully guaranteed after this change.
• Consumer rebalance: the partition count change triggers a consumer group rebalance; make sure applications are ready.
• Old data isn't moved; new partitions start empty, creating temporary imbalance until reassigned.

#Safe Strategies

Design the partition count up front with room to grow, or consider keys that stay stable. If an increase is unavoidable, do it during low load, communicate with the consumer teams, and monitor rebalance and lag after the change.

#Capacity Planning

#Broker Count and Replication Factor

A general rule: total required throughput divided by per-broker throughput, times the replication factor and failure tolerance. With a replication factor of 3, effective capacity is about one third of total disk capacity — because every record is stored on three brokers.

#Disk Sizing and Network

Consider: write throughput per second times retention time determines disk size per partition. With tiered storage (episode 11), local disks only hold hot data. Don't forget bandwidth: replication and recovery need network — when one broker dies, the remaining followers absorb additional replication load.

#Partition Count and Consumers

The partition count limits the maximum consumer parallelism per group. Plan partitions = target consumer group × growth reserve. Too few partitions limits throughput; too many adds metadata overhead and rebalance latency.

#Cruise Control

#Introducing Cruise Control

Cruise Control is an open-source system from LinkedIn that automates Kafka cluster operations: rebalancing, partition movement, and anomaly detection. It uses goal-based optimization: you define goals (for example even leader distribution, balanced disks), and Cruise Control computes and executes the best reassignment plan.

#Key Features

• Automated rebalancing: schedules partition movement based on goals without manual intervention.
• Anomaly detection: detects unhealthy brokers, unbalanced disks, and under-replicated partitions.
• Self-healing: catches problems and runs automated repair actions.
• Custom limits: controls data movement speed so it doesn't disturb production load.

#Closing

In this episode 20 you've understood horizontal scaling by adding brokers, partition and replica reassignment, the impact of adding partitions on keys and consumers, capacity planning principles, and Cruise Control for automatic rebalancing.

The key takeaways:

• New brokers don't automatically take partitions — reassignment is required.
• kafka-reassign-partitions.sh moves partitions without downtime.
• Adding partitions changes key hash results and can break ordering.
• Capacity planning covers throughput, disk, network, and the replication factor.
• The partition count limits maximum consumer parallelism.
• Cruise Control automates rebalancing and detects anomalies.

In the next episode 21 we'll optimize performance: performance tuning and optimization. You'll learn producer and consumer tuning, JVM and OS optimization for brokers, and topic configuration for the best throughput and latency.