Learn Apache Kafka - Topics & Partitions Deep Dive

#Introduction

Topics and partitions are Kafka's primary unit of storage and parallelism. If you choose the wrong number of partitions or the wrong key strategy, your application's performance and consistency will pay the price — and fixing it after production is very painful.

Episode 4 dissects topics from every angle: how to create, name, and configure them; the concept of partitions and their distribution across brokers; partitioning strategies with keys, round-robin, and custom partitioners; and the retention and cleanup policy parameters that govern the data lifecycle.

Take this episode seriously. Almost every Kafka architectural decision — from ordering, throughput, to storage costs — is rooted in how you design your topics and their partitions.

#Topic Fundamentals

#Creating Topics

Topics can be created explicitly with the CLI or automatically when a producer first sends to a topic that doesn't exist yet (if auto.create.topics.enable is active). For production, always create them explicitly:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders \
  --partitions 6 --replication-factor 3

6 partitions give you parallelism, and a replication factor of 3 tolerates two broker failures. Delete topics with the --delete flag, and list all topics with kafka-topics.sh --list.

#Topic Naming Rules

A topic name is the identity used by clients and automated tools, so follow the conventions: use lowercase, separate words with a hyphen (-), and avoid special characters. Names are limited to 249 characters and cannot use . or ... Good naming describes the domain and data type, for example payment.events.completed with dot separators for hierarchy, or orders-v1 for schema versioning.

#Listing and Describing

--list shows names only; --describe shows important details:

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

The output shows, per line: partition, leader, replica, and ISR. Leaders should be evenly spread, and all replicas should appear in the ISR for a healthy state.

#Partition Concepts

#Partition Count and Distribution

The number of partitions determines the read and write parallelism limit: the maximum number of active consumers per group equals the number of partitions, and writes are distributed evenly when there is no key. A general rule: start with enough partitions for your target throughput (for example 6-12), because adding partitions can change ordering and add metadata overhead.

Partitions are distributed across brokers with rack awareness in mind. Partition leaders are spread evenly so no single broker becomes a hotspot.

#Partition Key and Message Ordering

The key determines the destination partition. Records with the same key always go to the same partition — as long as the partition count doesn't change — so ordering is preserved per key. Without a key, the producer uses round-robin (or sticky batching) for even distribution.

#Leader Election and Preferred Leader

When the leader broker dies, the controller elects a new follower from the ISR as leader. Kafka maintains the preferred leader — the broker that should be leading based on the initial distribution — and performs --reassignment to restore balance after a broker recovers:

bin/kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --topic orders --partition 0 --election-type preferred

#Partitioning Strategies

#Key-Based Partitioning

The most common strategy for preserving order per entity: use the entity ID as the key.

order-001 -> partition 2
order-001 -> partition 2  (order preserved)
order-002 -> partition 0

Compute the partition key with murmur2(key) % numPartitions. The implication: an unbalanced key will create hot partitions.

#Round-Robin and Custom Partitioner

Without a key, the producer distributes records round-robin or with sticky batching: a group of records sticks to one partition within a single batch to maximize efficiency, then moves on. For special needs — for example placing records in partitions based on region — you can write a custom partitioner by implementing the Partitioner interface in Java and pointing to it via partitioner.class:

partitioner.class=com.example.RegionPartitioner

#Repartitioning Challenges

Increasing the partition count changes the key hash results so old ordering can get mixed up, and this operation cannot be reversed without reassignment. For that reason, design the partition count up front with room to grow, and if you must increase it, understand the impact on ordering and consumers.

#Topic Configuration

#Retention Time vs Size

Two main parameters govern how long data persists:

retention.ms=604800000
retention.bytes=-1

retention.ms=604800000 (7 days) or retention.bytes limits by size. A value of -1 means unlimited for that parameter. Retention is calculated per partition.

#Segments and Cleanup Policies

Segments are bounded by segment.ms and segment.bytes; only closed segments can be deleted or compacted. The cleanup policy is set via cleanup.policy:

• delete (default): expired segments are deleted based on retention.
• compact: the latest value per key is kept, old history is discarded.
• compact,delete: combines both.

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name orders \
  --add-config cleanup.policy=compact

#Min In-Sync Replicas and Compression

min.insync.replicas determines the minimum number of replicas that must acknowledge a write for the partition to be considered healthy — combined with acks=all on the producer to guarantee durability. compression.type determines the broker's compression codec for data compressed by the producer: producer (use the producer setting), gzip, snappy, lz4, or zstd. Large data can be stored more economically with zstd, at the cost of CPU.

#Closing

In this episode 4 you've mastered topics and partitions from A to Z: creating and naming topics, configuration parameters, partition distribution and leader election, key-based partitioning strategies, round-robin and custom partitioners, plus retention, segments, and the delete and compact cleanup policies.

The key takeaways:

• Partitions are the unit of parallelism and ordering; ordering is only per partition.
• The partition count determines the consumer group parallelism limit.
• The same key always goes to the same partition, preserving order per key.
• Repartitioning (adding partitions) changes hash results and can break ordering.
• Retention is based on retention.ms and retention.bytes; cleanup policy is delete or compact.
• min.insync.replicas plus acks=all guarantees high durability.

In the next episode 5 we'll write data to Kafka: producers — from basic configuration, APIs in Java, Python, Go, and Node.js, the fire-and-forget, synchronous and asynchronous delivery patterns, to tuning acks, retries, batching, compression, and idempotent producers for exactly-once semantics. Get ready to write your first code!