Learn Apache Kafka - Performance Tuning & Optimization

#Introduction

Kafka can handle millions of records per second, but that performance doesn't appear automatically — it's the result of configuration matched to your workload. Brokers, producers, consumers, and topics each have parameters that influence each other, and changes on one side often ripple to another.

Episode 21 covers tuning from four angles: producer, consumer, broker (including JVM and OS), and topic configuration. You'll understand the throughput versus latency trade-offs and how to tune the parameters that have the biggest impact on your workload.

#Producer Optimization

#Batching and Compression

The most influential parameters on producer throughput:

batch.size=131072
linger.ms=20
compression.type=zstd
acks=all

batch.size=131072 increases the batch size to 128KB and linger.ms=20 holds records for 20ms to combine writes — this dramatically increases throughput. compression.type=zstd reduces the bytes sent: zstd gives the best ratio, snappy and lz4 offer a CPU balance.

#Acks and In-Flight

acks=all gives the best durability; the trade-off is waiting for all ISRs to confirm. max.in.flight.requests.per.connection controls how many requests are sent without waiting — with idempotence enabled, a value of 5 is safe without reordering. For the lowest latency, use linger.ms=0 and small batches, but throughput drops.

#Consumer Optimization

#Fetch Size and Poll Interval

A slow consumer is often not about code, but about small fetch configuration:

fetch.min.bytes=1048576
fetch.max.wait.ms=500
max.poll.records=500

fetch.min.bytes=1048576 makes brokers wait until 1MB accumulates before responding — reducing round-trips and raising throughput. max.poll.records limits the number of records per poll so processing doesn't exceed max.poll.interval.ms.

#Partition Assignment Strategy

The assignment strategy determines how partitions are divided among consumers:

• RangeAssignor: default, per-topic division — can create imbalance across topics.
• RoundRobinAssignor: rotates partitions evenly across topics.
• StickyAssignor: keeps the old assignment as long as possible for stability during rebalance.
• CooperativeStickyAssignor: supports incremental rebalancing (episode 6) for large clusters.

Choose CooperativeStickyAssignor for smoother rebalances in large consumer groups.

#Broker Optimization

#JVM Tuning

Brokers are JVM applications. The two most impactful things:

• Heap size: enough for metadata and buffers, usually 4-6GB; don't make it too big because Kafka relies heavily on the OS page cache, not the heap.
• Garbage collector: the default G1GC is already good; avoid long GC pauses by monitoring kafka.server:type=KafkaRequestHandlerPool. Make sure reads go through the page cache, not the heap.

#OS-Level Tuning

sysctl -w vm.swappiness=1
sysctl -w vm.dirty_ratio=60
sysctl -w vm.dirty_background_ratio=5
sysctl -w net.core.wmem_max=209715200
sysctl -w net.core.rmem_max=209715200

vm.swappiness=1 reduces swap usage that can cause large GC pauses, and net.core.wmem_max enlarges network socket buffers. The file descriptor limit (ulimit -n) must be high enough because every client connection uses one fd.

#Disk and Page Cache

Kafka writes sequentially, so:

• Use SSD/NVMe disks for writes; the page cache handles most reads.
• Don't put the data log and OS on the same disk.
• The page cache is Kafka's biggest asset: frequently read data is cached in RAM automatically by the OS, and zero-copy transfers it to the network without copying through user space. The more RAM, the more hot data the page cache holds.

#Topic Configuration

#Segments and Retention

segment.bytes=1073741824
segment.ms=604800000
retention.ms=604800000
log.flush.interval.messages=100000

segment.bytes=1073741824 (1GB) reduces the number of small segments multiplying index operations; log.flush.interval.messages delays flushing to disk — with replication as the durability guarantee, less frequent flushing actually speeds up writes.

#Compression and Message Format

Make sure message.format.version is aligned with the broker version for the best performance. Consistent compression codecs between producer and broker avoid wasteful re-compression. Continuous monitoring is required: tuning that's right for one workload isn't necessarily right for another.

#Closing

In this episode 21 you've understood producer tuning with batching, compression, and acks, consumer tuning with fetch size and assignment strategy, broker optimization on JVM, OS, and page cache, and topic configuration for throughput and latency.

The key takeaways:

• Batching with batch.size and linger.ms is the biggest throughput lever.
• The page cache and zero-copy are why Kafka is so fast at reading.
• Keep the JVM heap small; use large RAM for the page cache.
• CooperativeStickyAssignor stabilizes rebalances in large groups.
• Segments and retention are set in line with data access patterns.
• Always measure a baseline before and after tuning.

In the next episode 22 we'll monitor the cluster: monitoring and observability. You'll learn JMX metrics, key metrics like under-replicated partitions and consumer lag, Prometheus and Grafana integration, and tools like Burrow for lag monitoring.