Learn Apache Kafka - Log Compaction

#Introduction

In episode 4 you already got a brief look at the compact cleanup policy. Now we dissect it fully. Log compaction is Kafka's mechanism for cleaning up logs differently from retention delete: instead of discarding data based on age, compaction discards all old versions of a key and only retains the latest value.

Why does this matter? Imagine the user-profiles topic: every profile update writes a new record with the user_id key. Over time, most of the data in the log is old versions that are no longer relevant. With compaction, Kafka condenses the log into one latest value per user — without losing current information.

Episode 10 covers the concept of compaction with tombstones, the configuration parameters that control its behavior, real-world use cases like changelog topics and materialized views, and how to monitor compaction health.

#The Concept of Log Compaction

#The Compact Cleanup Policy

With cleanup.policy=compact, brokers don't delete records by time; instead they maintain the log so that for every key only the latest value is stored. Older records with the same key are discarded by a background process called the cleaner.

Compaction preserves one snapshot per key while still keeping records with unique keys as long as retention applies. As a result, the amount of data in a compacted topic keeps approaching the number of keys, not the number of writes.

#Tombstones for Deletion

How do you fully delete a key? The answer is a tombstone: a record with a given key and a null value. The cleaner treats a tombstone as a "delete this key" marker and removes that key from the log along with all its previous versions.

Before : (k1,v1) (k1,v2) (k2,v1) (k1,v3) (k2,v2) (k2,null)
After  : (k1,v3) (k2,null)

The tombstone itself doesn't disappear immediately; it's held for delete.retention.ms before being deleted too. This gives consumers time to catch the deletion event.

#Retaining the Latest Value per Key

Note an important nuance: compaction is not a real-time guarantee. Records in an active, still-open segment cannot be compacted. Compaction only works on segments that are already closed and sufficiently "dirty" (containing many stale versions), so the newest data stays intact while the active segment is still receiving writes.

#Compaction Configuration

#Core Parameters

Compaction configuration is set per topic via kafka-configs.sh:

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name user-profiles \
  --add-config cleanup.policy=compact,min.compaction.lag.ms=60000,min.cleanable.dirty.ratio=0.5

Key parameters you need to understand:

• min.compaction.lag.ms: how long a record must at minimum survive before it can be compacted. Protects consumers that are still reading data with a delay.
• max.compaction.lag.ms: the maximum time a record may wait for compaction; ensures data doesn't pile up forever when the dirty ratio is low.
• delete.retention.ms: how long tombstones are retained after compaction.
• min.cleanable.dirty.ratio: the ratio of "dirty" log that must accumulate before the cleaner starts working. A value of 0.5 means the cleaner waits until half the segment contains stale versions.
• segment.ms and segment.bytes: determine when a segment closes, which is the precondition for compaction to work.

#Setting the Balance

cleanup.policy=compact
min.compaction.lag.ms=60000
max.compaction.lag.ms=300000
delete.retention.ms=86400000
min.cleanable.dirty.ratio=0.5

A rule of thumb: a low min.cleanable.dirty.ratio makes compaction more frequent (more CPU) but keeps the log more compact; a high value saves CPU but lets the log grow longer. The combination of min.compaction.lag.ms and delete.retention.ms must be planned so slow consumers don't lose data.

#Log Compaction Use Cases

#Changelog Topics and Materialized Views

Kafka Streams stores state in state stores and replicates it through changelog topics with cleanup.policy=compact. Since only the latest value per key is needed for state recovery, compaction keeps the changelog small. Materialized views built from compacted topics automatically represent the current state.

#State Store Backends

RocksDB and other state stores use compacted changelogs for recovery: when an instance crashes, it reads the changelog from the last checkpoint position. Without compaction, recovery would have to read the entire history; with compaction, only the latest snapshot per key.

#CDC and Data Snapshots

Change Data Capture patterns (episode 27) often write database changes to compacted topics. A single compacted accounts topic represents the latest database table: each key is a primary key, and the latest value is the current state of that row. A data warehouse can be rebuilt by reading the full snapshot at any time.

#Monitoring Compaction

#Cleaner Metrics

Brokers expose metrics for the cleaner process via JMX:

• log-cleaner-clean-time-percent: the percentage of time the cleaner is active; consistently above 50-70 percent signals high compaction load.
• log-cleaner-cleaner-buffer-utilization: cleaner buffer usage; close to 100 percent means the log is too large for the available buffer.
• log-cleaner-dirty-ratio: the average dirty ratio of the log being compacted.

#Compaction Lag and Dirty Ratio

Compaction is said to be lagging when there are old segments that never get cleaned, usually because max.compaction.lag.ms is too large or the cleaner lacks threads. Monitor two things: the number of segments exceeding max.compaction.lag.ms, and a dirty ratio that keeps rising. Both indicate the cleaner can't keep up with the write rate.

bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.log:type=LogCleaner,name=clean-time-percent --report-format=txt

kafka.tools.JmxTool is a built-in tool for reading JMX metrics from the terminal. In production practice, these metrics are more often collected into Prometheus (details in episode 22).

#Closing

In this episode 10 you've understood that log compaction condenses a log into the latest value per key, how tombstones delete keys, configuration parameters like min.compaction.lag.ms, max.compaction.lag.ms, delete.retention.ms, and min.cleanable.dirty.ratio, and the changelog topic, materialized view, state store, and CDC use cases.

The key takeaways:

• Compaction stores the latest value per key, not the full history.
• Tombstones (null values) are the way to delete a key from a compacted log.
• The cleaner only works on closed segments, so active data always stays intact.
• The dirty ratio controls compaction frequency; compaction lag means the cleaner is overwhelmed.
• Kafka Streams changelog topics and state stores depend on compaction.
• compact,delete combines per-key summarization with data age limits.

In the next episode 11 we'll discuss tiered storage (KIP-405) — Kafka's ability to offload old segments to object storage like S3, GCS, and Azure Blob. You'll learn remote storage configuration, the difference between local and remote retention, and the storage cost versus read latency trade-offs.