Learn Apache Kafka - Tiered Storage (KIP-405)

#Introduction

Until now, all Kafka data has been stored on broker disks. Long-term retention meant keeping expensive disks and spreading capacity across many brokers. Tiered storage (KIP-405, introduced as early access in Kafka 3.6+) changes this model: older log segments are offloaded to much cheaper object storage.

The concept is simple: hot data that's newly written and frequently read stays on the broker disks (hot tier), while old, rarely accessed segments are moved to S3, GCS, or Azure Blob (cold tier). Consumers can still read cold segments — the broker fetches them from object storage transparently.

Episode 11 covers the hot and cold tier model, how to configure remote storage, local versus remote retention, and the benefits and trade-offs you need to weigh.

#The Concept of Tiered Storage

#Hot Tier vs Cold Tier

A partition has two storage layers:

• Hot tier: active and closed segments still on broker disks. All writes happen here, and most new reads do too.
• Cold tier: closed segments that have been copied to object storage and released from broker disks. Consumers reading old offsets pull the data from object storage.

Movement between tiers is automatic and asynchronous. The broker remains the only access point for clients — applications don't need to know whether data comes from disk or object storage.

#Offloading Old Segments

Brokers mark closed segments as offload candidates based on their age. A background process copies the segments along with their indexes to object storage, then after successful verification deletes the local copy. If object storage hasn't received a segment yet, the data remains available from the hot tier — there's no availability gap.

#Kafka as Long-Term Storage

With tiered storage, retention is no longer limited by disk cost. You can store years of data at cheap object storage prices and replay the full history whenever needed — something that was previously expensive or impossible. This drives Kafka's use as a long-term system of record.

#Remote Storage Configuration

#Remote Storage Manager

To enable tiered storage, brokers need a RemoteStorageManager implementation. Kafka provides reference implementations for S3, GCS, and Azure Blob that can be configured via server.properties:

remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=io.kafka.shaded.glue.remote.GlueRemoteStorageManager
remote.log.storage.manager.impl.prefix=remote.storage
remote.storage.bucket.name=my-kafka-logs
remote.storage.region=ap-southeast-1

The configuration above uses the AWS Glue storage manager available in Confluent distributions; for open-source Kafka, you write or use a RemoteStorageManager plugin implementing the same interface. The remote.storage.bucket.name value points to the destination bucket, and the region determines the bucket location.

#Local vs Remote Retention

Retention can now be differentiated between local and remote:

local.retention.ms=604800000
remote.retention.ms=-1
log.retention.bytes=-1

local.retention.ms=604800000 limits the data stored on broker disks (7 days), while remote.retention.ms=-1 means segments in object storage never expire. With this combination, brokers hold a week of hot data on disk and store the entire history in the cloud indefinitely.

#Topic-Level Configuration

Tiered storage can also be enabled per topic via kafka-configs.sh:

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name orders \
  --add-config remote.storage.enable=true,local.retention.ms=86400000

Enabling per topic is useful when only part of the data needs long retention — for example an audit topic kept forever, while a metrics topic only needs one day of local retention.

#Benefits and Trade-Offs

#Main Benefits

• Unlimited retention: data is stored for years at low cost, enabling long-term replay and audit.
• Lower broker cost: broker disks can be smaller because they only hold hot data; object storage is far cheaper per GB.
• Easier scalability: capacity is no longer limited by the number of broker disks, reducing the need to add nodes.
• Implicit backup: the object storage copy becomes an additional layer of protection against broker disk failure.

#Trade-Offs to Understand

• Read latency: reading cold segments requires a round-trip to object storage, adding tens to hundreds of milliseconds of latency compared to local disk.
• Ops complexity: you now manage cloud credentials, bucket policies, and additional lifecycles.
• Egress and request costs: object storage operation costs (GET/PUT) can be significant for heavy read-replay workloads.
• Feature maturity: tiered storage is still relatively new; test thoroughly before using it for mission-critical workloads.

#Recovery Scenarios

When a broker loses its disk or needs to reload data, cold segments aren't lost — the data is still in object storage. A new broker can pull those segments whenever needed, so recovery no longer means losing history. This makes tiered storage an important part of the disaster recovery strategy covered in episode 25.

#Closing

In this episode 11 you've understood the hot and cold tier model, how to configure a remote storage manager and local versus remote retention, and the cost benefits and latency trade-offs of tiered storage.

The key takeaways:

• Tiered storage moves old segments from broker disks to object storage.
• The hot tier serves new writes and reads; the cold tier serves replay of old data.
• RemoteStorageManager determines the object storage target (S3, GCS, Azure Blob).
• local.retention.ms limits data on disk; remote.retention.ms governs data in the cloud.
• Tiered storage can be enabled per topic via kafka-configs.sh.
• Reading from the cold tier is slower, so factor in latency and egress costs.

In the next episode 12 we'll discuss Kafka Connect — a data integration framework for moving data between Kafka and other systems. You'll learn standalone and distributed worker architectures, source and sink connectors, converters and SMTs, and deployment via the REST API.