Learn Apache Kafka - Core Concepts & Architecture of Apache Kafka

#Introduction

Now that you understand why Kafka exists, it's time to understand what Kafka is structurally. Episode 2 is the most important conceptual foundation in the entire series: if you understand events, topics, partitions, offsets, and consumer groups, all the following episodes simply deepen the details of the same concepts.

Kafka is designed as a distributed commit log. That means data is stored as a sequence of records appended to a log, distributed across many nodes, and replicated to be resilient against failures. This paradigm is far different from relational databases, which organize data in tables that can be updated in place.

We'll dissect the structure of an event, how the log is split into partitions, how replicas maintain availability, the role of the controller broker in KRaft, and the segment file-based storage format. Take this seriously, because all of these terms will be used again and again in the next 33 episodes.

#Core Kafka Concepts

#Event (Record)

The basic unit of Kafka is called an event or record. An event is a fact that happens in the world: "user A clicked the checkout button at 10:00". Internally, a record contains:

• Key: optional, used to determine the partition and preserve ordering per key.
• Value: the main payload of the event, as serialized bytes.
• Timestamp: the time the event was produced or stored.
• Headers: optional metadata in key-value form.
• Offset: the position of the record within the partition, assigned automatically by the broker.

Events are immutable — once written, they cannot be changed. An update is simply writing a new event.

#Topics and Partitions

A topic is a category where events are published, for example orders or user-events. A topic is split into one or more partitions. Each partition is an ordered log that is appended sequentially:

[0] [1] [2] [3] [4] ...  <- offset

Partitions provide parallelism: producers can write to multiple partitions simultaneously, and a consumer group can read different partitions in parallel. A key point: ordering is only guaranteed within a single partition, not across partitions.

#Producers, Consumers, Brokers, and Clusters

A producer is an application that writes events to a topic. A consumer is an application that reads events from a topic. Both talk to a broker — a Kafka server that stores and serves data. Several brokers together form a cluster; each broker manages some partitions of various topics.

#Kafka Architecture

#Distributed Commit Log and Partition Leadership

Each partition is replicated to several brokers. One broker becomes the leader for that partition; the rest are followers. All writes and reads go through the leader, while followers replicate the data asynchronously. If the leader dies, one of the followers is elected as the new leader.

Each partition also maintains a list of ISR (In-Sync Replicas) — followers that are still in sync with the leader. Replicas that fall behind are removed from the ISR, and rejoin once they've caught up.

#Controller Broker and Metadata Management

Modern Kafka uses KRaft (KIP-500): cluster metadata — the list of brokers, topics, partitions, and leaders — is managed by a quorum of controller brokers using the Raft algorithm. One active controller leads, the rest are standby. This replaces the role of ZooKeeper in older Kafka versions.

The controller is responsible for deciding leader election, processing topic creation, and coordinating partition reassignment. If the active controller fails, the quorum elects a new controller automatically.

#Write Path and Read Path

When a producer sends a record, the broker writes it to the leader's partition log, then waits for follower acknowledgment according to the acks=all setting on the producer side. A consumer reads from the last offset recorded in its consumer group, and the broker returns the next batch of records. The read path leverages zero-copy, so data doesn't need to be copied repeatedly between the kernel and the application.

#Storage Architecture

#Log-Structured Storage and Segment Files

Each partition is stored as a directory containing a series of segment files. A segment is a .log file containing the actual records, with an .index (offset to byte position) and .timeindex (timestamp to offset) index. Closed (no longer active) segments become targets for retention and compaction:

ls /tmp/kraft-combined-logs/orders-0/
00000000000000000000.log
00000000000000000000.index
00000000000000000000.timeindex

The file name is the offset of the first record in that segment. The broker writes to the active segment until the segment size is reached or the segment time passes, then creates a new segment.

#Retention Policies and Log Compaction

Retention determines how long data is kept. Two modes are supported: delete, which removes segments older than retention.ms or exceeding retention.bytes, and compact, which keeps only the latest value per key (full details in episode 10). Kafka's default mode is delete.

#Offsets and Consumer Groups

#Offset as a Read Position

Each record within a partition has a unique offset, marking its position in the log. A consumer that stores the last processed offset can resume from that position after a restart — this is what enables replay and resume without data loss.

#Consumer Groups for Parallelism

A set of consumers sharing a group.id forms a consumer group. The partitions of a topic are divided evenly among the group members: if there are 6 partitions and 3 consumers, each consumer gets 2 partitions. If one consumer dies, its partitions are reassigned to other members — a process called rebalance.

Topic orders (6 partition)
consumer A -> partition 0,1
consumer B -> partition 2,3
consumer C -> partition 4,5

The number of consumers must not exceed the number of partitions, because at any time one partition can only be read by one consumer within a group. Offset management details and assignment strategies will be covered in episode 6.

#Closing

In this episode 2 you've built Kafka's conceptual map: events as immutable records, topics split into ordered partitions, brokers and clusters with leader-follower replication, KRaft controllers for metadata, segment file-based storage, and offsets and consumer groups as the mechanism for parallel consumption.

The key takeaways:

• Kafka is a distributed commit log: records are appended, not updated.
• Ordering is only guaranteed per partition, with the offset as each record's unique position.
• Every partition has a leader and ISR; the leader serves all writes and reads.
• The controller broker (KRaft) manages metadata and leader election without ZooKeeper.
• Storage is segment file-based: .log, .index, .timeindex.
• Consumer groups divide partitions among their members for parallelism.

In the next episode 3 we'll put these concepts into practice: installation and cluster setup — from binary installation, the ZooKeeper vs KRaft comparison, building single-node and multi-broker clusters, to getting to know all the Kafka CLI tools that will accompany you throughout the series. Get your terminal ready, because hands-on starts now!