Learn Apache Kafka - Installation & Cluster Setup

#Introduction

Now comes the first hands-on part. In episode 0 you already ran a standalone KRaft broker; episode 3 expands that into a complete understanding of installation methods, the difference between ZooKeeper and KRaft, and building a realistic multi-broker cluster.

You will look at three installation paths: binary (TAR/ZIP) for full control, Docker for reproducibility, and managed cloud for production without the operational burden. Most importantly, you will build a multi-broker cluster and verify its health with the official CLI tools.

Before we proceed, it's important to understand the ZooKeeper vs KRaft choice. This is an architectural decision that affects your cluster configuration, and KRaft is the future of Kafka. This episode will guide you through both with the correct practices.

#Installation Methods

#Binary Installation (TAR/ZIP)

The most basic method is to extract the official Apache archive:

curl -O https://downloads.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz
tar -xzf kafka_2.13-3.7.1.tgz
cd kafka_2.13-3.7.1

Inside you'll find bin/ for shell scripts, config/ for configuration files, and libs/ containing all the JARs. This method gives you full control and is suitable for learning or debugging.

#Docker Deployment

For reproducibility and fast testing, Docker is the best choice. The official ecosystem offers apache/kafka for a simple image, or confluentinc/cp-kafka for Confluent images with complete tooling. We'll discuss Docker in depth in episode 29; for now, just know that running Kafka via containers only requires a few environment variables.

#Cloud-Managed Kafka

For production without operations, use Confluent Cloud, AWS MSK, Azure Event Hubs, or Aiven. These services handle upgrades, replication, and monitoring. Episode 30 will cover them in detail; here you just need to know that they all still use the Kafka protocol, so the same client code works.

#ZooKeeper vs KRaft Mode

#Legacy Architecture with ZooKeeper

Before Kafka 3.x, cluster metadata was stored in ZooKeeper: which brokers are alive, who is the leader of each partition, and the topic list. ZooKeeper is a separate consensus system that adds an extra component to operate, and it becomes a source of failure and a scalability bottleneck as clusters grow.

#KRaft (KIP-500) and Migration

KRaft removes ZooKeeper: metadata is managed by a quorum of controllers that use the Raft algorithm. The benefits: simpler deployment, scalable metadata, faster cluster startup, and easier failure handling. KRaft has been production-ready since Kafka 3.3 and becomes the default mode in Kafka 4.x.

#Migration from ZooKeeper to KRaft

Migrating a running ZooKeeper cluster can be done with kafka-storage.sh using special flags:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID \
  -c config/kraft/server.properties

For new clusters, always go straight to KRaft. For old clusters, use mixed mode (ZooKeeper + KRaft) in the 3.x releases as a bridge, then move fully. This entire series uses KRaft.

#Setting Up Your First Cluster

#Single-Node Cluster

The simplest standalone configuration is provided in config/kraft/reconfig-server.properties. In essence, every broker needs node.id, process.roles, listeners, and log.dirs:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kraft-combined-logs

One node acts as both broker and controller. Run it with bin/kafka-server-start.sh config/kraft/reconfig-server.properties.

#Multi-Broker Local Cluster

To learn about replication, build three brokers. Create three configuration files with different node.id values, different listener ports, and different log.dirs:

bin/kafka-server-start.sh config/kraft/server-1.properties &
bin/kafka-server-start.sh config/kraft/server-2.properties &
bin/kafka-server-start.sh config/kraft/server-3.properties &

Each file uses controller.quorum.voters=1@localhost:9093,2@localhost:9094,3@localhost:9095, with one node acting as the active controller and the other two as standby. Make sure all of them use the same KAFKA_CLUSTER_ID from the format step.

#Verifying Cluster Health

Use the CLI to make sure all brokers are registered:

bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

This command shows the API versions each broker supports. If three brokers appear, your cluster is healthy. For a concise listing, kafka-metadata.sh --snapshot --cluster-id ... shows the KRaft metadata snapshot.

#Essential Kafka CLI Tools

Here are the CLI tools that will accompany you throughout the series:

• kafka-topics.sh — create, describe, and delete topics.
• kafka-console-producer.sh — send records from stdin for testing.
• kafka-console-consumer.sh — read records to stdout for testing.
• kafka-broker-api-versions.sh — check broker API versions.
• kafka-cluster.sh — manage KRaft clusters, including format and metadata.
• kafka-configs.sh — alter broker, topic, and user configuration.
• kafka-consumer-groups.sh — manage consumer groups and offsets.
• kafka-acls.sh — manage authorization (covered in episode 17).

Try running the most frequently used ones to test your cluster:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic test --partitions 3 --replication-factor 3
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test

The --describe command shows the partition distribution and leaders across brokers. A healthy output shows all replicas with a complete Isr status and well-spread leaders.

#Closing

In this episode 3 you've built your first Kafka cluster end to end: understood the three installation paths, distinguished ZooKeeper from KRaft, run single-node and multi-broker clusters, verified health with the CLI, and got to know the tools you'll use constantly.

The key takeaways:

• Binary installation gives full control; Docker is for reproducibility; cloud managed is for production.
• KRaft is the modern mode without ZooKeeper, production-ready since 3.3.
• Multi-broker clusters use controller.quorum.voters with one node as the active controller.
• kafka-broker-api-versions.sh and kafka-topics.sh --describe are your health verification gateways.
• Always format storage with kafka-storage.sh format before starting brokers.

In the next episode 4 we'll dive into topics and partitions — topic naming rules, configuration parameters, key-based partitioning strategies, round-robin, custom partitioners, and retention and cleanup policies. The three-broker cluster you just built will be the laboratory!