Learn Apache Kafka - Pre-Requisites Skills & Environment Setup

#Introduction

Welcome to the Learn Apache Kafka series! This series will take you to mastery of Apache Kafka — the distributed event streaming platform used by thousands of companies to process real-time data — from conceptual foundations to production readiness. In total there are 36 episodes organized into six phases.

Kafka is not just an ordinary message queue. It is a distributed system that stores events in a replayable commit log, with replication, partitioning, and high throughput. Because of this complexity, there are several foundational skills and software prerequisites you must have before you begin.

Episode 0 is your roadmap: we'll make sure your foundational skills are in place, set up the JDK, download Apache Kafka, prepare Docker, and run the first verification. Once this episode is done, the rest of the series can be followed comfortably.

#Foundational Skills You Must Master

#Distributed Systems and Networking Concepts

Kafka is a distributed system, so you need to understand basic concepts such as nodes, clusters, replication, consistency, and fault tolerance. Also understand basic networking: TCP/IP, ports, and DNS, because Kafka communicates between brokers and with clients over the TCP protocol.

#Pub/Sub Messaging and Data Serialization

Understand the publish/subscribe pattern: there is a producer that sends messages and a consumer that receives them. Get familiar with terms like queue, topic, and event. Equally important is data serialization — JSON, Avro, and Protobuf — because Kafka messages are just byte sequences whose format both sides must agree on.

#Java, CLI, and Async Programming

Kafka runs on the JVM, so you should be comfortable with Java version 11 at minimum. Also get used to the command line interface and the concept of async programming: message delivery in Kafka is asynchronous and uses callbacks, unlike typical synchronous function calls.

java -version

The output should show version 17 or 11. If not, install an LTS JDK first.

#Software and Tools to Prepare

#Java Development Kit (JDK) 11 or 17

Apache Kafka requires a JDK. For this series, use JDK 17 LTS:

sudo apt update
sudo apt install openjdk-17-jdk -y

After installation, verify with java -version and javac -version.

#Apache Kafka Binary Release

Download the latest release from the official Apache site, then extract it:

curl -O https://downloads.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz
tar -xzf kafka_2.13-3.7.1.tgz
mv kafka_2.13-3.7.1 ~/kafka
cd ~/kafka

Kafka 3.7.1 is a stable release that supports KRaft mode without ZooKeeper. Make sure the ~/kafka/bin directory contains CLI tools such as kafka-topics.sh and kafka-console-producer.sh.

#Docker and Docker Compose

In later episodes you will run Kafka along with its ecosystem (Schema Registry, Kafka Connect, ksqlDB) in a containerized way. Set up Docker now:

docker --version
docker compose version

#Supporting Tools

Also prepare kcat (formerly kafkacat) for quick testing, Offset Explorer or Conduktor as a GUI, and the SDK for your language of choice: Java, Python (confluent-kafka), Go (sarama), or Node.js (kafkajs). For development, you'll want at least 4GB of RAM (8GB recommended), 20GB of free storage, and a quad-core CPU.

#Verifying Your Environment

#Make Sure PATH and Executable Scripts Are Set Up

Add the Kafka directory to your PATH so the CLI commands are easy to access:

export PATH="$HOME/kafka/bin:$PATH"
echo 'export PATH="$HOME/kafka/bin:$PATH"' >> ~/.bashrc
kafka-topics.sh --version

Output 3.7.1 means the CLI tools are ready to use. The kafka-topics.sh script is the main gateway for managing topics, and we'll use it in almost every episode.

#Run Your First Single-Node Cluster

The most convincing verification is running a one-node KRaft broker. Kafka provides a starting configuration for this:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/reconfig-server.properties
bin/kafka-server-start.sh config/kraft/reconfig-server.properties

If the broker starts without errors and shows the log started (kafka.server.KafkaRaftServer), your environment is ready. The kafka-storage.sh format command creates the KRaft metadata log, and kafka-server-start.sh runs the broker.

#Summary of Skills You Must Master

A summary of the prerequisites you've prepared in episode 0:

• Java JDK 17 installed and verified.
• Apache Kafka 3.7.1 extracted and CLI tools on PATH.
• Docker + Docker Compose ready for containerized episodes.
• Basic concepts: distributed systems, pub/sub, serialization, and async.
• First KRaft cluster successfully run as a verification.

If anything is missing, stop and complete it before continuing. A strong foundation will make the next 35 episodes feel much lighter.

#Closing

In this episode 0 you've laid the groundwork for the entire series: understanding the foundational skills of distributed systems and pub/sub, setting up the JDK, downloading Kafka 3.7.1, configuring PATH, and running a KRaft broker for the first time.

The key takeaways:

• Kafka is a distributed event streaming platform that requires an understanding of distributed systems.
• JDK 11 or 17 is an absolute prerequisite because Kafka runs on the JVM.
• Kafka 3.7.1 supports KRaft mode, so it doesn't need ZooKeeper.
• Always verify your installation by running a broker before you start coding.
• Docker is needed for the containerized episodes in later phases.

In the next episode 1 we'll discuss the history, background, and why you need event streaming — from LinkedIn's data pipeline challenges, the birth of Kafka in 2011, to its comparison with RabbitMQ, Kinesis, and Pulsar. Make sure your environment is ready, because the Learn Apache Kafka journey is just getting started!