Learn Debezium - Pre-Requisite Skills & Environment Setup

#Introduction

Welcome to the Learn Debezium series! This series will take you to mastery of Debezium — an open-source distributed platform for change data capture (CDC) — from the conceptual foundations to production readiness. There are 23 episodes in total, arranged into six phases.

But before touching the debezium/connect image, there are a few core skills and software tools you must have. Why are these prerequisites important? Because Debezium is not just a database log reader. It's a source connector that lives inside Kafka Connect, so understanding Kafka, Kafka Connect, data serialization, and how databases record their own changes is a hard requirement for following this series.

Episode 0 is your roadmap: we'll make sure the core skills are in place, set up the environment with Docker, run Kafka in KRaft mode, turn on Kafka Connect together with the Debezium runtime, and perform the first verification. Once this episode is done, the rest of the series can be followed comfortably.

#Core Skills You Must Master

#Relational Databases and the Transaction Log

Debezium works by capturing the changes that databases record in their internal transaction log. Each database has its own way of doing this: MySQL writes a binlog, PostgreSQL uses the WAL (write-ahead log), MongoDB stores oplogs and change streams, while SQL Server has its own change data capture mechanism. You must understand the concepts of tables, primary keys, transactions, and how these logs work, because the entire CDC mechanism depends on them.

#Data Streaming and Event-Driven Architecture

CDC produces an event stream: a flow of events that represents data changes. Understand the basic streaming terms — event, stream, producer, and consumer — as well as the principles of event-driven architecture, in which systems are connected through events rather than direct calls between services. Without this foundation, you'll struggle to understand why Debezium produces events to Kafka in the first place.

#Apache Kafka and Kafka Connect

Kafka is the event store that forms the brain of the CDC pipeline. You must master:

• Topics and partitions: events are grouped into topics that can be split into multiple partitions for scale.
• Offsets: markers that track a consumer's read position within a partition.
• Consumer groups: a set of consumers that share the work of reading a topic.
• Kafka Connect: the framework that bridges data between Kafka and other systems, where Debezium sits as a source connector.

Understand that Kafka Connect abstracts away connection concerns and connector configuration: you only register a connector configuration, and the worker runs it. We'll break down its lifecycle details in episodes 3 and 11.

#Serialization and the Schema Registry

Events in Kafka are stored as byte arrays, so the serialization format determines how events can be consumed. Get to know the three main formats Debezium will use: JSON, Avro, and Protobuf. All three can be paired with a schema registry — Confluent Schema Registry or Apicurio — to manage data structures centrally. We'll cover these formats in detail in episodes 5 and 9.

#Software You Need to Prepare

#Docker and Docker Compose

The fastest way to run the whole stack is Docker. Verify your installation:

docker --version
docker compose version

Both commands above must output a version without errors. Docker Compose will be used to spin up Kafka, Kafka Connect, and the source database all at once.

#Kafka in KRaft Mode

Since Kafka 3.x, Zookeeper is no longer required — Kafka can run in KRaft mode, where a broker also acts as the cluster controller. This significantly simplifies the setup. We'll use the apache/kafka image, which supports KRaft out of the box, without worrying about a separate Zookeeper cluster.

#Kafka Connect and the Debezium Runtime

Debezium provides the official quay.io/debezium/connect image, which bundles a Kafka Connect worker complete with all the Debezium connectors in its plugin directory. This image will be the main worker throughout the series:

services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  connect:
    image: quay.io/debezium/connect:3.0
    ports:
      - "8083:8083"
    depends_on:
      - kafka
    environment:
      BOOTSTRAP_SERVERS: kafka:9092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect-configs
      OFFSET_STORAGE_TOPIC: connect-offsets
      STATUS_STORAGE_TOPIC: connect-status

Important note: the value BOOTSTRAP_SERVERS: kafka:9092 uses the service name inside the Docker network. Kafka Connect talks to the broker through that internal network, while port 8083 is exposed for REST access from the host.

#Source Database and Schema Registry

To observe changes, you'll need a source database. Throughout the series we'll use MySQL and PostgreSQL as our main examples, and touch on MongoDB and SQL Server in episode 3. Also prepare a Schema Registry — Apicurio or Confluent — which is optional at first, but required when we use the Avro and Protobuf formats in episodes 5 and 9.

#Hardware Minimum Requirements

This stack runs as several containers at once: a Kafka broker, a Kafka Connect worker, the source database, and later a schema registry. Recommended minimums for a local environment:

• 8GB RAM or more, because the Kafka and Connect JVMs are fairly memory-hungry.
• Quad-core CPU so the broker, worker, and database don't compete with each other.
• 20GB+ storage for Docker images, Kafka logs, and database data.

If your machine is below these specs, reduce the number of containers running at once or temporarily use a cloud service for Kafka.

#Verifying the Environment

Start the stack and verify that Kafka Connect is already responding:

docker compose up -d
sleep 15
curl -s http://localhost:8083/connectors

The command curl -s http://localhost:8083/connectors should return [] — the still-empty connector list. That's a healthy sign. Next, make sure the Debezium plugins are picked up by the worker:

curl -s http://localhost:8083/connector-plugins

Among that list you'll see classes such as io.debezium.connector.mysql.MySqlConnector and io.debezium.connector.postgresql.PostgresConnector. The presence of these classes means the Debezium runtime is ready to use.

#Conclusion

In episode 0 you've set the foundation for the entire series: understanding the concepts of transaction logs and CDC, data streaming, Kafka and Kafka Connect, setting up a Docker environment with Kafka in KRaft mode, starting the Debezium runtime, and verifying the available plugins.

The key takeaways:

• CDC works by reading the database transaction log, not by polling.
• Debezium is a source connector inside Kafka Connect that produces events to Kafka.
• Kafka 3.x runs without Zookeeper using KRaft mode.
• The quay.io/debezium/connect image bundles the worker and all Debezium plugins.
• First verification: curl localhost:8083/connectors should return an empty list.

In the next episode 1 we'll discuss history, background, and why CDC — from the evolution of batch ETL toward event streaming, the birth of Debezium from Red Hat through to becoming an Apache project, and real use cases such as database replication, analytics, and auditing. Make sure your environment is ready, because the Learn Debezium journey has just begun!