Learn Debezium - Core Concepts & Debezium Architecture

#Introduction

Episode 1 gave you the reasons why CDC and Debezium were born. Now it's time to take the engine apart. Episode 2 covers how Debezium works from an architectural perspective: who runs the connectors, where events are sent, how the read position is recorded, and how data is wrapped before it reaches Kafka.

The three key components you'll encounter are the connector (the logic that captures changes), Kafka Connect (the runtime that executes connectors), and Kafka topics (the final destination of events). Understanding these three from the start will make debugging in later episodes much easier.

#The Debezium Architecture from Top to Bottom

#Connectors, Kafka Connect, and Kafka Topics

One connector represents one source database. When a connector is run, it's split into one or more tasks that run inside Kafka Connect workers. Each task reads changes from the database and produces events to Kafka topics named topic.prefix.database.table.

database → Debezium connector → Kafka Connect worker → Kafka topics
                                                          ↓
                                        consumer groups + downstream systems

Because Debezium lives inside Kafka Connect, the entire Kafka Connect infrastructure determines its behavior: where offsets are stored, the commit frequency, and the converters in use. An example of the worker properties you'll often see:

bootstrap.servers: kafka:9092
group.id: 1
key.converter: org.apache.kafka.connect.json.JsonConverter
value.converter: org.apache.kafka.connect.json.JsonConverter
offset.storage.topic: connect-offsets
config.storage.topic: connect-configs
status.storage.topic: connect-status

Note that Kafka Connect needs three internal topics: config, offset, and status. These topics are what allow workers in a cluster to share configuration, read positions, and status across connectors.

#The Role of Kafka Topics as Output

Each single-row change produces one event sent to a topic based on the topic.prefix naming rule plus the database and table names. If your topic.prefix is dbserver1, then the customers table in the inventory database produces the topic dbserver1.inventory.customers. This convention matters because consumers can predict topic names without reading the configuration.

#How Debezium Reads the Change Log

Debezium uses the log-based capture approach: it doesn't re-execute queries; instead, it parses the change records the database already writes. Each database uses a different mechanism:

From that physical log, Debezium reconstructs the logical changes — which rows were added, modified, or removed, along with the values before and after. This is the main advantage of CDC: change details are obtained directly from the database without injecting additional load.

#Offsets, Heartbeats, and Snapshots

#Offset Storage and Recovery

To avoid reading the same data twice, Debezium stores offsets — markers of the last position in the database log — into the Kafka Connect offsets topic. When the worker restarts, the connector resumes from the last position, not from the beginning. This is what makes a CDC pipeline tolerant of restarts:

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

In that list you'll see connect-offsets, connect-configs, and connect-status alongside the data topics. Offsets are stored as compressed bytes, so don't expect to read their contents as plain text.

#Heartbeats and Snapshots

A snapshot is the mechanism by which Debezium reads all the data that already exists before it starts streaming. The snapshot.mode value determines its behavior: initial takes a snapshot then continues streaming, while initial_only stops after the snapshot completes. Meanwhile, a heartbeat is a periodic event sent to keep the read position recorded while the database is quiet, so lag doesn't spike when traffic picks up again:

heartbeat.interval.ms: 5000
snapshot.mode: initial
snapshot.fetch.size: 1024

With heartbeat.interval.ms: 5000, Debezium sends a marker every five seconds even when there are no data changes. This keeps the offset fresh and simplifies monitoring in episode 7.

#Schema Registry and Payload Formats

A Debezium event is not just raw data values. By default, an event is wrapped into two parts: a schema describing the structure, and a payload holding the actual values. The format is controlled by the converter installed on the worker.

Without a schema registry, you use the built-in JSON converter. For production environments, many teams switch to Avro or Protobuf, backed by a schema registry to manage the versions of data structures:

key.converter: io.confluent.connect.avro.AvroConverter
value.converter: io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url: http://schema-registry:8081
value.converter.schema.registry.url: http://schema-registry:8081

With schema.registry.url, each event carries a schema ID reference instead of the full schema. Payloads become smaller, and consumers can validate data structures before processing them. We'll cover these formats thoroughly in episodes 5 and 9.

#Each Connector's Internal Topics

Besides the three internal Kafka Connect topics, some Debezium connectors require additional topics. The MySQL connector uses a schema history topic to store table structure history, and the PostgreSQL and MongoDB connectors require replication slots on the source side. These internal topics and slots must be maintained as carefully as ordinary data topics — losing them can force the connector to re-snapshot from scratch.

Make sure internal topics aren't subject to overly short retention policies. If the schema history topic retention is trimmed, the connector can no longer reconstruct old schemas when offsets are reset, and recovery fails.

#Conclusion

Episode 2 gave you the Debezium architecture map: connectors read the transaction log, Kafka Connect executes and manages their lifecycle, events are sent to topics using a standard naming rule, offsets preserve the read position across restarts, snapshots fill in the initial data, and the schema registry protects payload structure.

The key takeaways:

• Debezium is a source connector that runs inside Kafka Connect, not a standalone application.
• Every event is sent to a topic following the topic.prefix.database.table pattern.
• Offsets are stored in an internal Kafka Connect topic so restarts don't lose the read position.
• Snapshots fill in the initial data; heartbeats keep offsets fresh while the database is quiet.
• The payload format is determined by the converter, and Avro or Protobuf requires a schema registry.

In the next episode 3 we'll do the installation and setup of your first connector: running MySQL as the source, registering a Debezium connector through the REST API, then verifying the output topic and the first CDC message. Have the Docker you verified in episode 0 ready.