Learn Apache Kafka - Change Data Capture (CDC) with Debezium

#Introduction

Many legacy systems store the truth in relational databases, while modern applications need that data moving in real-time. Change Data Capture (CDC) is the bridge: every database change — insert, update, delete — is published as an event to Kafka.

Debezium is an open-source CDC framework built on Kafka Connect. It reads the database transaction log and streams changes to topics without applications having to write twice. Episode 27 covers Debezium's architecture, connectors for various databases, snapshot modes, filtering and transformations, schema change handling, and monitoring practices.

#CDC Basics and Debezium Architecture

#Streaming Database Changes

CDC eliminates the need for applications to write events manually. Instead, Debezium observes the actual source of changes — the transaction log (binlog in MySQL, WAL in PostgreSQL, oplog in MongoDB) — and publishes every change as a Kafka record. Even changes missed by applications are captured.

#Debezium Architecture

Debezium runs as a source connector in Kafka Connect (episode 12):

{
  "name": "debezium-mysql-orders",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "db.internal",
    "database.user": "debezium",
    "database.dbname": "shop",
    "topic.prefix": "db",
    "snapshot.mode": "initial",
    "include.schema.changes": "true"
  }
}

connector.class determines the database driver, and topic.prefix becomes the topic name prefix — the connector above writes to the db.shop.orders topic for the orders table. snapshot.mode=initial means the connector takes an initial snapshot then continues with real-time changes.

#Database Support

Debezium supports MySQL, PostgreSQL, MongoDB, Oracle, SQL Server, and others. Each connector needs a database user with rights to read the transaction log. The logging position is stored in Kafka (the offset topic), so the connector can resume from its last position after a restart.

#Working with Connectors

#Table Inclusion and Exclusion

By default all tables in the database are connected. Restrict with table.include.list and column.exclude.list so topics don't fill with unimportant data:

{
  "name": "debezium-mysql-orders",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.dbname": "shop",
    "topic.prefix": "db",
    "table.include.list": "shop.orders,shop.order_items",
    "column.exclude.list": "shop.orders.credit_card"
  }
}

table.include.list limits to only the desired tables, and column.exclude.list prevents sensitive columns like credit_card from leaking into Kafka — important for data compliance (episode 33).

#Transformation Options

Debezium can combine transforms: add metadata, remove fields, or simplify the record structure. For many cases, Debezium records need cleaning before downstream use — for example dropping the heavy source part and keeping only after as the payload.

#CDC Patterns

#Real-Time Synchronization

The most common pattern: a production database replicates changes to a data warehouse, search index, or cache in real-time. CDC keeps secondary data always close to the primary without batch ETL. The same data can also feed analytics models and machine learning systems.

#Building Data Lakes

With CDC, all table changes stream to object storage (via a sink connector) forming a parquet-format data lake. Because every change is stored as an event, you have the full history — not just the latest snapshot.

#Microservices Integration and Legacy Modernization

Microservices can share legacy database data without touching legacy code: Debezium reads changes from the monolith's database and publishes events for new services. This is also the main path for the outbox pattern (episode 26) and gradual modernization.

#Best Practices

#Handling Schema Evolution

Database schemas change over time — columns are added, types change. Debezium handles this by sending schema change events (when include.schema.changes=true). Use a Schema Registry (episode 7) with backward/full compatibility so old consumers don't break when schemas change. Always test schema changes in staging before production.

#Monitoring CDC Lag

CDC falls behind if the connector is slow or the database releases the log slowly. Monitor:

• replication lag: the time difference between a database change and its write to Kafka.
• topic lag on the consumer side (episode 22).
• connect task state: a failed connector triggers retries; monitor status via the REST API.

curl -s http://localhost:8083/connectors/debezium-mysql-orders/status

curl -s http://localhost:8083/connectors/debezium-mysql-orders/status shows the connector and task state — RUNNING means healthy, FAILED needs immediate action.

#Error Handling and Performance Tuning

• Configure errors.tolerance and a dead letter queue (episode 12) for problem records.
• Set the connector's max.batch.size and poll.interval.ms so throughput matches the target.
• Monitor how fast the connector processes events; if it slows, check the source database and worker resources.
• Give the connector's database user only the minimum rights needed (least privilege).

#Closing

In this episode 27 you've understood CDC with Debezium: the database change streaming flow, connector architecture in Kafka Connect, snapshot modes, table and column filtering, synchronization and data lake patterns, and schema evolution, monitoring, and error handling best practices.

The key takeaways:

• Debezium reads the database transaction log and publishes changes to Kafka.
• Every change becomes an event: inserts, updates, and deletes are captured automatically.
• Table and column filtering keeps topics relevant and free of sensitive data.
• CDC powers data lakes and microservices integration without changing legacy systems.
• Schema changes must go through a Schema Registry so consumers don't break.
• Monitor CDC lag and connector status as part of observability.

In the next episode 28 we'll bring Kafka to Kubernetes: deployment with Strimzi. You'll learn the operator pattern, CRDs like Kafka and KafkaConnect, StatefulSets, automatic TLS management, and production considerations on K8s.