Learn Debezium - Integration with Consumers & Sinks

#Introduction

CDC events flowing into Kafka are useless if they just sit in topics. Episode 8 covers the consumption side: how applications read events, how stream processors process them in real time, and how sink connectors deliver data to other systems such as databases, Elasticsearch, and data warehouses.

There's one fundamental difference to understand from the start: the source connector (Debezium) produces, while the sink connector consumes. This pattern makes Kafka a hub — data changes once at the source, then is distributed to many destinations without adding load to the source database.

#Connecting CDC Events to Kafka Consumers

The simplest consumer is the console consumer for quick inspection:

docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic dbserver1.inventory.customers \
  --from-beginning \
  --property print.key=true

For real applications, consumers use libraries like the Kafka client, with the same principle: read key and value, check the op field, then act according to the operation. Because Debezium uses the primary key as the message key, applications can directly map events to records in the target database.

#Stream Processing with ksqlDB and Kafka Streams

Instead of writing consumer logic from scratch, you can use ksqlDB, which provides SQL for stream processing:

Defining a stream from a CDC topic

CREATE STREAM customers_stream (
  id INT,
  first_name VARCHAR,
  last_name VARCHAR,
  email VARCHAR
) WITH (
  KAFKA_TOPIC = 'dbserver1.inventory.customers',
  VALUE_FORMAT = 'json'
);

Note that CDC topics carry a schema and payload wrapper, so the VALUE_FORMAT = 'json' structure above is simplified — in production you use the Unwrap SMT first to strip the wrapper before a SQL consumer processes it.

Another strong alternative:

• Kafka Streams: a Java/Scala library for stateful stream processing applications.
• Apache Flink: a distributed streaming engine for heavy windowing and stateful processing.
• Apache Spark: suitable for micro-batches and heavier analytics.

All of them can read Debezium topics directly as an event source.

#Sink Connectors for Databases and Data Lakes

To stream data out of Kafka, register a sink connector. An example JDBC sink to PostgreSQL:

{
  "name": "jdbc-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "tasks.max": "1",
    "topics": "dbserver1.inventory.customers",
    "connection.url": "jdbc:postgresql://postgres:5432/warehouse",
    "connection.user": "warehouse",
    "connection.password": "warehouse",
    "insert.mode": "upsert",
    "pk.fields": "id",
    "auto.create": "true",
    "auto.evolve": "true"
  }
}

With insert.mode: "upsert", the sink overwrites rows based on the primary key — exactly matching CDC event semantics. Other common sink destinations:

• Elasticsearch for search and observability.
• Data lakes like S3 with Parquet or Avro format.
• Snowflake and other data warehouses for analytics.

#Use Cases: Materialized Views, Audit Logs, and Analytics Pipelines

Combining source and sink produces a powerful architectural pattern:

• Materialized views: ksqlDB or Kafka Streams aggregates CDC events into an always-up-to-date current view.
• Audit logs: every event with before and after is stored in full as a change history.
• Analytics pipeline: CDC events stream into the warehouse so reports reflect current data, not yesterday's batch.

MySQL → Debezium → Kafka topic → ksqlDB / Kafka Streams
                                       ├── materialized view
                                       ├── JDBC sink → warehouse
                                       └── Elasticsearch sink → search

The key to this pattern's success is keeping events append-only in Kafka, so the source of truth isn't split and every destination can read from the same point.

One important note: because Kafka guarantees ordering per partition, not per key across partitions, make sure the topic partition count matches the sink's needs. Sink consumers that require ordering must read the partition where the row key lives.

#Conclusion

Episode 8 closes the consumption loop: application consumers read events directly, stream processors handle them in real time, and sink connectors deliver data to databases, search, data lakes, and warehouses — all from the same event stream.

The key takeaways:

• Source connectors produce events; sink connectors consume and write to other systems.
• ksqlDB and Kafka Streams enable real-time processing without adding load to the source database.
• A JDBC sink in upsert mode mirrors CDC semantics well.
• CDC events are append-only in Kafka, forming a single source of truth.
• Materialized views, audit logs, and analytics pipelines are the primary use cases for this pattern.

In the next episode, episode 9, we'll discuss schema registry and data contracts — building data contracts for event schemas, managing versioning and compatibility rules, handling schema evolution on Debezium payloads, and consumer validation with Avro and Protobuf.