Learn Debezium - Performance & Throughput

#Introduction

When database traffic rises, the CDC pipeline must rise with it without becoming a bottleneck. Episode 15 covers how to measure and optimize Debezium throughput: speeding up snapshots, splitting work into parallel tasks, adjusting Kafka partitions, and making sure tuning doesn't disturb a production database serving OLTP systems.

The golden rule to always remember: CDC shares resources with the database's main workload. Aggressive tuning can speed up the pipeline, but it can also squeeze the source database until production applications slow down. Every decision is a trade-off.

#Optimizing Connector Throughput

Streaming throughput is controlled by a combination of worker and connector properties. On the worker side, these three properties determine batch size and queue:

max.batch.size: 2048
max.queue.size: 8192
offset.flush.interval.ms: 5000

With max.batch.size: 2048, the worker processes up to 2048 records per batch. On the connector side, speed up the snapshot with parallel threads:

{
  "snapshot.max.threads": "4",
  "snapshot.fetch.size": "2000",
  "poll.interval.ms": "500"
}

Measure throughput before and after tuning with the JMX metric MaximumLagInMs and records per second. Don't raise values blindly — test incrementally.

#Parallelism, Task Configuration, and Kafka Partitions

Throughput is also limited by topic partitions. A single partition can only be read by one consumer in one group, and events for one row always go to one partition:

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --alter --topic dbserver1.inventory.customers \
  --partitions 12

Adjust tasks.max to match the number of partitions so consumer tasks can use them. However, for source connectors like Debezium, one task per database is normal — true parallelism comes from running several connectors or adding workers, not just raising tasks.

#Tuning the Source Database for a CDC Workload

The source database must be configured so the change log stays available long enough and doesn't become a bottleneck:

For MySQL, make sure the binlog is row-based and doesn't expire too quickly:

server-id = 1
log-bin = mysql-bin
binlog_format = ROW
binlog_row_image = FULL
binlog_expire_logs_seconds = 2592000

With binlog_format = ROW, changes are recorded per row rather than per statement — a Debezium prerequisite. For PostgreSQL, set up wal_level = logical and enough max_replication_slots and max_wal_senders for Debezium slots.

#Minimizing the Impact on OLTP Systems

Resources used by a snapshot or streaming are resources not serving the application. Ways to minimize the impact:

• Schedule snapshots during quiet hours or use incremental snapshots.
• Limit batch size so the load per session is bounded.
• Monitor database metrics such as thread count and binlog throughput during a snapshot.

To control the load, enable more conservative batching limits on production connectors:

{
  "snapshot.fetch.size": "1000",
  "incremental.snapshot.chunk.size": "256",
  "max.batch.size": "500"
}

The value incremental.snapshot.chunk.size: "256" keeps chunks small so the pause between chunks gives OLTP workloads room. Observe database metrics for several minutes after tuning to make sure there are no negative spikes.

#Measuring Before and After Tuning

Without measurement, tuning is just guesswork. Establish a baseline before changing anything:

• Record events per second throughput under normal load.
• Record the maximum streaming latency from JMX metrics.
• Record the source database load — CPU, threads, and I/O.

After changing one property, compare the results against the baseline. If throughput doesn't rise but database load spikes, revert the change and look for another cause — often the bottleneck is in the network or topic partitions, not the worker.

Test every change in staging first, then apply to production one at a time. Small measured changes are easier to track than changing many properties at once.

#Conclusion

Episode 15 gives you a way to measure and optimize performance: batch tuning on the worker, faster snapshots with parallel threads, adjusting partitions and tasks, preparing the database log for CDC, and limiting load so OLTP systems aren't squeezed.

The key takeaways:

• Throughput is controlled by max.batch.size, max.queue.size, and snapshot.max.threads.
• Topic partitions limit consumer parallelism; adjust as needed.
• MySQL needs binlog_format = ROW; PostgreSQL needs wal_level = logical.
• Aggressive snapshots add load to the source database — schedule carefully.
• Measure database and JMX metrics before and after every tuning change.

In the next episode, episode 16, we'll discuss multi-database and multi-cluster CDC — handling many source databases and multi-cluster topologies, connector configuration for heterogeneous schemas, and cross-database data integration patterns.