Learn Debezium - Multi-Database & Multi-Cluster CDC

#Introduction

Companies rarely have a single database — usually there's MySQL for transactions, PostgreSQL for other services, MongoDB for catalogs, and a SQL Server left over from the past. Episode 16 covers how Debezium handles many sources at once, both within a single Kafka Connect cluster and across clusters.

There are two kinds of complexity you'll face: operational complexity (managing many connectors running together) and data complexity (different schemas that must coexist in one event ecosystem). Both need a clear strategy.

#Handling Many Source Databases

A single Kafka Connect cluster can run many connectors, one per source database. The key is a unique topic.prefix so output topics don't collide:

{
  "name": "mysql-inventory",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "mysql",
    "topic.prefix": "mysql-db",
    "database.include.list": "inventory"
  }
}
{
  "name": "postgres-orders",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "topic.prefix": "pg-db",
    "schema.include.list": "public"
  }
}

With topic.prefix: "mysql-db" and pg-db, events from MySQL and PostgreSQL live in separate namespaces and won't overwrite each other. Both connectors share the same worker resources, so watch the cluster capacity.

#Multi-Cluster Topologies

As scale grows, separate connectors into their own clusters by domain or region:

cluster-transactional:  mysql-connector, sqlserver-connector
cluster-analytics:      postgres-connector, mongodb-connector

Reasons to separate clusters:

• Blast radius: one connector's failure doesn't disturb other connectors.
• Configuration isolation: converters and transforms differ per cluster.
• Separate scaling: a heavily loaded cluster can be scaled on its own.

To combine events from several clusters, use Kafka MirrorMaker or Cluster Linking as covered in episode 14.

#Configuring for Batch Sources and Heterogeneous Schemas

Different source databases produce different event structures — MySQL uses binlog coordinates, PostgreSQL uses LSN. Before consumers process the events, they need normalization. Normalization strategy:

{
  "transforms": "unwrap",
  "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
  "transforms.unwrap.add.fields": "op,table,source.ts_ms"
}

The ExtractNewRecordState transform strips the schema and payload wrapper, keeping only after plus selected fields — including op, the table name, and timestamp. This way events from MySQL, PostgreSQL, and MongoDB have a consistent shape for downstream consumers.

#Coordination Between Connectors and Consumer Groups

When one consumer wants to read events from many databases at once, it can join a single consumer group reading many topics. Kafka guarantees ordering per partition, not between partitions — so the consumer design must tolerate events arriving out of order across sources.

docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic mysql-db.inventory.customers,pg-db.public.orders \
  --property print.key=true

If cross-source ordering matters — for example synchronizing relations between tables — use source.ts_ms to sort on the consumer side, or consider running a merge transform in Kafka Streams.

#Cross-Database Data Integration Patterns

Common integration patterns:

• Data hub: all databases send events to one Kafka, downstream picks what it needs.
• Consolidation: events from several sources are combined into one unified model.
• Federation: each source stays autonomous, and integration happens at the query or consumer layer.

Consolidation often uses one topic per business concept. With the routing from episode 6, customers events from MySQL and users events from PostgreSQL can be routed into one cdc-customers topic before further normalization.

#Tracking the Origin of Every Event

After events from various sources are combined, you still need to know where they came from. The source field on Debezium events stores the connector, database, table, and original log coordinates. Preserve this metadata during normalization — don't throw it away when unwrapping with ExtractNewRecordState.

Use the add.fields property to copy important metadata to the top level of the payload:

{
  "transforms": "unwrap",
  "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
  "transforms.unwrap.add.fields": "source.connector,source.db,source.table"
}

With add.fields: "source.connector", the origin field is directly available in the payload without unpacking the nested source object. This makes it easy for downstream to filter or group events by their source.

#Conclusion

Episode 16 extends the pipeline's scope: many connectors in one cluster with unique topic.prefix values, cluster separation for isolation and scale, normalization of heterogeneous events, consumer group coordination, and cross-database integration patterns.

The key takeaways:

• One cluster can run many connectors as long as each topic.prefix is unique.
• Multi-cluster gives failure isolation and separate scaling.
• ExtractNewRecordState normalizes events from different sources into one shape.
• Ordering is only guaranteed per partition; sort across sources on the consumer side if needed.
• Choose the data hub, consolidation, or federation pattern based on business needs.

In the next episode, episode 17, we'll discuss custom SMT and connector extensions — writing your own Single Message Transforms, extending Debezium with new transformation plugins, and masking, enrichment, and payload normalization use cases.