Learn Debezium - Filtering, Routing, & Topic Design

#Introduction

By default, Debezium captures every accessible database and table and sends them all to topics with automatic naming. That's convenient at small scale, but in production you often need only a subset of tables, or you want to control topic naming and distribution so consumers can work comfortably.

Episode 6 covers three main tools: filtering to select tables and columns, routing to change topic destinations, and topic design for healthy naming and partitioning. By the end of this episode you'll use Single Message Transforms to assemble this logic.

#Including and Excluding Tables and Databases

Filtering starts at the connector level. The include and exclude properties work as a whitelist and blacklist:

{
  "database.include.list": "inventory",
  "table.include.list": "inventory.customers,inventory.orders",
  "column.include.list": "inventory.customers.id,inventory.customers.email"
}

The combinations you need to understand:

• database.include.list restricts the databases being captured.
• table.include.list and table.exclude.list control tables — the include list takes precedence.
• column.include.list and column.exclude.list control columns, useful for hiding sensitive columns.

Prefer table.include.list over exclude when the number of target tables is small, because a whitelist is easier to predict when new tables appear.

#Routing Events to Different Topics

Sometimes you want to combine several tables into one topic, or change the topic name format. RegexRouter is the built-in Kafka Connect transform used most often:

{
  "transforms": "route",
  "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
  "transforms.route.regex": "(.*)\\.inventory\\.(.*)",
  "transforms.route.replacement": "cdc-$2"
}

With the configuration above, events from dbserver1.inventory.customers are routed to topic cdc-customers, and events from dbserver1.inventory.orders to cdc-orders. This pattern is very useful for combining events from many tables or removing prefixes you don't need.

Debezium also provides TopicRouting, which moves events between topics based on payload content — ideal for dynamic routing based on column values.

#Topic Naming and Partitioning Strategies

Topic naming determines the consumer experience and load balance. Recommended rules:

• Use a system or domain name as the topic prefix, for example orders, rather than the physical database name.
• One table, one topic is the easiest default to manage.
• For large tables, add partitions so throughput is distributed across multiple consumers.

Partitions are determined when the topic is created:

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic cdc-customers \
  --partitions 6 --replication-factor 1

Debezium uses the row's primary key as the message key, so events for the same row always land in the same partition and stay in order. When adding partitions, remember that ordering is only guaranteed within a single partition, not across partitions.

#Using Transforms and Custom Routing Logic

In addition to routing, several built-in transforms round out topic design:

{
  "transforms": "filter",
  "transforms.filter.type": "org.apache.kafka.connect.transforms.Filter",
  "transforms.filter.predicate": "isTruncate",
  "predicates": "isTruncate",
  "predicates.isTruncate.type": "org.apache.kafka.connect.transforms.predicates.TopicNameMatches",
  "predicates.isTruncate.pattern": "dbserver1.inventory.audit.*"
}

Common combinations used in production:

• Filter drops unwanted events, for example truncate events or audit tables.
• ValueToKey replaces the message key with a specific column for repartitioning needs.
• InsertField adds static metadata to the payload, useful for marking an environment.

When chaining several transforms, pay attention to order: the transforms list is executed in comma-separated order, and transforms are optional — without the list, events are sent as-is.

#Predicates for Advanced Routing Conditions

In addition to matching topic names, Kafka Connect supports predicates to execute a transform only under certain conditions — for example, only when the source comes from a specific database or when a header has a particular value. Combining predicates and transforms makes routing logic very expressive without writing custom code.

{
  "transforms": "drop",
  "transforms.drop.type": "org.apache.kafka.connect.transforms.Filter",
  "transforms.drop.predicate": "isArchive",
  "predicates": "isArchive",
  "predicates.isArchive.type": "org.apache.kafka.connect.transforms.predicates.RecordIsTombstone"
}

With RecordIsTombstone, the transform discards tombstones before they reach the consumer. This predicate matters when downstream consumers aren't ready to handle a null value.

#Conclusion

Episode 6 completes your toolkit for designing event flows: connector-level filtering limits the scope, RegexRouter controls topic destinations, naming and partitioning keep the load balanced, and transforms complete the custom routing logic.

The key takeaways:

• Include and exclude lists act as whitelists and blacklists at the database, table, and column levels.
• RegexRouter changes topic names with a regex and replacement pattern.
• The primary key becomes the message key, so per-row ordering is preserved within one partition.
• The transforms list executes in the order it's defined.
• Always design topic naming before adding partitions or routing.

In the next episode, episode 7, we'll discuss monitoring, handling failures, and debugging — monitoring connector status and lag, reading logs and error handling, and building alerts for connector failures.