Learn Debezium - Distributed Deployment & High Availability

#Introduction

So far we've been running a single Kafka Connect worker — a single point of failure. If that worker dies, all connectors stop with it. Episode 12 covers running a Kafka Connect cluster with multiple workers, so the failure of one worker doesn't stop the pipeline and load can be shared between workers.

Distributed mode is Kafka Connect's default behavior when you're not using standalone mode. All decisions — task scheduling, load distribution, and recovery — are made automatically by the cluster using group coordination in Kafka.

#Running a Kafka Connect Cluster in Distributed Mode

The key to a cluster is multiple workers sharing the same group.id, bootstrap.servers, and three internal topics. Add a second worker in compose:

  connect-2:
    image: quay.io/debezium/connect:3.0
    ports:
      - "8084:8083"
    environment:
      BOOTSTRAP_SERVERS: kafka:9092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect-configs
      OFFSET_STORAGE_TOPIC: connect-offsets
      STATUS_STORAGE_TOPIC: connect-status

Note the GROUP_ID: 1 matching the first worker. With the same group id, both workers join one cluster and share the work.

Verify from the REST API — both workers serve the same endpoints because coordination happens inside the cluster:

curl -s http://localhost:8083/connectors

#High Availability and Fault Tolerance

When one worker dies, tasks running on that worker are automatically moved to another worker. This recovery happens through Kafka Connect's group rebalancing mechanism:

worker-1 (task 0, 1) ── mati ──► rebalance
worker-2 (task 2)    ──────────► worker-2 (task 0, 1, 2)

Because offsets are stored in a shared topic, a moved task continues from its last position — no data is re-read from the beginning. This is the advantage of storing offsets in Kafka rather than on the worker's local disk.

For extra redundancy on the source side, run the connector in a different cluster with a separate snapshot, or use heartbeat so the read position stays fresh during failover.

#Balanced Workloads and Worker Scaling

Debezium distributes work according to the number of tasks. The tasks.max configuration determines how many tasks can be created, and the cluster spreads them across all workers:

{
  "tasks.max": "4"
}

With tasks.max: "4", the connector creates four tasks spread evenly across both workers. When load rises, add a new worker — Kafka Connect automatically rebalances tasks without stopping the connector:

docker compose up -d connect-3

The rebalance process moves some tasks to the new worker. To minimize the pause, some Kafka Connect versions support cooperative rebalancing, which moves tasks one at a time.

#Source Data Redundancy and Connector Failover

A Kafka Connect cluster handles worker failures, but not source failures. If the primary database dies, the connector still fails even though the cluster is healthy. Source redundancy is achieved with patterns:

• Read replica: point the connector at a replica ready to take over when the primary dies.
• Active-passive: run the connector in two regions, with only one actively streaming.
• Planned failover: switch the connector's database connection via DNS or a load balancer.

For fast failover, also prepare the connector's connection retry properties:

{
  "connect.timeout.ms": "30000",
  "connect.backoff.max.delay.ms": "60000",
  "connect.backoff.initial.delay.ms": "1000"
}

With connect.backoff.max.delay.ms: "60000", the connector retries with pauses of up to a minute when the database is unavailable, giving the source time to recover.

#Identical Worker Configuration

For the cluster to work correctly, all workers must use the identical core configuration: bootstrap.servers, group.id, and the three internal topic names. A mismatch in any of these splits workers into different clusters — a common symptom is a connector registered through one worker that isn't visible on another.

GROUP_ID: 1
CONFIG_STORAGE_TOPIC: connect-configs
OFFSET_STORAGE_TOPIC: connect-offsets
STATUS_STORAGE_TOPIC: connect-status

If OFFSET_STORAGE_TOPIC: connect-offsets doesn't match across workers, tasks can read offsets from different topics and produce inconsistent read positions. Standardize the configuration through compose templates or Helm charts to avoid manual errors.

#Conclusion

Episode 12 lifts your pipeline from a single point to a resilient cluster: several workers share the work, worker failures are recovered automatically, tasks spread evenly and can be scaled, and source redundancy prepares for true failover.

The key takeaways:

• Distributed mode is active when multiple workers share the same group.id and internal topics.
• Offsets in Kafka allow task failover without re-reading data from the start.
• tasks.max distributes load across all workers; adding workers rebalances tasks.
• A Connect cluster doesn't protect against source database failures.
• Source redundancy uses read replicas or an active-passive pattern.

In the next episode, episode 13, we'll discuss auditability and data observability — tracing database changes and reconstructing the event stream, audit trails for CRUD and schema changes, and observability with metrics, logs, and tracing.