Learn Debezium - Operational Readiness & Runbooks

#Introduction

A CDC pipeline will fail — the question isn't "whether", but "when and how fast it recovers". Episode 19 covers operational readiness: runbook documents with exact steps when a connector fails, procedures for restarting snapshots and replaying data, backup strategies, and drills to test the recovery plan.

A good runbook isn't theory: it contains concrete commands that can be executed during a panic. With regular practice, teams don't have to experiment in the middle of a production incident.

#Writing Runbooks for Connector Failures

The first step in every incident is diagnosis. Standardize this step in the runbook:

curl -s http://localhost:8083/connectors/inventory-connector/status | jq
docker logs connect --tail 100

If the connector is in FAILED state, restart it safely:

curl -s -X POST http://localhost:8083/connectors/inventory-connector/restart
curl -s -X POST http://localhost:8083/connectors/inventory-connector/tasks/0/restart

Don't delete a connector immediately when it fails — first remove the root cause, because deleting and re-registering can trigger an expensive new snapshot.

#Snapshot Restart and Data Replay

When offsets are corrupted or data needs to be reloaded, you need a controlled snapshot re-run procedure. Three scenarios:

• Full replay: delete the connector and topic, then re-register with snapshot.mode: initial.
• Partial re-snapshot: use an incremental snapshot for specific tables without stopping streaming.
• Replay from a specific point: set the initial offset before starting the connector.

To run a re-snapshot without stopping other streaming, use the signal table as in episode 4:

Triggering a single-table re-snapshot

INSERT INTO inventory.debezium_signal
  (id, type, data) VALUES
  ('replay-1', 'execute-snapshot', '{"data-collections": ["inventory.orders"]}');

Before triggering execute-snapshot, make sure consumers are ready to receive the op: r events that will re-enter.

#Incident Response for Connector Lag and Schema Issues

The two most common incidents are high lag and schema mismatch.

High lag — response steps:

1. Identify the source of lag: a slow database, a worker running out of resources, or network issues.
2. Check the JMX Lag metric and worker logs.
3. Temporarily speed things up by raising max.batch.size, then normalize after recovery.

Schema mismatch — response steps:

1. Identify rejected events from the DLQ or error logs.
2. Compare the latest schema in the schema registry with the consumer schema.
3. Update the consumer or give a default to the new column on the source side.

Store both procedures as part of the runbook so anyone on the team can execute them.

#Backing Up Config, Offsets, and Source Metadata

Recovery capability depends on backup availability:

• Config: connector definitions are already in git via GitOps (episode 11).
• Offsets: the connect-offsets topic should be backed up so read positions can be restored.
• Source metadata: the last binlog or LSN position as an extra reference.

Secure the offset topic by mirroring it to a backup cluster:

docker exec -it kafka /opt/kafka/bin/kafka-mirror-maker.sh \
  --consumer.config /kafka/consumer.properties \
  --producer.config /kafka/producer.properties \
  --whitelist connect-offsets

The kafka-mirror-maker.sh --whitelist connect-offsets command above copies the offset topic periodically to a backup cluster, so recovery doesn't depend on a single machine.

#Recovery Drills and Failover Testing

Documents without practice are just hope. Schedule regular drills:

• Snapshot restart drill: stop the connector, remove part of the offsets, recover using the runbook.
• Worker failover drill: stop one worker, make sure tasks move automatically.
• Replay drill: replay one table with an incremental snapshot and verify the consumer.

Measure key metrics during drills — recovery time objective (RTO) and data loss tolerance — then improve the runbook based on findings. Successful drills make teams confident when a real incident comes.

#Minimal Runbook Checklist

A good runbook covers the following elements:

• The symptom that triggers the incident, for example a FAILED connector state.
• Definite diagnostic commands, with expected example output.
• Sequential remediation steps, from the safest to the most drastic.
• An escalation procedure and who to contact when steps don't work.
• A post-mortem section to record findings and improvements.

Store the runbook in the repository alongside the connector configuration, so its version always matches the current pipeline state.

#Conclusion

Episode 19 brings you into production-ready mode: diagnosis and restart runbooks, snapshot restart and replay procedures, incident response for lag and schema, config and offset backups, and recovery drills that test the plan for real.

The key takeaways:

• Document diagnosis and restart commands in an easily accessible runbook.
• Don't delete a failing connector without removing the root cause.
• Re-snapshots can be done per table with the signal table.
• Backing up config, offset, and source metadata is a prerequisite for recovery.
• Schedule drills and measure RTO to validate the runbook.

In the next episode, episode 20, we'll discuss real-world use cases and patterns — microservice synchronization, analytics pipelines, CDC-based caches, event sourcing and audit logs, read scaling, and end-to-end architecture.