Learn Debezium - Monitoring, Handling Failures, & Debugging

#Introduction

A healthy connector in the morning can fail silently in the middle of the night. Episode 7 equips you with the operational habits you must have: monitoring connector status, measuring lag, reading the right logs, handling errors without stopping the pipeline, and building automated alerts.

The key to all of this is a single source of truth — the Kafka Connect REST API and JMX metrics — plus the habit of checking health regularly. Without it, a small problem like one schema change can turn into hours of data loss.

#Monitoring Connector and Task Status

The Kafka Connect REST API gives complete connector status along with each task:

curl -s http://localhost:8083/connectors/inventory-connector/status | jq

The state field on the connector and tasks shows UNASSIGNED, RUNNING, PAUSED, or FAILED. To filter for the important parts:

curl -s http://localhost:8083/connectors/inventory-connector/status | \
  jq '{connector: .connector.state, tasks: [.tasks[].state]}'

Get into the habit of checking all connector statuses at once:

curl -s http://localhost:8083/connectors

#Measuring Offset and Lag

For source connectors, "lag" means the difference between the latest position in the database log and the position already captured. Debezium exposes these metrics via JMX under the io.debezium domain:

jcmd $(pgrep -f 'debezium' | head -1) GC.heap_info

A common production practice is to expose JMX to Prometheus using the JMX exporter, then monitor metrics such as the number of rows remaining in a snapshot and streaming lag. For sink pipelines, lag can be checked via the consumer group:

docker exec -it kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group jdbc-sink

The LAG column shows how many events the consumer hasn't processed yet. Lag that keeps climbing without stopping is the main signal that a pipeline has problems.

#Logs and Error Handling

When a connector fails, the first step is to read the container logs:

docker logs connect --tail 200

In addition to logs, Debezium provides error handling at the connector level:

{
  "errors.tolerance": "all",
  "errors.deadletterqueue.topic.name": "cdc-dlq",
  "errors.deadletterqueue.context.headers.enable": "true"
}

With errors.tolerance: "all", records that fail to process don't stop the connector; instead they're sent to the cdc-dlq topic along with context headers. This is far safer than having the connector stop entirely because of one bad record.

#Handling Common Problems

#Record Parsing Error

These usually happen when the payload format doesn't match — for example, a new unrecognized column. Fix it by ensuring the consumer schema is in sync with the latest schema, or send the problematic records to the DLQ as in the example above.

#Schema Mismatch

When a table changes, the event schema changes with it. If consumers aren't ready, events get rejected. The solution: use a schema registry to validate compatibility and update consumers before a new schema becomes active.

#Database Unavailable

If the source database is down, the connector fails to read and can enter FAILED status. Debezium will retry the connection according to configuration, but make sure retry and heartbeat are active so the read position isn't lost.

#Building Alerts for Connector Failures and Data Loss

The metrics already exported to Prometheus can serve as the basis for alerts. An example alert rule for a failed connector:

groups:
  - name: debezium-alerts
    rules:
      - alert: ConnectorStateFailed
        expr: debezium_connector_metrics_state == 3
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Connector tidak sehat: {{ $labels.connector }}"

Additional rules we recommend:

• Alert when streaming lag exceeds a threshold, for example 5 minutes.
• Alert when the task count drops below what's configured.
• Alert when records arrive in the DLQ, which indicates data that failed to process.

#Conclusion

Episode 7 brings you into operational mode: checking status via the REST API, monitoring lag with JMX and consumer groups, handling errors with tolerance and DLQ, diagnosing common problems, and building Prometheus alerts for failures.

The key takeaways:

• The REST API GET /connectors/{name}/status is the primary source of truth for connector status.
• Lag is monitored via JMX for sources and consumer groups for sinks.
• errors.tolerance and the DLQ prevent a single failed record from stopping the whole pipeline.
• Check container logs before diagnosing a connector failure.
• Alerts for FAILED state, high lag, and records in the DLQ are the minimum you must have.

In the next episode, episode 8, we'll discuss integration with consumers and sinks — connecting CDC events to Kafka consumers, stream processors such as ksqlDB and Kafka Streams, and sinks to databases, data lakes, and data warehouses.