Learn Debezium - Snapshot, Streaming, and Offset Management

#Introduction

When a connector first starts, it doesn't immediately read new changes. Debezium runs a snapshot: it reads all the data that already exists, then transitions to streaming subsequent changes. Understanding these two phases is critical because they determine how long the pipeline takes to cover a large database and how the read position is recorded.

Episode 4 also covers what happens when the worker restarts, how snapshots of large tables are split into chunks, and how schema changes — such as adding a column — are handled without stopping the data flow. All of this is the foundation of stable CDC operations.

#The Initial Snapshot versus Change Streaming

The snapshot phase replicates the state of the database when the connector first runs; events in this phase use op set to r. The streaming phase captures changes after the snapshot position, with op set to c, u, or d. Snapshot behavior is controlled by snapshot.mode:

snapshot.mode: initial
snapshot.fetch.size: 2000
snapshot.max.threads: 4

The snapshot.mode choice must match the situation. For an empty, newly created table, schema_only is faster. For production, initial is the safest default.

#Understanding Offset Storage, Restart, and Recovery

Offsets store Debezium's read position in the database log — for example, the MySQL binlog coordinates or the PostgreSQL LSN. This data is stored in the connect-offsets topic and committed periodically according to offset.flush.interval.ms.

offset.flush.interval.ms: 5000
offset.flush.timeout.ms: 5000

When the container restarts, Kafka Connect reads the stored offsets and the connector resumes from that position:

docker compose restart connect
curl -s http://localhost:8083/connectors/inventory-connector/status

If the offsets are lost or deliberately deleted, Debezium restarts the snapshot according to snapshot.mode. That's why you should never delete the connect-offsets topic without a clear reason.

#Snapshot Chunking and Initial Data Load

For tables with millions of rows, a single snapshot would consume memory and block streaming for too long. The solution is the incremental snapshot, which breaks the work into small chunks and interleaves it with the real change stream.

Incremental snapshots are triggered through a signal table. Define the signal table in the configuration:

{
  "signal.data.collection": "inventory.debezium_signal",
  "incremental.snapshot.chunk.size": "1024"
}

Then trigger the snapshot with an INSERT into the signal table:

Start an incremental snapshot

INSERT INTO inventory.debezium_signal
  (id, type, data) VALUES
  ('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["inventory.customers"]}');

With incremental.snapshot.chunk.size: 1024, Debezium reads the table 1024 rows at a time, emits events with op set to r, then returns to capturing streaming changes between chunks.

#How Debezium Handles Schema Changes

Database schemas aren't static; tables often gain new columns. Debezium handles this according to each database's mechanism:

• MySQL and SQL Server: schema changes are recorded to the schema history topic, and DDL events can be published to Kafka.
• PostgreSQL: the schema is read directly from the internal catalog, so changes are recognized immediately.
• MongoDB: the schema is treated as flexible; each document carries its own structure.

For MySQL, make sure the schema history topic is always available and don't alter the structure of its events:

{
  "include.schema.changes": "true",
  "schema.history.internal.kafka.topic": "schema-changes.inventory"
}

With include.schema.changes: "true", every DDL statement is also sent as an event, so consumers can trace the evolution of the table structure over time.

#Healthy Offset Practices in Operations

A few practices keep offsets healthy over the long run:

• Monitor the commit frequency: offsets committed too rarely risk losing more position on a crash.
• Back up the offset topic: copy connect-offsets to another location as a recovery fallback.
• Don't edit offsets manually: manually modifying offset bytes almost always ends up corrupting the read position.

If offsets truly can't be recovered, the last resort is to delete the offsets and let the connector run a fresh snapshot according to snapshot.mode. This process is expensive for large tables, so get into the habit of backing up before drastic actions.

#Conclusion

Episode 4 explained the data lifecycle in Debezium: the snapshot fills in the initial state, streaming continues the changes, offsets preserve the position across restarts, incremental snapshots handle large tables, and schema changes are handled through schema history and DDL events.

The key takeaways:

• Snapshots use op: r; streaming uses op: c, u, or d.
• Offsets are stored in connect-offsets and keep restarts from losing position.
• Incremental snapshots split large tables into chunks interleaved with streaming.
• Schema history must be preserved for MySQL and SQL Server.
• snapshot.mode must be chosen to match the source database condition.

In the next episode 5 we'll dissect payload format and schema evolution — the before, after, source, op, and ts_ms event structure, a comparison of JSON versus Avro versus Protobuf, and integration with the schema registry.