Learn Debezium - Payload Format & Schema Evolution

#Introduction

Every change captured by Debezium is wrapped into an event with a structure that's consistent across all connectors. Consumers that understand this structure can interpret any change without knowing the source database. Episode 5 dissects that structure, then compares the three main serialization formats and how to keep schemas evolving safely.

Understanding the event anatomy isn't just theory: decisions like using JSON or Avro, setting the decimal representation mode, and choosing the schema registry compatibility rules will determine whether your pipeline survives for years or breaks the moment a single column is added.

#The Anatomy of a Debezium Event

A Debezium event consists of two large blocks: schema and payload. The payload is the part most often read:

{
  "payload": {
    "before": null,
    "after": {
      "id": 1004,
      "first_name": "Anne",
      "last_name": "Kretchmar",
      "email": "annek@noanswer.org"
    },
    "source": {
      "version": "3.0.0.Final",
      "connector": "mysql",
      "name": "dbserver1",
      "db": "inventory",
      "table": "customers",
      "server_id": 223344,
      "ts_ms": 1710000000000
    },
    "op": "c",
    "ts_ms": 1710000000123
  }
}

The important fields inside the payload:

• before: the row values before the change, null for inserts.
• after: the row values after the change, null for deletes.
• source: metadata about the change's origin — connector, database, table, and log coordinates.
• op: the operation type.
• ts_ms: the time the event was created.

The op value has standard codes:

With the before and after combination, consumers can reconstruct the complete change: who changed, from what value, to what value.

#Payload Formats: JSON versus Avro versus Protobuf

Debezium events can be serialized in three main formats:

JSON is the easiest to start with and suits prototyping, but every event carries the full schema, which wastes bandwidth. Avro and Protobuf hand the schema description over to the schema registry, making events small and validated.

#Integration with the Schema Registry for Schema Evolution

For Avro or Protobuf to work, every worker must point to a schema registry:

key.converter: io.confluent.connect.avro.AvroConverter
value.converter: io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url: http://schema-registry:8081
value.converter.schema.registry.url: http://schema-registry:8081

When a table changes — for example, an email column is added — Debezium creates a new schema version. The schema registry stores the version history and enforces the compatibility rules before a new schema is approved. Older consumers that only need the old columns can still read the new events as long as the schema stays compatible.

#Field Mapping and Schema Conversion Best Practices

Database data types don't always have a direct equivalent in the event format. Debezium provides properties to control the conversion:

{
  "decimal.handling.mode": "double",
  "interval.handling.mode": "string",
  "time.precision.mode": "adaptive",
  "binary.handling.mode": "bytes"
}

A few guidelines to keep in mind:

• decimal.handling.mode set to precise keeps full precision but produces a byte representation; double is simpler but risks precision loss.
• interval.handling.mode set to string explicitly preserves the meaning of PostgreSQL intervals.
• Avoid changing the conversion modes on a connector that's already running, because the event schema will change and could break compatibility.

#Conclusion

Episode 5 opened the black box of Debezium events: the payload with before, after, source, op, and ts_ms, standard operation codes, a comparison of serialization formats, the schema registry's role in schema evolution, and control over data type conversion.

The key takeaways:

• A Debezium payload always consists of before, after, source, op, and ts_ms.
• op determines the operation type: c, u, d, r, or t.
• JSON is easy to read but wasteful; Avro and Protobuf are more compact and validated.
• The schema registry stores schema versions and enforces compatibility rules.
• Data type conversion modes must stay stable to preserve event compatibility.

In the next episode 6 we'll discuss filtering, routing, and topic design — limiting which tables are captured, routing events to different topics, topic naming and partitioning strategies, and the use of transforms.