Learn Debezium - Schema Registry & Data Contracts

#Introduction

When data flows between teams, the event structure is the unwritten agreement that gets violated most often. One team adds a column, another team is left confused. Episode 9 covers how a schema registry turns that agreement into an enforceable contract, complete with versioning, compatibility rules, and automatic validation.

In episode 5 you already saw the schema registry's role in payload formats. Now we step up a level: not just storing schemas, but building a data contract — an explicit agreement about the shape and meaning of every event consumed by many parties.

#Building a Data Contract for Event Schemas

A data contract answers three questions: what the event structure looks like, who's responsible for it, and how the event may evolve. In Debezium practice, this contract takes the form of an Avro or Protobuf schema registered in the schema registry.

Start by enabling the schema registry in your compose stack:

  schema-registry:
    image: confluentinc/cp-schema-registry:7.8.0
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:9092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

Once the worker uses the AvroConverter and points to schema.registry.url, every Debezium event automatically registers its schema the first time the connector runs.

#Managing Versioning and Compatibility Rules

The schema registry stores the history of every schema per subject. Debezium subjects are usually named topic.prefix.database.table-key or -value:

curl -s http://localhost:8081/subjects | jq
curl -s http://localhost:8081/subjects/dbserver1.inventory.customers-value/versions | jq

The compatibility rules determine how freely a schema can evolve:

Set compatibility per subject with:

curl -s -X PUT http://localhost:8081/config/dbserver1.inventory.customers-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}'

With BACKWARD, adding a column must provide a default — this forces teams to think twice before changing an event that many consumers already use.

#Handling Schema Evolution on Debezium Payloads

Database table changes are translated by Debezium into new schema versions. Common scenarios:

• Adding a column with a default: backward compatible, old consumers keep reading.
• Removing a column: only safe with forward compatibility or if no consumer needs it.
• Changing a data type: almost always breaks compatibility, requires a special migration strategy.

Debezium produces a value schema that merges all table columns. So before changing a table schema in production, first check its compatibility with the related subject:

curl -s -X POST http://localhost:8081/compatibility/subjects/dbserver1.inventory.customers-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{...skema baru...}"}'

#Using Avro and Protobuf for Consumer Validation

Validation on the consumer side is the last layer of the data contract. Avro consumers load schemas from the registry, while Protobuf uses generated .proto files. Both give benefits:

• Safe deserialization: events that don't match the schema are rejected immediately.
• Controlled evolution: schema changes are approved by the registry before reaching production.
• Living documentation: the schema itself is the documentation of the data structure.

#Subjects for Key and Value

Debezium registers two subjects for every topic: one for the key and one for the value. The key subject usually contains the primary key schema — for example dbserver1.inventory.customers-key — while the value subject contains the schema of all table columns. This separation matters because keys and values often evolve at different speeds.

When you add a new column to a table, only the value subject changes. Consumers that rely only on the key can keep reading without issue as long as the key subject is untouched.

#Compatibility Strategies for Debezium

Because Debezium generates schemas from database tables, choose a compatibility level aligned with how your tables change:

• Use BACKWARD when column additions happen often and should be given defaults.
• Use FULL when you don't want to worry about the direction of evolution.
• Avoid NONE on production subjects unless you fully understand the risks.

Remember that Debezium sends the complete schema with every event in JSON mode, but with Avro or Protobuf the schema is sent once and events only carry the schema ID. This difference is why Avro mode is far more bandwidth-efficient on high-traffic topics.

#Conclusion

Episode 9 turns event schemas from a footnote into a contract: the schema registry stores versioning, compatibility rules enforce safe evolution, and Avro or Protobuf validate events on the consumer side.

The key takeaways:

• A data contract is realized as a schema registered in the schema registry.
• Subjects follow the topic-key and topic-value pattern with version history.
• The BACKWARD, FORWARD, and FULL compatibility levels control schema evolution.
• Table changes must be checked for compatibility before being deployed to production.
• Avro and Protobuf provide automatic validation on the consumer side.

In the next episode, episode 10, we'll discuss security and data privacy — securing database connections with TLS, authentication and service accounts, payload encryption in Kafka, and GDPR compliance and data masking.