Learn Apache Kafka - Message Serialization & Schema Management

#Introduction

Kafka stores data as a sequence of bytes that doesn't care about format. The consequence: you decide how data is encoded, and that format must be agreed upon by producers and consumers. The serialization format choice is one of the most impactful decisions for long-term compatibility.

Episode 7 covers String, JSON, Avro, and Protobuf — from how each one works, their advantages and disadvantages, to when to choose which. Then we dive into the heart of schema management: the Confluent Schema Registry, schema evolution, compatibility types, and subject naming.

By the end of this episode, you'll be able to design schemas that evolve safely without breaking running consumers — a problem that often becomes a production nightmare.

#Serialization Formats

#String and JSON

The simplest: the StringSerializer sends plain text, and JSON uses libraries like Jackson or Gson to convert objects. JSON is human-readable and very flexible, but there's no formal schema — producers and consumers just hope each other "stays compatible" until one day it changes.

The problem appears during evolution: if a producer adds a field, old consumers usually survive (they're tolerant), but if a field is renamed or removed, applications can fail silently. Without a schema registry, there's no place to validate.

#Avro

Avro is a compact binary format with a mandatory schema defined in JSON. Every record has a schema; when reading, consumers use the specified schema. Avro is popular in the Kafka ecosystem for two reasons: data size is small compared to JSON, and it has strong schema evolution support — new fields, removed fields, and default values are handled explicitly.

#Protobuf

Google's Protobuf is also binary and compact, with .proto schemas compiled into code in many languages. Its advantages: very fast serialization, broad cross-language support, and fine-grained control over field layout with explicit field numbers. The downside: the schema can't be read directly from the data like Avro can without a registry.

#Custom Serializers

For special cases — proprietary formats or bit-level control needs — you can write your own serializer by implementing Serializer and Deserializer in Java, or use other formats like MessagePack, Thrift, and CBOR. The principle is always the same: serialize at the producer, deserialize at the consumer, and maintain version compatibility.

Human-readable:  JSON   >  Avro (schema JSON)
Binary/compact:  Protobuf ~= Avro ~= MessagePack
Schema-driven:   Avro == Protobuf > JSON (without registry)

#Schema Registry

#The Role of the Confluent Schema Registry

Schema Registry is a service that stores schema versions and ensures every change is compatible before it's used. Producers register a schema and get an ID; when sending a record, that ID is embedded in the message header. Consumers read the ID, fetch the schema from the registry, and deserialize correctly — even if the schema is a different version from what was previously compiled.

This solves the "JSON without a schema" problem: evolution is controlled, and old data can always be read with new schemas. Schema Registry has become a standard component in Confluent Platform and many Kafka deployments.

#Subject Naming and Versioning

Every schema is registered under a subject. The default naming strategy is TopicNameStrategy: each topic has a topic-name-value and topic-name-key subject. Every schema change produces a new version under the same subject:

orders-value  v1 -> v2 -> v3  (all compatible BACKWARD)

Other strategies: RecordNameStrategy (subject from the record name, suitable for multiple schemas in one topic) and TopicRecordNameStrategy (a combination of both).

#Compatibility Types

The registry validates schema changes against the configured compatibility type:

• BACKWARD (default): consumers with the new schema can read data written with the old schema. Rules: new fields must have defaults, fields cannot be removed.
• FORWARD: old consumers can read new data.
• FULL: both directions; the strictest.
• NONE: no validation, flexible but dangerous.

curl -s http://localhost:8081/subjects | jq
curl -s http://localhost:8081/subjects/orders-value/versions | jq

curl -s http://localhost:8081/subjects accesses the Schema Registry REST API — the first endpoint shows all subjects, the second lists the versions of one subject.

#Working with Avro

#Defining an Avro Schema

Avro schemas are written in JSON. An example for an order event:

{
  "type": "record",
  "name": "Order",
  "namespace": "com.example",
  "fields": [
    {"name": "order_id", "type": "string"},
    {"name": "amount", "type": "double"},
    {"name": "status", "type": "string", "default": "pending"}
  ]
}

The status field is given a default so this new schema is backward compatible with the old schema that didn't have it.

#Generic vs Specific Records

Avro has two ways of using schemas: generic records, which read and write fields dynamically via field names (flexible, no generated code), and specific records, which use generated classes from the schema (type-safe, faster, suitable for Java). For Python and Go, libraries like fastavro or confluent-kafka use the schema referenced directly.

#Avro Producer Example in Java

props.put("key.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");
 
Order order = Order.newBuilder()
    .setOrderId("order-001")
    .setAmount(125.50)
    .build();
producer.send(new ProducerRecord<>("orders", order));

KafkaAvroSerializer automatically registers the schema with the registry and embeds the schema ID in every record.

#Protobuf and JSON Schema

#Protobuf Advantages

Protobuf provides the lowest serialization latency among the popular formats and very mature cross-language support. Every field has a unique number in the .proto, so adding fields doesn't disturb old ones:

syntax = "proto3";
message Order {
  string order_id = 1;
  double amount = 2;
  string status = 3;
}

#JSON Schema Validation

For teams that want to stay with JSON, JSON Schema provides formal validation: it defines field types, required fields, and constraints. Kafka Connect also supports JSON Schema as a schema format. Its performance is lower than binary formats, but the human readability advantage is often worth it for non-critical data.

#Schema Design Best Practices

A few practices that will save you later:

• Give new fields a default so they're backward compatible.
• Never remove a field without going through a deprecation cycle.
• Use a union with null for optional fields.
• Set FULL compatibility when data is highly critical.
• Schema versioning: use descriptive field names and a stable namespace.

#Closing

In this episode 7 you've understood the serialization choices from String, JSON, Avro, Protobuf to custom serializers, the role of the Schema Registry in managing schemas and their IDs, schema evolution with various compatibility types, subject naming strategies, and schema design best practices.

The key takeaways:

• Kafka stores bytes: the serialization format is a long-term compatibility decision.
• Avro and Protobuf are compact and schema-driven; JSON is readable but without a formal schema.
• The Schema Registry stores schemas and validates evolution with compatibility types.
• The topic-value and topic-key subjects are the units where schema versions live.
• New fields must get a default to be backward compatible; don't remove fields casually.
• BACKWARD, FORWARD, FULL, and NONE control how strictly schema changes are accepted.

In the next episode 8 we discuss delivery quality guarantees: message ordering and delivery semantics — ordering per partition and per key, global ordering limits, at-most-once, at-least-once, and exactly-once semantics with idempotent producers and transactional messaging. This is the foundation for understanding Kafka transactions in episode 9!