Learn Apache Kafka - Kafka in Microservices Architectures

#Introduction

When you break a monolith into microservices, the hardest problem isn't code, it's communication. Synchronous request-response makes services wait on each other, and transactions across services are almost impossible to keep atomic. Kafka offers a way out by moving communication to events.

Episode 26 covers how Kafka becomes the backbone of event-driven architectures: choreography and orchestration patterns, domain events, the saga pattern for distributed transactions, CQRS with materialized views, and the transactional outbox for reliably bridging databases and Kafka.

#Event-Driven Microservices

#Event-Driven Architecture Patterns

In an event-driven architecture, services communicate through events, not direct calls. The publisher service writes events to topics; other services subscribe without knowing each other. The main advantage: decoupling — publishers don't care who their consumers are, and adding consumers doesn't change the publisher.

#Choreography vs Orchestration

• Choreography: no coordinator; every service reacts to events and publishes the next event. Distributed and flexible, but flows are hard to trace and errors are hidden.
• Orchestration: one service (the orchestrator) commands other services through events/commands and supervises results. Flows are clear, but the orchestrator becomes a point of complexity.

Choose choreography for simple, independent flows; choose orchestration when sequencing and control matter.

#Event Storming and Domain Events

Event storming is a workshop for mapping business events into domain events — for example OrderCreated, PaymentCompleted, InventoryReserved. Every event becomes a candidate Kafka record. Domain events should use business domain language (not technical), so they become a shared contract between services.

When designing topics for microservices, never create one "super" topic containing all events of all domains. Separate by domain and version, for example order-events and payment-events, and use a Schema Registry (episode 7) so event contracts can evolve without breaking consumers.

#The Saga Pattern

#Distributed Transactions

Business transactions often span several services: place an order, pay, reduce stock. Keeping atomicity across different databases requires the saga pattern: a sequence of local steps, each step writing its own database and publishing an event. Kafka provides the event channel to coordinate these steps.

#Compensating Transactions

When a step fails, a saga must roll back the steps that already succeeded — with compensating transactions. For example: if payment fails after stock was reduced, run a compensation that restores the stock:

OrderService: receive order  -> publish OrderCreated
PaymentService: process payment -> publish PaymentCompleted
InventoryService: reduce stock -> publish InventoryReserved
If failed: publish PaymentFailed -> compensate InventoryRestocked

PaymentFailed triggers compensation; other services respond with reversing events. An orchestrated saga can run with an orchestrator that reads status from topics and sends the next commands.

#The CQRS Pattern

#Command Query Responsibility Segregation

CQRS separates write operations (commands) and read operations (queries) into different models. Kafka supports this naturally: command services write events to topics, while query services build a projection from the events to serve fast reads. Write and read data don't need to live in the same database.

#Event Sourcing and Materialized Views

With event sourcing, all state changes are stored as a sequence of events — a single source of truth. A materialized view is a projection of the current state built from the events. Kafka Streams (episode 13) is the ideal tool: each consumer builds its own view from the same topic without touching the command service's database.

#Projections with Kafka Streams

KTable<String, OrderState> orders = builder
    .stream("order-events")
    .groupByKey()
    .aggregate(OrderState::new,
        (key, event, state) -> state.apply(event),
        Materialized.as("order-view-store"));

Materialized.as("order-view-store") builds a per-order projection that's always up to date — a query service only reads the state store (interactive queries in episode 13) without touching another database.

#The Outbox Pattern

#Transactional Outbox

How do you guarantee an event is sent exactly when the database changes? Writing to the database and then sending the event has a gap: a crash in between leaves state and events inconsistent. The transactional outbox solves it: write the database change and an outbox record in one database transaction:

BEGIN;
INSERT INTO orders (id, status) VALUES ('order-001', 'CREATED');
INSERT INTO outbox (event_id, topic, payload, created_at)
VALUES (uuid, 'order-events', '{"id":"order-001"}', now());
COMMIT;

BEGIN; ... COMMIT; guarantees the outbox record only exists if the main change succeeded. A relay then reads the outbox and publishes to Kafka — if a crash happens, the outbox hasn't been read and will be sent again, giving reliable at-least-once.

#Change Data Capture and Debezium

Instead of writing your own relay, use CDC (Change Data Capture): Debezium reads the database transaction log and publishes changes — including outbox rows — to Kafka automatically. Full details are covered in episode 27. The transactional outbox plus Debezium combination is the industry standard for database-Kafka integration that doesn't lose events.

#Closing

In this episode 26 you've understood Kafka's role in microservices: event-driven architecture with choreography and orchestration, domain events, the saga pattern with compensating transactions, CQRS with materialized views, and the transactional outbox with Debezium.

The key takeaways:

• Event-driven communication decouples services through Kafka.
• Choreography is flexible, orchestration is controlled — choose per business flow.
• Sagas coordinate distributed transactions with compensation.
• CQRS and event sourcing are built on Kafka's event log.
• The outbox writes the database and event in one transaction for consistency.
• Debezium and CDC reliably stream the outbox to Kafka.

In the next episode 27 we'll discuss Change Data Capture with Debezium — turning database changes into an event stream, Debezium architecture, connector configuration, snapshot modes, and best practices for schema evolution and lag monitoring.