Learn Apache Kafka - Message Ordering & Delivery Semantics

#Introduction

The question most often asked by every team adopting Kafka: is message order guaranteed? and can messages be lost or delivered twice? The answer isn't black and white — both depend on the design decisions of your topics, producers, and consumers.

Episode 8 separates myths from facts: you'll understand the ordering Kafka guarantees (per partition and per key), why global ordering is so expensive, and the three delivery semantics levels — at-most-once, at-least-once, and exactly-once (EOS). At the end, we dissect the EOS implementation with the idempotent producer and transactional API that will be deepened in episode 9.

This is one of the most conceptual episodes, so take it slowly. Semantic decisions determine the consistency of your entire system.

#Ordering Guarantees

#Partition-Level Ordering

Kafka guarantees ordering within a single partition: records are written in arrival order and read in the same order. This is Kafka's core guarantee. Consumers in one group reading one partition will always see records in the same order they were produced.

This guarantee depends on one condition: the producer must not send two records to the same partition concurrently with reordering caused by retries. This is why the idempotent producer sets max.in.flight.requests.per.connection=5 while still guaranteeing order — sequence numbers at the broker detect reordering.

#Key-Based Ordering

Because the same key always goes to the same partition, all records with the same key are guaranteed to be ordered. This is the most common way to preserve order per entity: key = order_id, user_id, or device_id. All events for one entity will be processed sequentially, while different entities can be processed in parallel.

#Global Ordering and Timestamp Ordering

Kafka does not guarantee global ordering — records in different partitions have no overall order. Getting global ordering means forcing all records into one partition, which sacrifices all parallelism and throughput. In practice, consistency per key is sufficient for almost all cases.

Ordering based on event time (the timestamp in the record) is also not guaranteed by Kafka — the log order is ingestion order. For event-time processing, use windowing in Kafka Streams (episode 13).

#Delivery Semantics

#At-Most-Once

In at-most-once, records are sent and offsets committed before processing. If processing fails, the record is skipped — data can be lost. Configuration that produces this: acks=0 on the producer, and the consumer committing offsets before processing. Suitable for non-critical metrics where losing data is more acceptable than duplicates.

#At-Least-Once

In at-least-once, records are committed after processing. If a crash happens between processing and commit, the record is reprocessed — nothing is lost, but there can be duplicates. Configuration: acks=all on the producer, the consumer commits after processing. This is the default level used by almost all deployments, and duplicates are handled with idempotent consumers — processing the same record produces the same effect.

#Exactly-Once Semantics (EOS)

Exactly-once means every record is processed exactly once, with no loss and no duplication — from the perspective of the downstream system. This is the most difficult and most expensive level. Kafka achieves EOS through three mechanisms that work together:

1. Idempotent producer (enable.idempotence=true) prevents duplicates from retries.
2. Transactional API makes cross-partition delivery atomic.
3. Read-committed consumer (isolation.level=read_committed) only reads records from committed transactions.

# Producer
enable.idempotence=true
transactional.id=order-processor-1
 
# Consumer
isolation.level=read_committed

#Implementing Exactly-Once

#Idempotent Producers as the Foundation

Idempotence works by giving each producer a producer ID (PID) and a sequence number per partition. The broker stores the (PID, partition, sequence) pair to detect duplicate records and reject out-of-order ones. The result: retries never produce duplicates, and records from one partition always arrive in the correct order.

#Transactional API

To make multiple sends atomic — for example writing to two topics at once — use the transactional API:

producer.initTransactions();
try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("orders", "order-001"));
    producer.send(new ProducerRecord<>("order-events", "order-001"));
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
}

All records in a transaction are visible to read_committed consumers only when commitTransaction() succeeds. If it fails, abortTransaction() ensures nothing is visible.

#Transaction Coordinators and Read Committed

Every transaction is managed by a transaction coordinator — a broker that tracks transaction status and stores the result in the internal __transaction_state topic. Consumers with isolation.level=read_committed hold records from uncommitted transactions and discard records from aborted transactions; read_uncommitted sees everything, including uncommitted records.

#EOS Use Cases and Trade-Offs

Use EOS for flows that genuinely require atomicity and zero duplicates: end-to-end stream processing, fund transfers, or critical state synchronization. The trade-offs are real: lower throughput, higher latency, and more operational complexity. For most pipelines, at-least-once with idempotent consumers is more than adequate.

#Trade-Offs and Choices

A summary guide to choosing delivery semantics:

At-most-once : metrics may be lost, peak throughput
At-least-once: safe default, handle duplicates at the consumer
Exactly-once : critical state, ready to pay the throughput cost

Before choosing EOS, ask: is the impact of duplicates truly harmful? If not, at-least-once is far cheaper to operate. If yes, make sure the entire pipeline uses Kafka's transactional features so the promise is fulfilled.

#Closing

In this episode 8 you've understood Kafka's ordering guarantees: per partition and per key, with global ordering not guaranteed. You also understood the three delivery semantics — at-most-once, at-least-once, and exactly-once — along with the EOS mechanisms: idempotent producers, the transactional API, and read-committed consumers.

The key takeaways:

• Ordering is guaranteed per partition; records with the same key are ordered.
• Global ordering is not guaranteed and is very expensive.
• At-least-once with idempotent consumers is the common production default.
• EOS requires idempotent producers, transactions, and read_committed together.
• The transaction coordinator tracks transaction status in an internal topic.
• End-to-end EOS only holds if the entire pipeline uses Kafka.

In the next episode 9 we'll work through Kafka transactions in practice: the initTransactions, beginTransaction, send, commitTransaction, and abortTransaction lifecycle, the read-process-write pattern, configuring transactional.id and transaction.timeout.ms, and their impact on throughput and latency. Get your cluster ready for real transaction testing!