Learn Apache Kafka - Kafka Streams: Stream Processing

#Introduction

So far you've been reading and writing data with producers and consumers. What if you want to count, join, or aggregate data as it flows? Kafka Streams is a Java library that turns Kafka into a stream processing engine — without needing a separate cluster like Flink or Spark Streaming.

Kafka Streams' advantages: it runs as an ordinary JVM application, uses Kafka as state storage, and inherits all Kafka guarantees including exactly-once. No new infrastructure; you write a topology, and the library handles parallelism, failover, and recovery.

Episode 13 covers the basic concepts of KStream, KTable, and GlobalKTable, DSL operations, stateful processing with RocksDB, windowing for time-based aggregation, and exactly-once processing guarantees.

#Kafka Streams Fundamentals

#A Stream Processing Library, Not a Framework

An important distinction: Kafka Streams is a library, not a server framework. You don't run a special daemon; your application is a JVM process that uses Kafka Streams. That means standard deployment: a jar, a Docker image, or any orchestrator. One application can be scaled by running multiple instances; topic partitions are divided among the instances.

#KStream, KTable, and GlobalKTable

Three main abstractions represent data:

• KStream: an immutable stream of records — every event is a new fact. Analogy: a log.
• KTable: a mutable view — each key only stores the latest value. Analogy: a table/database snapshot.
• GlobalKTable: a KTable that replicates all data to every instance, suitable for lookups of small reference data.

#Topology Concept

Every Streams application defines a topology: a graph of processors connecting sources (input topics), operators (filter, map, aggregate), and sinks (output topics). This topology is built with the Streams DSL, then run by KafkaStreams:

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> orders = builder.stream("orders");
orders.filter((key, value) -> value.contains("PAID"))
      .mapValues(value -> value.toUpperCase())
      .to("paid-orders");
 
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

builder.stream("orders") creates a source from a topic, and each operator flows data into the next one until to() writes the result to the destination topic.

#Streams DSL

#Stateless Operations

Stateless operations process one record without remembering other data:

• filter / filterNot: discards records that don't satisfy the predicate.
• map / mapValues: changes the key or value.
• flatMap: splits one record into many records.
• branch: splits a stream into several branches based on conditions.
• merge: combines two streams into one.

#Stateful Operations

Stateful operations need old data: groupBy and aggregate group records and compute aggregations; join combines two streams by key. An aggregation example:

KTable<String, Long> orderCount = orders
    .filter((key, value) -> value.contains("PAID"))
    .groupByKey()
    .count(Materialized.as("order-count-store"));

groupByKey().count() counts records per key and stores the result in a state store named order-count-store — a KTable whose latest value can be queried.

#Stateful Processing

#State Stores with RocksDB

Aggregation and join results are stored in state stores. The default local storage is RocksDB, a fast embedded key-value database. Each application instance has a copy of the state store for the partitions it's responsible for.

#Changelog Topics and Recovery

Every state store has a changelog topic (with cleanup.policy=compact, see episode 10) that records every state change. When an instance crashes or a rebalance happens, the new instance restores state by reading the changelog from the last position — this is what makes stream processing fault-tolerant without losing state.

#Interactive Queries

Stored state isn't only for internal use — it can be queried directly by other applications via Interactive Queries:

curl -s "http://localhost:8080/state/order-count-store/key/order-001"

curl -s http://localhost:8080/state/order-count-store reads the current value from the state store. Because state is distributed, applications need to ask the metadata API where a partition lives before fetching the value — a pattern that lets Kafka Streams act as a real-time materialized view.

#Windowing

#Window Types

Time-based aggregation divides a stream into windows:

• Tumbling window: non-overlapping windows of fixed duration, for example every 1 minute.
• Hopping window: overlapping windows, for example 1-minute duration with a 30-second advance.
• Sliding window: windows that slide based on event time, used in joins.
• Session window: dynamic windows that close after a period of inactivity — suitable for user sessions.

#Grace Period

Streaming data often arrives late. The grace period determines how long late records are still accepted after a window ends. After the grace period passes, records are considered late and discarded:

KTable<Windowed<String>, Long> perMinute = orders
    .groupByKey()
    .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(Duration.ofSeconds(10)))
    .count();

TimeWindows.of(Duration.ofMinutes(1)) creates a one-minute tumbling window, and .grace(Duration.ofSeconds(10)) allows a ten-second lateness tolerance.

#Exactly-Once Processing

#processing.guarantee

Kafka Streams integrates transactions (episode 9) natively. Configuration:

processing.guarantee=exactly_once_v2

With processing.guarantee=exactly_once_v2, Streams automatically uses transactional producers and read_committed consumers: offsets, state, and output are committed in the same transaction, so reprocessing doesn't produce duplicates.

#EOS in Streams

Concretely: when an instance processes a batch, all writes (to output topics and changelogs) plus consumed offsets are in one transaction. If a crash happens, the batch is reprocessed from the start but the old transaction is aborted — the result is the same, no duplicates. This is the easiest end-to-end exactly-once to achieve in the Kafka ecosystem, without writing manual transaction code.

#Closing

In this episode 13 you've understood Kafka Streams as a stream processing library, the difference between KStream, KTable, and GlobalKTable, stateless and stateful operations, state stores with RocksDB and changelog topics, windowing with grace periods, and exactly-once processing.

The key takeaways:

• Kafka Streams is a JVM library, not a separate server framework.
• KStream is a stream of events; KTable is a snapshot per key.
• State stores are RocksDB-based with changelog topics for recovery.
• Windowing divides time-based aggregation; grace periods accommodate late data.
• processing.guarantee=exactly_once_v2 gives automatic end-to-end EOS.
• Scale with additional instances using the same application.id.

In the next episode 14 we'll discuss ksqlDB — SQL for stream processing built on top of Kafka Streams. You'll learn the difference between streams and tables, persistent, push, and pull queries, and deploying its server and CLI.