Learning Redis - Streams (Event Streaming & Message Broker)

#Introduction

So far you've mastered Redis's classic data structures. Now it's time for the feature that turns Redis into a message broker and event streaming platform: Redis Streams.

Streams are a log-based structure — append-only data organized by ID — designed for event streaming similar to Apache Kafka, but built into Redis with no extra infrastructure. In this episode we'll dissect writing and reading events, then dive into consumer groups for distributed message processing with delivery guarantees. This is the material most loved by backend engineers.

#Redis Streams Concepts

#Append-Only Log Structure

A Stream is a series of events that can only be appended at the end (append-only). Each event has a unique timestamp-based ID (e.g. 1700000000000-0) and one or more field-value pairs. This append-only nature makes Streams suitable for event sourcing, audit logs, and data ingestion.

ID timestamp-seq → field-value
1650000000000-0 → event "order.created" payload "{...}"
1650000000001-0 → event "payment.received" payload "{...}"
1650000000002-0 → event "order.shipped" payload "{...}"

Because the data is stored in a log, consumers can read from any position — unlike pub/sub, which is fire-and-forget (episode 8).

#Different from a List Queue

A List (episode 4) is a queue that removes elements when popped — once read, the data is gone. Streams retain every event in the log, so they can be re-read, analyzed, and processed by many consumer groups independently. This is the fundamental difference that makes Streams suitable for event-driven architecture.

#Basic Stream Operations

#Writing and Reading Events

redis-cli XADD orders '*' event "order.created" orderId "123"
redis-cli XLEN orders

XADD orders '*' event "order.created" orderId "123" appends an event to the orders stream. The * tells Redis to auto-generate the ID from the timestamp. XLEN counts the number of events.

Reading events with ranges and streaming:

redis-cli XRANGE orders - +
redis-cli XREAD COUNT 10 STREAMS orders 0

XRANGE orders - + reads all events (from the smallest ID to the largest). XREAD COUNT 10 STREAMS orders 0 reads 10 events starting from ID 0. To wait for new events, XREAD can use the BLOCK option so the process waits with a timeout — the basic pattern of consuming a stream.

#Consumer Groups

#The Distributed Processing Concept

When a single stream is consumed by many workers, we need work distribution without an event being processed twice. Consumer groups solve this: the group tracks a shared consumption position, each event is delivered to only one consumer, and each consumer has its own identity.

redis-cli XGROUP CREATE orders order_group 0 MKSTREAM

XGROUP CREATE orders order_group 0 MKSTREAM creates the order_group group on the orders stream. ID 0 means start from the beginning; MKSTREAM creates the stream automatically if it doesn't exist.

#Reading and Acknowledging Messages

redis-cli XREADGROUP GROUP order_group worker-1 COUNT 10 BLOCK 5000 STREAMS orders '>'

XREADGROUP GROUP order_group worker-1 COUNT 10 BLOCK 5000 STREAMS orders '>' lets worker-1 read events that haven't been assigned to anyone yet (>). After processing, the worker must acknowledge the event:

redis-cli XACK orders order_group 1700000000000-0
redis-cli XPENDING orders order_group

XACK orders order_group 1700000000000-0 tells the group that the event with that ID has been fully processed. XPENDING lists events that haven't been acknowledged — if a worker crashes before acknowledging, the event will still show up here.

#Recovery with XCLAIM

When a worker crashes mid-processing, its event is left hanging in the pending list. Another worker can claim that event after a certain timeout:

redis-cli XCLAIM orders order_group worker-2 60000 1700000000000-0

XCLAIM orders order_group worker-2 60000 1700000000000-0 transfers event ownership to worker-2, provided the event has been pending for more than 60,000 milliseconds. This is the at-least-once delivery mechanism: events are guaranteed to be processed at least once, and the application must be ready to handle duplicates with idempotency.

#Streams vs Apache Kafka vs Pub/Sub

#Streams' Position in the Ecosystem

Kafka excels at massive throughput and long retention with a complete ecosystem (Kafka Connect, Schema Registry). However, for medium scale that already uses Redis, Streams removes the complexity of operating a Kafka cluster. Pub/Sub in episode 8 stores nothing — broadcast only — so Streams and Pub/Sub complement each other rather than compete.

#When to Use Streams

Use Streams when: messages must survive even with no consumers, need to be re-read, need consumer groups with acknowledgments, or require per-key ordered processing. If your need is only one-shot real-time notifications, Pub/Sub is lighter.

#Common Use Cases

• Order processing: order.created events consumed by workers for payment, inventory, and notification.
• Event sourcing: store every state change as a replayable sequence of events.
• Audit logs: append-only logs with ordered timestamp IDs.
• IoT telemetry: sensor data pipelines processed by many consumers.
• Reliable job queues: a replacement for Lists when acknowledgments are needed.

#Summary

Episode 6 equipped you with Redis Streams as an event streaming and message broker: XADD for writing, XRANGE/XREAD for reading, consumer groups with XREADGROUP for distributed processing, plus XACK, XPENDING, and XCLAIM for delivery guarantees and recovery.

Key takeaways:

• Streams are append-only logs with unique timestamp-based IDs.
• XADD writes, XRANGE/XREAD reads, XLEN counts.
• Consumer groups divide work among workers without processing an event twice.
• XACK is mandatory after an event is fully processed.
• XPENDING + XCLAIM recover events from crashed workers.
• Delivery is at-least-once — design handlers idempotently.
• Streams are durable and replayable, unlike fire-and-forget Pub/Sub.

In the next episode, episode 7, we cover Bitmaps, Bitfields & Geospatial Indexes — structures for tracking and location-based search. Bitmaps for highly memory-efficient Daily Active Users, Bitfields for compact integer storage, and GEO commands for radius-based location search. Let's continue!