Learn RabbitMQ - RabbitMQ Streams

#Introduction

So far, every pattern we've learned is consumptive: once a message is acked, it disappears from the queue forever. But there's a category of workload that actually wants to store an event stream and read it multiple times — event sourcing, audit logs, and real-time analytics. It's for these needs that RabbitMQ Streams was born in version 3.9.

Streams differ fundamentally from queues: data is stored as an append-only log that isn't deleted after being read. Consumers mark their position with an offset, can go back (replay), and several consumer groups can read the same log independently. The semantics are similar to Kafka, but living inside the same RabbitMQ.

This episode explains how streams work, their key features like offset tracking and retention, their dedicated protocol that uses a separate connection, and guidance on when to use streams and when to stick with queues.

#Introducing Streams

#Log-Based Messaging and Non-Destructive Consumption

A stream is an append-only structure: new messages are always added at the end of the log, and consumption doesn't delete data. This is the opposite of a queue, where messages are removed after being acked. That's why streams suit workloads that need re-reading.

Another fundamental difference: streams are replicated (similar to quorum queues) and optimized for high throughput and efficient disk writes.

#Creating a Stream

Streams are declared through the rabbitmq_stream plugin with the queue type stream:

rabbitmqadmin declare queue name=events.stream \
  arguments='{"x-queue-type":"stream"}'

Once the stream plugin is active, AMQP clients can publish and consume a stream like a regular queue, with the difference that consumption happens with basic_consume at a specific position.

One important difference to remember: streams don't support queue features like TTL, dead letter exchanges, and priority. If you need those features, use a quorum queue. Streams sacrifice these features for speed and large storage capacity.

#Stream Features

#Offsets and Consumer Groups

Each stream consumer stores its own read position (offset), or shares one within a consumer group. With a group, several consumers share the read load with synchronized offsets; without a group, each consumer reads the entire log from its own position. This lets two groups of applications read the same events independently.

#Replay from Offsets and Retention

Because the log isn't deleted when read, consumers can restart from any offset — including the beginning of the log. Retention policies limit how long data is kept:

rabbitmqadmin declare queue name=audit.stream \
  arguments='{"x-queue-type":"stream","x-max-age":"7D","x-max-length-bytes":"20GB"}'

The x-max-age=7D argument keeps stream data for 7 days, and x-max-length-bytes limits the total size — older data is deleted beyond that.

Offsets are committed by consumers automatically at a certain interval, or manually through the client API. A consumer joining an existing consumer group starts from the last stored offset — or from the beginning of the log if no offset is stored, depending on configuration. Understand this behavior when redeploying consumers: make sure the starting offset matches your application's replay needs.

#Stream Filtering

RabbitMQ 3.13+ supports stream filtering: consumers can register a filter when subscribing, so only matching messages are sent — saving bandwidth when only part of the events are needed. Filtering is applied broker-side before messages are sent.

#Super Streams for Partitioning

A single very large stream can become a bottleneck because all data flows through one partition. A super stream (3.13+) splits one logical stream into several physical streams partitioned by key. Publishers pick a partition via hashing on the routing key, so the load spreads evenly and each partition can scale independently. A super stream is the answer when a single stream is no longer enough for the required throughput.

#When to Use Streams

#Suitable Use Cases

Streams are the right choice for: event sourcing (a complete history of state changes), audit logs (data must be available for investigation), time-series data, and fan-out with replay to many consumers that need to read the same events. Thanks to their high throughput, streams also suit analytics pipelines.

#Streams vs Queues

A guide to choosing:

• Queue — each message processed once, work queue patterns, RPC, TTL, DLX.
• Stream — events read by many consumers, need replay, historical data, large throughput.

If you need both, combining them in the same RabbitMQ is legitimate: use queues for commands and streams for events.

#Protocol and Stream Clients

#Native Stream Protocol vs AMQP

Streams can be accessed via plain AMQP (easy, one connection for everything), or via the native stream protocol (rabbitmq_stream on port 5552) which offers the best performance with a dedicated connection. Official stream clients are available for Java, Go, .NET, and Rust; AMQP clients in other languages can still use streams.

ss -tlnp | grep 5552

The ss -tlnp command verifies that port 5552 (the native stream protocol) is listening after the plugin is enabled.

#Monitoring Streams and Offsets

Stream consumption lagging far behind adds disk and network load. Monitor consumer positions relative to the end of the log:

rabbitmqctl list_streams name consumer_count

list_streams shows each stream with its number of active consumers. If the gap between consumer offsets and the log end keeps growing — visible on the stream page in the Management UI — the consumers can't keep up with the write rate. Solutions: add partitions (super stream), increase consumer parallelism, or lower retention.

#Stream Performance Optimization

For maximum throughput: use the native stream protocol, set larger publish batches, and increase the frame size. Monitor stream metrics in the Management UI — consuming streams with a slow offset adds disk load, so adjust retention to your replay needs.

#Conclusion

In episode 21 you understood streams as a Kafka-style append-only log, managed offsets and consumer groups, did replay and retention, used stream filtering, and chose between streams and queues based on workload.

Key takeaways:

• Streams store data append-only; consumption doesn't delete messages.
• Offsets let consumers re-read from any position.
• Consumer groups share the load; without a group, each consumer reads fully.
• Retention is set with x-max-age and x-max-length-bytes.
• Stream filtering (3.13+) saves bandwidth at the broker.
• Super streams split one large stream into several partitions.
• The native stream protocol on port 5552 gives the best performance.

In the next episode we will optimize performance tuning — batching and async publishing, prefetch tuning, queue type selection, lazy queues, up to system-level tuning like the Erlang VM, file descriptors, and disk I/O. This is what separates a smoothly running broker from one that stutters under load!