Learn NATS - JetStream Streams

#Introduction

Episode 7 gave you your first introduction to JetStream. Episode 8 goes deeper: how to configure a stream with precision. This is where architectural decisions are made — how long messages are kept, how messages are removed, whether they're stored in files or memory, and how many copies exist.

A wrong choice in this episode can lead to data loss or wasted disk space in production. Let's break down every part of stream configuration, then move on to sourcing, mirroring, and the dedupe window.

#Basic Stream Configuration

#Defining Stream Subjects

Every stream must have a name and a list of captured subjects. A stream can capture many subjects at once using wildcards:

nats stream add ORDERS --subjects "orders.>" --subjects "payments.charged"

The nats stream add ORDERS --subjects "orders.>" command creates an ORDERS stream storing every message under orders.> and payments.charged. A subject can only be captured by one stream within an account — NATS rejects overlapping streams.

#All Options in One Command

To avoid interactive mode, all options can be passed at once:

nats stream add ORDERS \
  --subjects "orders.>" \
  --retention limits \
  --storage file \
  --max-age 30d \
  --max-bytes 1G \
  --replicas 3

The --max-age 30d scheme shows messages survive at most 30 days, --max-bytes 1G caps the total size, and --replicas 3 keeps three copies. These three decisions are the core of data lifecycle management.

#Retention Policies

#Limits: Keep Until Full

Limits is the default mode: messages are kept until they exceed an age or size bound, then the oldest are removed. This suits event logs with a defined time horizon, for example storing order events for 30 days.

#Interest: Keep While There Are Subscribers

Interest keeps messages as long as there's a consumer interested in reading them. When all consumers have processed the messages and no one is waiting, the messages are removed. Suited for workloads where messages don't need to be kept after every party has processed them.

#WorkQueue: Each Message Used Once

WorkQueue is designed for job queues: each message may only be read by one consumer. As soon as a message is acked, it's immediately removed from the stream. No replay, no broadcast.

The WorkQueue pattern is the basis of the work queue we'll build in full in episode 11.

#Storage and Resource Limits

#File or Memory

The storage option determines the storage medium:

• File: data is written to disk, surviving server restarts.
• Memory: data lives in RAM, very fast but lost when the server dies.

nats stream add CACHE --subjects "cache.>" --storage memory --retention limits

nats stream add CACHE --subjects "cache.>" --storage memory suits temporary data like caches. For any data that must not be lost, always choose file storage.

#Limiting Age and Size

The two most common limits:

subjects: ["orders.>"]
retention: limits
storage: file
max_age: 7d
max_bytes: 2GiB
max_msgs: 1000000
discard: old

The max_age: 7d block removes messages older than 7 days, and discard: old specifies that when full, the oldest messages are removed first. Combining these bounds prevents the disk from filling up without limit.

#Replication with Replicas

#Three Copies for Resilience

The replicas field determines how many copies of a stream are stored on different servers within a cluster. A value of 1 for single-node, 3 for a cluster tolerating the loss of one node, 5 for tolerating two nodes.

nats stream edit ORDERS --replicas 3

The nats stream edit ORDERS --replicas 3 command changes the copy count of an existing stream. Each copy is synchronized via Raft consensus — a topic we'll discuss in episode 14. A practical rule: replicas must always be smaller than the cluster's node count.

#Sourcing and Mirroring

#Stream Sourcing Across Subjects

Sourcing lets a stream pull messages from a subject whose source actually originates from another stream:

nats stream add ORDERS_ANALYTICS --subjects "analytics.>" --sources ORDERS

--sources ORDERS makes ORDERS_ANALYTICS copy every message entering the ORDERS stream. This is a popular pattern for building data marts: the main stream stays clean, and derived streams are created for analytics without disturbing the main flow.

#Mirroring Across Clusters or Regions

Mirroring copies the entire contents of a stream from another stream — usually in a different cluster or region — for disaster recovery:

name: ORDERS_DR
mirror:
  name: ORDERS
  external:
    api: nats://dr-cluster:4222

The mirror block above makes the ORDERS_DR stream mirror the ORDERS stream located in a disaster recovery cluster. This is the foundation of the cross-region DR strategy we'll build in episode 21.

#Dedupe Window and Exactly-Once Publishing

#Preventing Duplicate Messages

JetStream provides a dedupe window: every publisher can include a Nats-Msg-Id header. If a message with the same ID enters again within the dedupe window, the duplicate is rejected. This is the first step toward exactly-once publishing.

nats pub orders.created "pesanan-1" --id order-2026-08-10-001

nats pub orders.created "pesanan-1" --id order-2026-08-10-001 gives the message a unique identity. If a client loses the ack and then re-sends a message with the same ID, JetStream detects the duplicate and returns the ack without storing the message a second time.

#The Dedupe Window

The default dedupe window is 2 minutes, adjustable per stream:

nats stream edit ORDERS --dedupe-window 5m

--dedupe-window 5m extends the window to 5 minutes. Dedupe only guarantees the publish isn't duplicated; for idempotent processing you'll still need the additional mechanisms covered in episode 11.

#Conclusion

Episode 8 equipped you with full control over streams: defining subjects, choosing the Limits, Interest, or WorkQueue retention, deciding between file and memory storage, limiting max_age and max_bytes, configuring replication, and understanding sourcing, mirroring, and the dedupe window for exactly-once publishing.

Key takeaways:

• A stream captures subjects with retention, storage, and resource limit options.
• Limits for time-bounded event logs, Interest for many consumers, WorkQueue for job queues.
• File storage for data that must survive, memory for temporary data.
• max_bytes must be set so the disk doesn't fill up.
• replicas determines the copy count; it must be smaller than the node count.
• Sourcing copies messages between streams; mirroring is for cross-region DR.
• The dedupe window with Nats-Msg-Id prevents duplicate publishing.

In episode 9 next, we'll discuss JetStream consumers — the difference between pull and push consumers, durable versus ephemeral, ack, nack, ackWait, and max delivery semantics for redelivery, as well as deliver_policy and replay for determining where a consumer starts reading. This is where reading from a stream is configured with precision.