Learn NATS - Stream Manager & Work Queues

#Introduction

Episode 10 introduced data storage; episode 11 assembles it into the pattern most often used in production: the work queue. A reliable job queue where each job is processed by exactly one worker, and no job is lost even if a worker crashes.

We'll build a work queue with a WorkQueue stream and pull consumers, then close the last gap: exactly-once and idempotency. This is the backbone pattern of job processing systems.

#The Work Queue Pattern

#Basic Concept

A work queue differs from broadcast: each message (job) is processed by exactly one worker. In Core NATS, queue groups already do load balancing, but without storage guarantees. JetStream adds them with a WorkQueue stream.

publisher --> stream JOB (retention WorkQueue)
                  └── pull consumer
                        ├── worker-1 processes the job
                        ├── worker-2 processes the job
                        └── worker-3 processes the job

The publisher -> stream JOB (retention WorkQueue) flow shows jobs are first stored in the stream, then distributed to workers. Because the stream stores messages, jobs aren't lost even when all workers are busy.

#Creating a WorkQueue Stream

Create a stream with WorkQueue retention:

nats stream add JOB --subjects "job.>" --retention workqueue --storage file

The nats stream add JOB --subjects "job.>" --retention workqueue command creates a JOB stream that keeps each message for only one consumer. Once a job is acked, it's removed from the stream — no replay, no duplication between workers.

#Pull Consumers for Job Queues

#Why Pull

For many workers, a pull consumer is the right choice: each worker pulls jobs according to its capacity, applies natural backpressure, and doesn't receive more than it can handle.

nats consumer add JOB WORKER --pull --ack explicit --max-deliver 3 --ack-wait 60s

--pull --ack explicit creates the consumer workers use to pull jobs. --max-deliver 3 limits redelivery to at most 3 times, --ack-wait 60s gives the worker one minute to ack.

#Fetching Jobs from the CLI

From the CLI, a worker can pull jobs one at a time:

nats consumer next JOB WORKER

nats consumer next JOB WORKER fetches the next job from the stream. Because the consumer is pull-based, jobs don't arrive unrequested — the worker controls its own work rhythm.

#Go Client for Workers

In code, the worker pattern uses fetch:

nc, _ := nats.Connect("nats://localhost:4222")
js, _ := nc.JetStream()
 
for {
    msgs, _ := js.Fetch("JOB", "WORKER", 1, nats.MaxWait(30*time.Second))
    if len(msgs) == 0 {
        continue
    }
    for _, msg := range msgs {
        processJob(msg.Data)
        msg.Ack()
    }
}

js.Fetch("JOB", "WORKER", 1, nats.MaxWait(30*time.Second)) pulls one job with a 30-second wait, and msg.Ack() marks it done. This loop is the heart of nearly every NATS worker in production.

#Exactly-Once and Idempotency

#Publish Guarantees vs Process Guarantees

Two levels of guarantees must be distinguished:

• Exactly-once publish: a message enters the stream exactly once.
• Exactly-once processing: the process effect happens exactly once.

exactly-once publish  -> guaranteed by the server (dedupe window)
exactly-once process  -> guaranteed by the application (idempotency)

The separation of exactly-once publish -> guaranteed by the server matters: the dedupe window prevents duplicate messages from entering the stream, but a worker that crashes after processing but before acking will receive the message again. Handling the second part is the application's job.

#Publisher Dedupe Window

#Preventing Duplicate Publishing

As discussed in episode 8, publishers can include an ID for dedupe:

Publish a job with dedupe

import asyncio
import nats
 
async def main():
    nc = await nats.connect("nats://localhost:4222")
    js = nc.jetstream()
    await js.publish("job.payroll", b"payroll-2026-08", headers={"Nats-Msg-Id": "job-2026-08-10-001"})
 
asyncio.run(main())

headers={"Nats-Msg-Id": "job-2026-08-10-001"} gives the publish a unique identity. If the client loses the ack and re-sends with the same ID, JetStream rejects the duplicate. This ensures each job enters the stream exactly once.

#A Sufficient Dedupe Window

nats stream edit JOB --dedupe-window 10m

--dedupe-window 10m extends the dedupe window to 10 minutes. The value must be longer than the maximum time the client waits for an ack, so retries after a timeout still fall within the window.

#Idempotency Keys and Synchronization

#Handling Reprocessing

Even with publish dedupe, a worker can receive the same job twice — for example after a crash before acking. The solution is an idempotency key used to reject duplicate effects:

Idempotency with the KV store

from datetime import timedelta
 
entry = await kv.get("processed", key="job-2026-08-10-001")
if entry is None:
    entry = await kv.create("processed", key="job-2026-08-10-001", value=b"done", ttl=timedelta(days=7))
    if entry is not None:
        await process_job(payload)

The kv.create(...) logic only succeeds if the key doesn't exist. If two workers receive the same job, only one succeeds in creating the key — the other knows the job is already processed. That's exactly-once processing without an external database.

#Synchronization with a Database

For more complex transactions, synchronize with your application database:

worker receives job --> start DB transaction
                        ├── update job status = PROCESSING
                        ├── run the process
                        ├── update job status = DONE
                        └── commit + JetStream ack

The update job status = PROCESSING step in the database becomes a guard: if the job arrives again, the worker sees the DONE status and refuses to reprocess. Combining an idempotency key in KV with database transactions gives you a double protection layer.

#Conclusion

Episode 11 assembled the production work queue pattern: a WorkQueue stream storing each job for one worker, pull consumers giving workers rhythm control, the dedupe window for exactly-once publishing, and idempotency keys — via KV store or database — for exactly-once processing.

Key takeaways:

• A WorkQueue stream keeps each message for exactly one consumer.
• Pull consumers give workers natural backpressure and rhythm control.
• nats consumer next fetches a job; clients use Fetch or fetch in workers.
• The dedupe window with Nats-Msg-Id ensures publishing exactly once.
• Exactly-once processing requires an idempotency key on the application side.
• The KV store or a database serves as the reprocessing guard.

In episode 12 next, we'll discuss accounts & multi-tenancy — logical isolation between teams and services via accounts, users with subscribe, publish, and response permissions, then nsc for creating JWT-based accounts and users, as well as exporting and importing subjects between accounts. One NATS server starts serving many tenants.