Learn NATS - Troubleshooting

#Introduction

No system runs without problems. Episode 19 arms you against the most common issues in NATS: slow consumers, endless redelivery, full storage, lost quorum, and subject collisions.

For each problem, we'll cover its symptoms, its causes, and how to put out the fire with the diagnostic tools you already know. Let's start with the most frequent problem.

#Slow Consumers and Backpressure

#Symptoms and Causes

A slow consumer occurs when a consumer can't keep up with the publisher's speed. Symptoms: nats consumer report shows pending climbing, and messages pile up in the stream.

nats consumer report ORDERS
nats server check consumer

nats consumer report ORDERS shows pending per consumer. nats server check consumer verifies the health of all consumers in one step.

#Solutions

• Add workers to the same queue group or pull consumer.
• Increase the number of pull consumers so the load is divided.
• Slow the publisher down with smaller batches.
• Inspect the worker process: there may be a blocking call or a leak.

#Endless Redelivery

#The Redelivery Loop

Endless redelivery happens when a worker never manages to ack even though max_deliver isn't reached — usually because ack_wait is too short for long-running work, or the handler uses the wrong ack.

nats consumer report ORDERS

nats consumer report ORDERS shows the Redeliveries column. A high and steadily rising number means messages keep failing to be processed.

#Solutions

• Extend ack_wait to match normal processing duration.
• Inspect the handler code: make sure msg.Ack() is called on the success path.
• Raise max_deliver only if truly necessary, then route to the dead letter.

nats consumer edit ORDERS WORKER --ack-wait 5m --max-deliver 5

--ack-wait 5m gives the process more time. A practical rule: ack_wait must be larger than your worst reasonable processing time.

#Full Storage Due to max_bytes

#A Disk That Keeps Growing

An unbounded stream will keep growing until it fills the disk. Symptoms: server crashes, publishing fails, and the log shows storage errors.

nats stream report
df -h

nats stream report shows storage usage per stream; df -h checks the server's disk space. If the disk is nearly full, action must be taken immediately.

#Solutions

• Set max_bytes on every stream.
• Shorten max_age for data that doesn't need to be kept long.
• Remove unused streams or consumers.

nats stream edit ORDERS --max-bytes 10G --max-age 30d

--max-bytes 10G --max-age 30d limits the stream to 10 GB or 30 days, whichever comes first. These two bounds prevent the disk from exploding.

#Lost Quorum

#A Cluster That Fails to Reach Consensus

Quorum loss happens when the number of healthy nodes is less than a majority. Symptoms: a replicas 3 stream shows declining health, publishes to the stream fail, and no leader is elected.

nats stream report
nats server check cluster

nats server check cluster verifies cluster health and quorum. If a cluster loses quorum, streams can't accept writes.

#Solutions

• Bring the dead node back as soon as possible.
• Keep an odd node count for maximum tolerance.
• Don't restart all nodes at the same time.
• Check network connectivity between nodes on port 6222.

#Subject Collisions

#Two Streams Fighting Over a Subject

A subject collision occurs when two streams capture the same subject within one account. NATS rejects creating the second stream with the subject already covered by another stream error.

nats stream report
nats stream info ORDERS

nats stream info ORDERS shows the list of captured subjects. Comparing subject lists across streams reveals overlaps.

#Solutions

• Design a subject schema with unique prefixes per domain.
• Use wildcards carefully — orders.> and orders.* can collide.
• If a collision is unavoidable, merge the subjects into one stream.

#The Complete Diagnostic Toolkit

#A Structured Investigation Flow

When something goes wrong, run the following steps in order:

nats server check
nats stream report
nats consumer report
nats server check ping

nats server check ping checks server latency. Add debug logging when needed:

nats-server -D

The -D flag enables debug logging. Turn it on only during investigations, then turn it off — full debug logging in production degrades performance as discussed in episode 18.

#Conclusion

Episode 19 trained you to put out fires: slow consumers addressed by adding workers and pull consumers, endless redelivery fixed through the right ack_wait, full storage prevented with max_bytes and max_age limits, quorum loss handled by keeping an odd node count, and subject collisions avoided with disciplined naming schemes.

Key takeaways:

• Pending climbing steadily is a sign of a slow consumer; add workers or fix the handler.
• High redelivery usually means ack_wait is too short.
• Every stream must have max_bytes and max_age so the disk stays safe.
• Quorum loss requires a healthy node count greater than a majority.
• Subject collisions are prevented with careful naming schemes and wildcards.
• nats server check and the two reports are your primary diagnostic weapons.

In episode 20 next, we'll discuss the latest stable features in v2.14 — the release evolution from v2.10 to v2.12 then v2.14, upgrade and backward compatibility guidance, plus the 2026 features: high-throughput publishing, server-side message scheduling, sourcing and mirroring improvements, and the in_client and out_client metrics. Your knowledge will be updated to the latest release.