Learn RabbitMQ - Memory & Resource Management

#Introduction

Every process that doesn't manage its memory will be killed by its own memory — and RabbitMQ is no exception. If messages pile up and memory exceeds the limit, the node crashes. If the disk fills up, the node stops processing. Managing resources is not optional; it's an inseparable part of operating a broker.

This episode covers the two main alarms that protect RabbitMQ: the memory alarm and the disk alarm. You'll understand how node memory is calculated, when messages are paged to disk, how lazy queues work, and finally the resource limits that can be set per node and per vhost.

This is an operationally important episode: without understanding these alarms, you won't be able to read the symptoms of a broker dying under load.

#Memory Management

#Memory Alarms and Thresholds

RabbitMQ monitors node memory usage. When it passes the high watermark, all publishing connections are blocked (as in episode 14). The default is 0.4 — 40 percent of total memory:

Set the memory high watermark

vm_memory_high_watermark.relative = 0.6

Raising the value gives queues more headroom, but increases the OOM risk. Lowering it makes the broker safer at the cost of buffer capacity.

#Memory Calculation Strategies and Paging

RabbitMQ calculates memory from several sources: queue messages, Erlang processes, connections, channels, and other buffers. There are different calculation strategies (rss vs allocated), but the default is good enough for most cases.

The rss strategy uses the resident set size of the Erlang process, while allocated estimates memory from runtime-reported allocations. allocated usually signals earlier, while rss is more accurate near real OOM conditions. Change the strategy only if you have a measured reason — for example, memory tends to spike fast without a visible trigger.

When memory approaches the limit, the node starts paging queue messages to disk to relieve memory pressure:

Control paging to disk

vm_memory_high_watermark_paging_ratio = 0.5

vm_memory_high_watermark_paging_ratio at 0.5 means paging starts at 50 percent of the high watermark — before the full alarm blocks publishers. Paging keeps the node alive but adds disk load.

#Monitoring Memory Usage and Paging

To see how many messages have been paged to disk, compare the messages column with messages_paged_out:

Check total messages vs paged-out messages

rabbitmqctl list_queues name messages messages_paged_out

messages_paged_out shows messages written to disk because of memory pressure. If the number keeps rising with load, the node is under memory pressure — an early sign to lower the high watermark or shorten the queue. Also monitor the memory section in rabbitmqctl status to see usage sources per component.

#Lazy Queues

#Moving Messages to Disk Earlier

A lazy queue is a classic queue that writes messages to disk immediately when received, instead of holding them in memory first. The consequence: node memory is far safer for large queues, at the cost of slightly higher latency because every publish touches disk.

Declare a lazy queue

rabbitmqadmin declare queue name=ingest \
  arguments='{"x-queue-type":"classic","x-queue-mode":"lazy"}'

#Use Cases and Performance Implications

Lazy queues suit queues that are consistently large: mass data ingest, log buffers, or queues whose consumers often lag. Trade-off: publish throughput drops compared to the default mode, while consumption works normally. If a queue is usually short and drained quickly, the default mode is better.

#Resource Limits

#Connection, Channel, and Queue Limits

Nodes can have resource limits set with rabbitmqctl set_node_limits, and per vhost with set_vhost_limits (which we saw in episode 13):

Limit file descriptors per vhost

rabbitmqctl set_vhost_limits -p tenant_b '{"max-connections":50,"max-queues":200}'

The max-connections and max-queues limits above protect the node from one overly greedy tenant. For channels, the per-connection default limit is 2047.

#Per-Vhost and File Descriptor Limits

Besides memory and disk, file descriptors are an often-ignored resource: every connection uses several FDs. Raise ulimit -n according to your estimated connection count, and set per-vhost connection limits to keep fairness between tenants.

#Setting Limits for the Whole Node

Besides per vhost, limits can also be set at the node level to protect the broker from overall load:

Limit connections and queues at the node level

rabbitmqctl set_node_limits '{"max-connections":1000,"max-queues":5000}'

set_node_limits limits the total connections and queues across the whole node. Combine it with per-vhost limits: the node limit is the last safety net, while per-vhost limits ensure each tenant gets a fair share. The low_connection and low_queue_count alarms will appear in the Management UI when approaching the limits.

#Disk Management

#Disk Space Alarms

If free disk space drops below the disk free limit, the node stops processing messages to prevent total disk exhaustion. The default is relative — 50 MB or a certain percentage:

Set an absolute disk free limit

disk_free_limit.absolute = 2GB

disk_free_limit.absolute = 2GB ensures the broker always keeps at least 2 GB of free disk space.

#Log Rotation and Message Store Compaction

RabbitMQ logs keep growing; set up rotation so they don't fill the disk. The message store also manages segment files containing already-acked messages — old segments are deleted or compacted automatically over time. Monitor disk usage in the Management UI and alert before hitting the disk alarm.

Warning

Don't raise the memory high watermark too high without measuring. A late alarm means the OOM killer executes the node — and recovering from a crash is more painful than a blocked connection.

#Conclusion

In episode 23 you managed memory and disk alarms, memory calculation and paging strategies, lazy queues, per-vhost connection and queue limits, and log rotation and message store compaction.

Key takeaways:

The memory alarm blocks publishers when memory passes the high watermark.
Paging writes queue messages to disk to relieve memory pressure.
Lazy queues write messages to disk earlier for large queues.
Limit connections and queues per vhost for fairness between tenants.
File descriptors limit the node's connection capacity.
The disk alarm stops processing when the disk is almost full.
Log rotation and store compaction keep the disk healthy.

In the next episode we will build monitoring and observability — reading metrics from the Management UI and HTTP API, using rabbitmqadmin and rabbitmqctl, integrating Prometheus with the metrics endpoint, putting together Grafana dashboards, and managing logs with levels, locations, and centralized aggregation. Now your broker will be able to "talk"!