Learn RabbitMQ - RabbitMQ Clustering

#Introduction

Every RabbitMQ running on a single machine is a single point of failure. If that node dies, all its queues and messages are lost, and every service depending on it stops. For production, you need more than one node: this is a cluster.

A RabbitMQ cluster combines several nodes into one logical broker. Queues, exchanges, bindings, users, and permissions are synchronized across all nodes. Clients just connect to any node and see the same topology. The interesting part: inside a cluster, routing decisions and node state keep being shared — some resources are replicated, others are only stored on one node.

This episode covers how to form a cluster, node discovery mechanisms, inter-node security with the Erlang cookie, adding and removing nodes, and the thorniest issue in distributed systems: network partitions and strategies for handling them.

#Cluster Fundamentals

#Basic Cluster Concepts

In a cluster, every node has a unique name like rabbit@node1. Metadata such as users, vhosts, queues, exchanges, and bindings is replicated to all nodes — so the topology is always consistent no matter where clients connect. However, queue message contents are not replicated by default; we'll resolve this in episode 20 with quorum queues.

#Erlang Cookie Authentication

All nodes must share the Erlang cookie — a shared secret that authenticates nodes to each other. The cookie is stored in a file (usually ~/.erlang.cookie). If cookies differ, a node can't join the cluster. Store the cookie with secure permissions and never commit it to a repository.

#Classic Clustering vs Peer Discovery

RabbitMQ offers two formation approaches:

• Classic clustering — a first node is chosen, then other nodes join manually.
• Peer discovery — nodes find each other via DNS, cloud APIs (AWS), or Kubernetes. This approach suits automation better.

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_dns
cluster_formation.dns.hostname = rabbits.discovery.svc.cluster.local

#Checking Cluster Status and Health

Once the cluster is formed, check the current state with rabbitmqctl cluster_status:

rabbitmqctl cluster_status

The cluster_status command output shows the list of running nodes, each node's disk status, and the active partition handling mode. Get used to running it after every cluster change — and make it part of your troubleshooting runbook. Often one stopped node causes panic, when cluster_status actually shows the other nodes are still healthy and serving.

#Cluster Setup

#Creating a Manual Cluster

Say you have three nodes: rabbit@node1, rabbit@node2, rabbit@node3. Start node2 and node3, then join them to node1:

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app
rabbitmqctl cluster_status

The stop_app → reset → join_cluster → start_app sequence is the standard procedure for joining a cluster. reset clears the node's state so it can accept cluster metadata.

#Adding and Removing Nodes

To add a node, repeat the procedure above. To remove a node, run from another node:

rabbitmqctl forget_cluster_node rabbit@node2

The forget_cluster_node command ejects the node from the cluster. If that node is permanently dead, run the command with the --offline flag from a still-running node.

#Handling Permanently Lost Nodes

If a node fails and won't come back — for example, its machine was replaced — the node must be ejected from the cluster in a special way. From another node that's still alive:

rabbitmqctl forget_cluster_node rabbit@node2 --offline

The --offline flag is needed when the node being forgotten is unreachable. This procedure stops the remaining cluster from waiting on the dead node and keeps the metadata consistent. Make sure important data from the dead node is already handled before running this command.

#Cluster Partitioning and Split-Brain

If the network between nodes is cut, the cluster can split into several "islands", each considering itself whole — this is split-brain. The chosen partition mode determines what happens when the network recovers:

• pause_minority — minority nodes stop; avoids conflicts (default).
• autoheal — the minority partition restarts and aligns with the majority.
• ignore — the partition is left as-is; risks data conflicts.

cluster_partition_handling = autoheal

#Cluster Behavior

#Queue Distribution and Load Balancing

Non-replicated queues only live on the node where the queue was declared. Client connections must be directed to the node owning the queue — which is why clients usually use several nodes as endpoints. Connection load balancing can be done with DNS round-robin or a TCP load balancer in front of port 5672.

There's one important consequence: because metadata is replicated, every node knows where each queue lives. A client connected to any node can still publish or consume — the broker receiving the operation forwards it to the queue's owner node. What's unavailable is only access to the contents of non-replicated queues when their owner node is down.

#Node Failure Handling

If a node dies, other nodes keep serving the same topology, but non-replicated queues on the dead node can't be accessed until the node returns. This is why quorum queues matter: their messages remain accessible from other live nodes.

If a node returns after a pause, it automatically re-synchronizes metadata with other nodes on reconnect, and its own queues become accessible again. For queues whose data is crucial, episode 20 will show how a quorum queue makes recovery independent of any single node.

#Conclusion

In episode 19 you formed a multi-node cluster with the Erlang cookie, understood classic clustering vs peer discovery, added and removed nodes, and handled network partitions with the pause_minority, autoheal, and ignore modes.

Key takeaways:

• Metadata is replicated to all nodes; queue contents are not.
• The Erlang cookie authenticates nodes; keep it secret and don't commit it.
• Cluster join procedure: stop_app, reset, join_cluster, start_app.
• Peer discovery eases automatic formation via DNS/AWS/K8s.
• Split-brain is handled with pause_minority, autoheal, or ignore.
• A TCP load balancer helps distribute client connections.
• Non-replicated queues can't be accessed when their owner node is down.

In the next episode we will achieve high availability with quorum queues — building replication on the Raft consensus algorithm, comparing it with classic mirrored queues, configuring the replication factor, handling poison messages, and migrating from classic queues. This is the queue type you must use for production!