Learn RabbitMQ - Federation & Shovel

#Introduction

So far everything has been within a single cluster — one group of nodes sharing metadata and a cookie. But the real world has farther-reaching needs: brokers in the Jakarta and Singapore regions that must exchange messages with each other, or an on-premise broker sending data to the cloud. Connecting separated brokers is exactly what Federation and Shovel do.

These two plugins take RabbitMQ out of a single cluster and turn it into a network of brokers. Federation pulls messages from upstream exchanges or queues to downstream ones continuously. Shovel moves messages from one point to another — between brokers, between vhosts, even between clusters — like a reliable pipe.

This episode explains how both work, when to use which, and how to design multi-datacenter topologies and message transfer patterns that withstand WAN network failures.

#The Federation Plugin

#Federating Exchanges and Queues

Federation works with an upstream-downstream model: the downstream broker pulls messages from the upstream broker. For exchanges, the downstream creates a "link" that ingests messages from the upstream exchange and broadcasts them to locally bound queues. For queues, the downstream pulls messages from the upstream queue.

Federation's advantages: messages are consumed locally, and links keep running — if the upstream dies, the link automatically retries once it recovers.

#Configuring Federation

Enable the plugin, define the upstream, then set a policy:

rabbitmq-plugins enable rabbitmq_federation
rabbitmqctl set_parameter federation-upstream dc-sg \
  '{"uri":"amqp://arman:pass@broker-sg:5672","expires":3600000}'

Then apply a policy that determines which exchanges are federated:

rabbitmqctl set_policy federation-orders '^orders\.' \
  '{"federation-upstream-set":"dc-sg"}' --apply-to exchanges

The set_policy command above makes every exchange starting with orders. federated to the dc-sg upstream.

#Federating Queues and Monitoring Links

Federation isn't only for exchanges; queues can also be federated with --apply-to queues. The downstream pulls messages from upstream queues whose names match the pattern:

rabbitmqctl set_policy federation-q '^notif\.' \
  '{"federation-upstream-set":"dc-sg"}' --apply-to queues

To make sure links run healthily, check the status of each link:

rabbitmqctl list_federation_links status

list_federation_links shows each link with its status and last message. A failed link shows a traceable error — usually wrong credentials or an unreachable upstream.

#Use Cases: Multi-Datacenter and Geo-Distribution

Federation fits: cross-region topology replication, local consumption in each DC for low latency, and distributing data to regional branches. Because the downstream pulls, you can control when and how much is synchronized.

#The Shovel Plugin

#Point-to-Point Forwarding

Shovel moves messages from a source (queue or exchange) to a destination (exchange or queue) — across vhosts, brokers, or clusters. Unlike Federation, Shovel is more precise: you specify the exact source and destination.

#Static Shovels via Configuration

Static shovels are defined in rabbitmq.conf:

shovels.migrate-orders = [
  {source, [{uris, ["amqp://src-host:5672"]}, {queue, "orders.queue"}]},
  {destination, [{uris, ["amqp://dst-host:5672"]}, {queue, "orders.import"}]}
]

The shovel above moves messages from the orders.queue queue on the source broker to the orders.import queue on the destination broker.

#Dynamic Shovels via the Management UI

Dynamic shovels are managed through the Management UI or API without restart — ideal for temporary setups like migrations. In the UI: open the Admin > Shovels > Add a shovel tab, fill in the source and destination, then activate.

#Advanced Shovel Configuration

Two important parameters for reliable transfer over slow networks: ack_mode and reconnect_delay:

shovels.reliable = [
  {source, [{uris, ["amqp://src-host:5672"]}, {queue, "jobs.queue"}]},
  {destination, [{uris, ["amqp://dst-host:5672"]}, {queue, "jobs.import"}]},
  {ack_mode, on_confirm},
  {reconnect_delay, 5}
]

ack_mode=on_confirm makes the shovel only delete a message from the source after the destination confirms receipt — guaranteeing no message is lost along the way. reconnect_delay=5 sets the delay in seconds before retrying a dropped connection. The combination makes the shovel resilient to WAN network fluctuations.

#Shovel vs Federation and the WAN

#Comparison and Selection

• Federation — continuous replication for many exchanges/queues, downstream pulls messages, one-way synchronization.
• Shovel — controlled, precise message transfer, suitable for migrations, one-off forwarding, and custom topologies.

Rule of thumb: need continuous synchronization between clusters? Use Federation. Need to move messages from one point to a specific point, like a data migration? Use Shovel.

#Reliable WAN Connections

Broker-to-broker connections crossing a WAN face high latency and interruptions. Both plugins handle this with automatic reconnection and internal buffering. For multi-DC topologies, consider data locality: consumers read from the nearest broker (downstream), instead of calling upstream over the WAN every time.

Don't forget to monitor the link's internal buffer length: if the downstream is slower than the upstream, the buffer grows and adds latency — a signal that the downstream needs scaling or the publish rate needs to be reduced. WAN latency can't be removed, but it can be managed with the right configuration.

#Conclusion

In episode 26 you replicated exchanges and queues between clusters with Federation, moved messages point-to-point with static and dynamic Shovels, compared when to use each, and designed reliable WAN connections for multi-datacenter topologies.

Key takeaways:

• Federation uses an upstream-downstream model; the downstream pulls.
• Federation fits continuous cross-region replication.
• Shovel moves messages from one point to another precisely.
• Static shovels are defined in configuration; dynamic ones via the Management UI.
• Federation for synchronization, Shovel for migration and forwarding.
• Both support automatic reconnection over WAN networks.
• Prioritize data locality so consumers read from the nearest broker.

In the next episode we will open multi-protocol support — enabling the MQTT plugin for IoT, STOMP for browsers, and WebSockets for real-time messaging, understanding QoS level mapping and the Web-STOMP and SockJS patterns. One RabbitMQ, many protocols at different doors!