Learn RabbitMQ - High Availability - Quorum Queues

#Introduction

In episode 19 we found the cluster's weakness: non-replicated queues only live on one node, and their messages are lost if that node dies. The long-term solution to this problem is the quorum queue — a queue type based on the Raft consensus algorithm that replicates messages across several nodes and guarantees data safety as long as a majority of nodes are alive.

Quorum queues are the official replacement for the old classic mirrored queues mechanism, which has been deprecated since RabbitMQ 3.8 and will be removed in version 4.0. Since its introduction in 3.8, the quorum queue has become the recommended queue type for production: durable by design, high message accuracy, and immune to the failure of some nodes.

This episode covers how Raft works behind a quorum queue, replication configuration, poison message handling with delivery limits, the deduplication feature, performance characteristics, and migration guidance from classic queues.

#Introducing Quorum Queues

#Raft Consensus and Leader-Follower Replicas

A quorum queue uses Raft, the same consensus algorithm used by many modern distributed systems. Every queue has one leader and several followers spread across different nodes. All publishing and consuming goes through the leader; followers keep syncing data. A message is considered safe after being replicated to a majority of nodes.

If the leader dies, surviving followers hold an election and promote a new leader. The queue stays available as long as a majority of replicas are alive. The formula: with N replicas, the queue survives the failure of up to (N-1)/2 nodes.

This safety model is rooted in how Raft works: a new publish is only considered successful after a majority of replicas confirm the write to disk, not merely to memory. That's why a quorum queue is far safer than a classic queue, which writes to disk at certain intervals.

#Quorum vs Classic Mirrored Queues

The conclusion: for high availability, the quorum queue is the clear choice.

#Configuring Quorum Queues

#Declaring with x-queue-type

A quorum queue is declared with the x-queue-type=quorum argument:

rabbitmqadmin declare queue name=orders.q durable=true \
  arguments='{"x-queue-type":"quorum"}'

The replication factor is determined automatically from the number of available nodes, or set explicitly with x-quorum-initial-group-size:

rabbitmqadmin declare queue name=orders.q durable=true \
  arguments='{"x-queue-type":"quorum","x-quorum-initial-group-size":3}'

x-quorum-initial-group-size=3 spreads the queue's replicas across three nodes. An odd number of replicas is recommended so a leader election always produces a clear majority.

#Dead Lettering and Poison Message Handling

A quorum queue supports dead lettering with special semantics: instead of moving a message to the DLX immediately, a poison message is given a delivery limit. If the limit is exceeded, the message is moved to the DLX. This prevents bad messages from looping endlessly in the queue:

rabbitmqadmin declare queue name=orders.q durable=true \
  arguments='{"x-queue-type":"quorum","x-delivery-limit":5}'

With x-delivery-limit=5, messages that fail to process more than 5 times are discarded to the DLX (if configured). The delivery limit has been available since RabbitMQ 3.10.

#Features and Considerations

#Message Deduplication

Quorum queues (3.11+) support deduplication: if a message is published again with the same x-deduplication-header within a certain window, the duplicate is discarded. This feature is useful for publisher idempotency:

Publish with a deduplication header

properties = pika.BasicProperties(
    headers={"x-deduplication-header": "order-A-001"},
)
channel.basic_publish(exchange="", routing_key="orders.q",
                      body=b'{}', properties=properties)

The x-deduplication-header header makes re-publishing with the same value not duplicate the message in the queue.

One important note on deduplication: this feature only removes duplicates at the publish-to-queue stage, not guaranteeing once-processing at the consumer side. For end-to-end guarantees, still design consumers that are idempotent based on your own message_id or business headers.

#Performance Characteristics and Migration

A quorum queue is somewhat slower than a classic queue on a single node because of consensus costs and aggressive disk writes. But its reliability is far higher. For migrating from classic queues, use Shovel (episode 26) to copy messages with minimal downtime, or blue-green with parallel queues. Avoid changing the queue type in place — a declaration with a different type will be rejected by the broker.

#Monitoring Quorum Queues

To monitor replica health and leader presence:

rabbitmqctl list_quorum_queues name type leader online members

list_quorum_queues shows the leader, online nodes, and all replica members. If a queue shows an empty leader or online members fewer than a majority, the leader election hasn't completed — usually due to network issues between nodes.

#Quorum Queue Limitations

Not every classic queue feature is available in quorum queues. Some unsupported things: transient and exclusive queues, per-message TTL (x-message-ttl), queue expiration (x-expires), priority queues, and lazy mode. If your patterns depend on these features, consider a classic queue with its HA consequences, or redesign the pattern to fit the quorum model.

A quorum queue still supports dead lettering, but with the delivery limit semantics already discussed — not the direct move of every failed message. Understand these limitations before choosing a queue type, because a declaration with unsupported arguments will be rejected by the broker.

#Conclusion

In episode 20 you understood the Raft consensus behind quorum queues, compared them with classic mirrored queues, configured replication and delivery limits, used the deduplication feature, and considered migrating from classic queues.

Key takeaways:

• Quorum queues are Raft-based: messages are safe as long as a majority of replicas are alive.
• One leader serves publishing and consuming; followers keep syncing.
• Mirrored queues are deprecated — switch to quorum.
• x-queue-type=quorum marks the queue; the replication factor can be set explicitly.
• Delivery limits prevent poison messages from looping endlessly.
• Deduplication maintains publisher idempotency.
• The best migration is via shovel or blue-green, not changing types in place.

In the next episode we will enter RabbitMQ Streams — a Kafka-style append-only log, non-destructive consumption with offsets, replication and replay from any position, retention policies, and when to use streams instead of queues. This opens a new dimension of RabbitMQ for high throughput!