Learn RabbitMQ - Work Queues - Task Distribution

#Introduction

The Hello World pattern in episode 4 only used one consumer. In the real world, a single consumer is not enough for heavy tasks like rendering videos or sending thousands of emails. This is where the work queue comes in: one queue, many workers, and messages distributed automatically.

But building a correct work queue is not just running several consumers at once. There are three mechanisms you must master: manual acknowledgment so messages aren't lost when a worker crashes, durability so messages survive a broker restart, and prefetch count so messages don't pile up on one slow worker.

This episode is the point where RabbitMQ shows its advantage over direct communication: the combination of ack, durability, and QoS produces a system that guarantees every task gets processed, even when workers and brokers have problems.

#Work Queue Pattern

#Round-Robin Dispatching

By default, RabbitMQ delivers messages to consumers in a round-robin fashion: the first message to worker A, the second to worker B, the third to A again, and so on. This distribution is even and simple — every worker receives roughly the same number of messages.

The problem is that round-robin ignores how busy each worker is. If worker A is very slow, messages keep flowing toward it and piling up. This is why we need fair dispatch, covered at the end of the episode.

queue ──► worker A (messages 1, 3, 5)
     └──► worker B (messages 2, 4, 6)

#Running Multiple Workers

To run several workers at once, just run the same consumer in multiple processes or threads:

python worker.py &
python worker.py &

Declare the same queue in all workers. A single message only reaches one worker — it is not duplicated to all of them.

#Message Acknowledgments

#Manual vs Automatic Acknowledgment

The auto_ack=True we used in episode 4 is dangerous: as soon as the message is sent to the worker, the broker deletes it from the queue, without knowing whether the worker successfully processed it. If the worker crashes mid-process, the message is lost forever.

The solution is manual acknowledgment:

Consumer with manual ack

import pika, time
 
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue="task_queue", durable=True)
 
def callback(ch, method, properties, body):
    print(f"memproses: {body}")
    time.sleep(2)
    ch.basic_ack(delivery_tag=method.delivery_tag)
 
channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue="task_queue", on_message_callback=callback)
print("menunggu task...")
channel.start_consuming()

The ch.basic_ack call sends a Basic.Ack to the broker. Until the ack is received, the message is considered "not done" and is not deleted.

#Basic.Nack and Basic.Reject

Besides ack, AMQP provides Basic.Reject to reject a single message, and Basic.Nack to reject one or many messages at once. Both can mark whether the message should be requeued or discarded to a dead letter exchange:

Reject a message with requeue

ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)

The ch.basic_nack command with requeue=False sends the message to a dead letter queue if a DLX is configured — full topic in episode 11.

#Delivery Timeout

RabbitMQ 3.12 introduced the delivery timeout (delivery_timeout): if the consumer doesn't send an ack for a message within a certain time limit, the broker will close the consumer's connection. The default value is 30 minutes, and it can be set in rabbitmq.conf:

consumer_timeout = 600000

The consumer_timeout value above is in milliseconds, and you can disable it with consumer_timeout = 0 (not recommended).

#Message Durability

#Durable Queues and Persistent Messages

Declaring the queue alone is not enough to guarantee messages survive a broker restart. There are two layers: the queue must be durable, and the messages must be persistent (delivery_mode=2). Both are required — one alone is not enough.

Publish a persistent message

channel.queue_declare(queue="task_queue", durable=True)
channel.basic_publish(
    exchange="",
    routing_key="task_queue",
    body=b"task berat",
    properties=pika.BasicProperties(delivery_mode=2),
)

If the queue is re-declared without durable=True in the same environment, RabbitMQ will refuse because the declarations are incompatible — an error message will appear in the log.

#Performance Trade-Off

Persistence has a cost: every persistent message is written to disk before the broker sends the publisher ack. This increases publish latency. The trade-off: durability for important data, transient for data that can be lost, like logs or ephemeral notifications.

#Fair Dispatch with Prefetch

#The Prefetch Count Concept

Round-robin doesn't care about worker speed. Prefetch count changes this behavior: it limits how many messages can be sent to one consumer before that consumer sends an ack. With prefetch_count=1, a worker only receives one message at a time, and the next message is only sent after the current one is acked.

rabbitmqctl eval 'application:get_env(rabbit, channel_prefetch_count).'

Or in the client SDK, use channel.basic_qos(prefetch_count=1) like the example above. A value of 1 is the fairest but adds round-trip overhead; values of 10-100 are usually optimal for throughput.

#Optimal Prefetch Values

There is no universal number. Start with prefetch_count=1 for long-running heavy tasks, and increase it if messages are small and processed quickly. Monitor consumer utilization in the Management UI; if it stays at 90 percent or above, the prefetch is too high.

#Conclusion

In episode 5 you built a correct work queue: round-robin dispatching, manual acknowledgment with Basic.Ack/Nack/Reject, two-layer durability for queues and messages, and fair dispatch with prefetch count so each worker works within its capacity.

Key takeaways:

• Round-robin distributes messages evenly but ignores how busy workers are.
• Manual ack prevents message loss when a worker crashes mid-process.
• Basic.Nack with requeue=False discards the message to the DLX.
• The queue must be durable and messages persistent to survive a restart.
• Persistence adds latency — choose based on your data needs.
• prefetch_count=1 guarantees fairness; the optimal value depends on the workload.
• Delivery timeout disconnects consumers that never ack.

In the next episode we will let go of the default exchange and learn about exchange types: direct, fanout, topic, headers, and the default exchange, complete with routing keys, binding keys, and implementing the publish/subscribe pattern. This concept turns RabbitMQ from a mere queue into a flexible message router!