Learn RabbitMQ - Flow Control & Backpressure

#Introduction

Imagine a store whose entrance is never closed: if thousands of visitors arrive at once, the store will burst. Message brokers face the same problem — if publishers flood the broker non-stop, memory and disk run out and the entire node can crash.

RabbitMQ solves this with flow control: a series of mechanisms that slow down publishers when the broker is overwhelmed. Instead of crashing, the broker blocks connections or withholds credit between internal processes. Consumers don't need to know anything, but publishers must understand these mechanisms so their applications don't stall.

This episode dissects the flow control layers: from the lowest TCP backpressure and credit-based flow control inside the node, to the memory alarm that blocks all connections. Then we build publisher confirms — the official way for publishers to know their messages were really accepted by the broker — and practical strategies for handling backpressure.

#Flow Control Mechanisms

#TCP Backpressure and Credit-Based Flow Control

When consumers are slow, queues pile up, and the node's memory starts to fill. RabbitMQ stops reading data from the TCP socket of the publishing connection, so the client's receive buffer fills up and TCP holds back delivery. This is called TCP backpressure — the most basic mechanism.

Inside the node, Erlang processes communicate through credit-based flow control: every process must have credit before sending a message to another process. If the receiving process is busy, credit runs thin and the sender automatically slows down. This mechanism keeps RabbitMQ from flooding internally.

#The Connection Blocked State and Memory Alarm

When a critical condition is reached, RabbitMQ blocks all publishing connections and sends a connection.blocked notification to clients:

Connection.blocked reason=memory_limit
Connection.unblocked

Modern client SDKs translate this into callbacks. In pika:

Blocked and unblocked callbacks

def on_blocked(connection, reason):
    print(f"publisher diblokir: {reason}")
 
def on_unblocked(connection):
    print("publisher dibuka kembali")
 
connection.add_on_connection_blocked_callback(on_blocked)
connection.add_on_connection_unblocked_callback(on_unblocked)

The add_on_connection_blocked_callback callback tells the application when the broker blocks publishing because of a memory alarm. A well-behaved application pauses publishing and holds a buffer internally.

#Publisher Confirms

#Enabling Confirm Mode

Publisher confirms is a feature that tells the publisher the message has been accepted by the broker (and, if persistent, written to disk). Without confirms, publishers can never be sure their messages weren't lost.

Enable confirm mode

channel.confirm_delivery()

Once confirm_delivery() is enabled, every publish receives a confirmation or a nack.

#Synchronous and Asynchronous Confirms

Confirms can be waited for synchronously per message:

Synchronous confirm with timeout

try:
    channel.basic_publish(exchange="", routing_key="q", body=b"data")
    channel.confirm_delivery()  # hanya contoh; aktivasi sekali
except pika.exceptions.UnroutableError:
    print("pesan gagal di-routing")

More efficient: batch confirms — publish many messages, then wait for a single confirmation at the end of the batch. For the highest throughput, use asynchronous confirms with a callback that tracks each delivery tag.

#Handling Nacks

If the broker sends a nack for a message, the message was rejected (for example, a full queue with reject-publish). The publisher must keep that message and resend it or put it in an internal queue. Never stay silent — a nacked message is considered lost from the publisher's side.

#Handling Backpressure

#Publisher Throttling Strategies

When connection.blocked occurs, don't keep forcing publishes. The right strategy:

• Pause publishing and hold messages in an internal buffer until unblocked.
• Apply a buffer size limit so the application's memory doesn't blow up too.
• Reduce batch size and lower publishing intensity.

#Queue Limits and Disk Monitoring

Backpressure at the broker is usually triggered by queues growing out of control. Set x-max-length or x-overflow to limit them, and monitor disk and memory metrics proactively:

rabbitmqctl status | grep -A 2 -i memory
rabbitmqctl status | grep -A 2 -i disk

The rabbitmqctl status command shows current memory and disk usage — both are the primary alarms that trigger the blocked state.

#Conclusion

In episode 14 you understood all of RabbitMQ's flow control layers: TCP backpressure, credit-based flow control, the memory alarm, and the connection blocked state; enabled both synchronous and asynchronous publisher confirms; and put together publisher throttling and disk monitoring strategies.

Key takeaways:

• TCP backpressure and credit control prevent internal flooding.
• The memory alarm blocks connections and sends connection.blocked.
• Applications must pause publishing when a connection is blocked.
• Publisher confirms give certainty that the broker accepted messages.
• Batch confirms boost throughput; async confirms are the highest.
• A nack means the message was rejected — handle it, don't ignore it.
• Set queue limits and monitor memory and disk proactively.

In the next episode we will manage user management and authentication — creating users through the Management UI and rabbitmqctl, assigning user tags, applying password policies, and comparing authentication backends: internal, LDAP, OAuth 2.0, x509 certificates, and the HTTP backend. RabbitMQ security starts with who is allowed to log in!