Learn Apache Kafka - Quotas & Rate Limiting

#Introduction

In a cluster shared by many teams, one greedy application can consume all the broker bandwidth and starve other applications — a problem known as the noisy neighbor. Quotas in Kafka address this by limiting how much resource each user or client-id may use.

Episode 19 covers the quota types — producer, consumer, request, and controller mutation — how to configure them at the user, client-id, or combined level, the throttling mechanics behind the scenes, and using quotas for multi-tenancy, cost control, and SLAs.

#Quota Types

#Producer and Consumer Quotas

Two main quotas govern data throughput in bytes per second:

• Producer quota: limits the write rate (bytes/sec) from producers. Protects brokers from write bursts and keeps disk and network consumption in check.
• Consumer quota: limits the read rate (bytes/sec) from consumers. Prevents one consumer from pulling data too fast and throttling other clients.

#Request and Controller Mutation Quotas

• Request quota: limits the percentage of broker processing time a client-id uses — stopping clients that flood the broker with metadata requests or small API calls.
• Controller mutation quota: limits operations that mutate cluster metadata (creating/deleting topics, reassignment) so one user can't flood the controller.

#Configuring Quotas

#User and Client-id Levels

Quotas are applied per user (the authentication principal), per client-id, or as a user+client-id combination. The most specific combination wins:

bin/kafka-configs.sh --bootstrap-server localhost:9093 \
  --alter --add-config 'producer_byte_rate=10485760,consumer_byte_rate=10485760' \
  --entity-type users --entity-name app-producer

kafka-configs.sh --alter --entity-type users sets a 10 MB/s quota for the app-producer principal's writes and reads. Quotas are configured per user or per client-id, and used as the default if a principal has no rule of its own.

#User + Client-id and Defaults

bin/kafka-configs.sh --bootstrap-server localhost:9093 \
  --alter --add-config 'producer_byte_rate=5242880' \
  --entity-type users --entity-name app-producer \
  --entity-type clients --entity-name analytics-batch
 
bin/kafka-configs.sh --bootstrap-server localhost:9093 \
  --alter --add-config 'producer_byte_rate=20971520,consumer_byte_rate=20971520' \
  --entity-type clients --entity-name default

--entity-type clients --entity-name default sets the default for all client-ids without explicit rules. The most specific user+client-id quota applies to that particular combination, overriding the per-user or per-client-id quota.

#Dynamic Quota Updates

Quotas can be changed at any time without restart — brokers apply the changes dynamically. This is useful for responding to load spikes: temporarily raise one team's quota, then restore it when done. The same kafka-configs.sh --alter is used to adjust values instantly.

To remove a specific quota and return to the default, use --delete-config:

bin/kafka-configs.sh --bootstrap-server localhost:9093 \
  --alter --delete-config 'producer_byte_rate' \
  --entity-type users --entity-name app-producer

kafka-configs.sh --alter --delete-config removes the specific rule so the principal returns to the default. Always verify the result with --describe to make sure the intended values are actually in effect.

#Throttling Mechanics

#How Brokers Enforce Quotas

Brokers track usage per connection and compute the maximum allowed value based on recent window metrics. When usage exceeds the quota, brokers add a delay (throttle time) to responses so clients can't send faster than allowed. librdkafka and modern client libraries understand this throttling and adjust their speed.

#Monitoring Metrics

It's important to monitor the impact of quotas:

• throttle-time: how much additional time is imposed per request due to quotas. A high value signals a client nearing its limit.
• produce-throttle-time / fetch-throttle-time: the throttle breakdown for writes and reads.
• Also monitor request-rate and bytes-out/in per client to identify clients approaching their limits.

#Dynamic Behavior

Throttling is responsive: when a client slows down and usage drops below the quota, throttling stops. So quotas aren't an instant hard limit — they're a smooth rate governor, not a connection cutter. A healthy application simply raises its internal limit; a misbehaving application can still "flood" briefly before throttling kicks in.

#Quota Use Cases

#Multi-Tenancy and Noisy Neighbors

Quotas are the primary tool for resource isolation in multi-tenant clusters. Each team gets a write and read limit; one team flooding can't degrade another team's SLA. Combine with ACLs (episode 17) for full control: ACLs determine what's allowed, quotas determine how much.

#Cost Control and SLAs

• Cost control: quotas prevent runaway clients that waste bandwidth and storage — important when costs are billed per usage or in cloud deployments.
• SLA enforcement: quotas guarantee broker capacity isn't consumed by one application, so p95 latency stays healthy for all clients.

#Closing

In this episode 19 you've understood the quota types, configuration at the user and client-id levels, delay-based throttling mechanics, and using quotas for multi-tenancy, cost control, and SLAs.

The key takeaways:

• Producer and consumer quotas limit throughput in bytes/sec.
• Request and controller mutation quotas limit broker processing load.
• Quotas can be set per user, per client-id, or combined.
• Default quotas protect the whole cluster from clients without rules.
• Brokers enforce quotas by adding throttle time, not cutting connections.
• Combine quotas with ACLs for full multi-tenant isolation.

In the next episode 20 we'll scale the cluster: scaling Kafka clusters. You'll learn adding brokers, partition reassignment, capacity planning, and an introduction to Cruise Control for automatic rebalancing and self-healing.