Learn KEDA - Message Queue Scalers

#Introduction

In episode 7 we played with basic scalers that rely on metrics. Now we enter the area where KEDA is used most in the real world: message queues and streams. A queue is the most honest representation of workload — every item is a job that hasn't been done yet, and its count rises before CPU ever gets busy. That's the most proactive scale-up signal, and the basis of every pattern in this episode.

#Managed Queues in the Cloud

#AWS SQS: Queue Length with IRSA

To access SQS without an access key inside the pod, KEDA uses IAM Roles for Service Accounts (IRSA): TriggerAuthentication points to provider: aws, and the role-arn annotation is attached to the KEDA operator's ServiceAccount. Azure uses azure-workload, GCP uses gcp — the pattern is identical.

AWS SQS ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: order-worker
  minReplicaCount: 0
  maxReplicaCount: 30
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: keda-trigger-auth-aws-credentials
      metadata:
        awsRegion: ap-southeast-3
        queueURL: https://sqs.ap-southeast-3.amazonaws.com/123456789012/orders
        queueLength: "10"
        activationQueueLength: "5"

queueLength: "10" means every 10 messages produce 1 pod; activationQueueLength: "5" prevents premature scale-up. For the following scalers only the triggers fragment is shown — the ScaledObject wrapper is identical to the example above. Check that everything is registered via kubectl get scaledobject -n production.

#Azure Service Bus

Service Bus is scaled via Azure AD Workload Identity — no connection string. The target is messageCount per replica.

#GCP Pub/Sub

On GCP, the trigger is fired by the number of unacknowledged (undelivered) messages in the subscription.

Tip

The best practice is the same across all three cloud providers: avoid hardcoding credentials in the ScaledObject, use podIdentity so key rotation is handled automatically by the cloud, and set activation parameters so small batches don't trigger pointless scaling.

#Self-Hosted Brokers

#RabbitMQ: Queue Depth

RabbitMQ is scaled based on queueLength — target messages per replica. The host is taken from a secret via secretTargetRef, the same pattern used by Redis Streams.

RabbitMQ trigger

    - type: rabbitmq
      authenticationRef:
        name: rabbitmq-auth
      metadata:
        protocol: amqp
        queueName: jobs
        queueLength: "5"
        activationQueueLength: "1"

#Kafka: Consumer Lag and Group

On Kafka, KEDA measures consumer lag: the difference between the last offset in the topic and the offset already processed by the consumer group. High lag means many messages waiting — a perfect scale-up signal.

Kafka consumer lag trigger

    - type: kafka
      metadata:
        topic: order-events
        bootstrapServers: kafka-broker.kafka.svc:9092
        consumerGroup: order-processor
        lagThreshold: "100"
        activationLagThreshold: "10"

Note

One Kafka ScaledObject manages one consumerGroup. If a single group consumes many topics, list them all in topic separated by commas so the lag is calculated in aggregate.

#Redis Streams

Redis Streams is scaled based on the length of pending entries in a specific consumer group — counting only messages that haven't been acknowledged, which is more accurate than stream length.

Redis Streams trigger

    - type: redis-streams
      authenticationRef:
        name: redis-auth
      metadata:
        stream: jobs
        consumerGroup: workers
        pendingEntriesCount: "5"
        lag: "5"

#NATS JetStream

NATS JetStream is scaled based on lagThreshold — messages not yet processed on a subject within a stream.

Warning

A self-hosted broker is a single point of failure for autoscaling. Always add a fallback in the ScaledObject and monitor keda_scaler_errors_total. If the broker goes down, KEDA must not scale down the consumers that are exactly what's needed to drain the messages.

#Conclusion

Cloud: AWS SQS (queue length + IRSA), Azure Service Bus (message count), GCP Pub/Sub (subscription size).
Self-hosted: RabbitMQ (queue depth), Kafka (consumer lag per group), Redis Streams (pending entries), NATS JetStream (lag per subject).
The best signal is backlog, not CPU — always combine a target with activation parameters.

Queues aren't the only source of signals. In episode 9 we cover database scalers (PostgreSQL, MySQL, Redis list, MongoDB) and an introduction to the KEDA HTTP Add-on with HTTPScaledObject for scaling HTTP workloads to zero. See you there!