Learn KEDA - Troubleshooting & Monitoring

#Introduction

In episode 17 we explored many scalers and integrations. The more you can automate, the more that can fail without you noticing. Messages pile up in the queue, yet replicas don't rise — incidents start with a silent autoscaler. This episode covers Troubleshooting & Monitoring: the correct diagnostic sequence, metrics you can rely on, and solutions for the most common issues in KEDA v2.20.2. The goal is simple: go from "why isn't it scaling?" to "here's the evidence, here's the cause, here's the fix" in minutes.

#Diagnosis: A Structured Flow

Don't guess. Follow the sequence from the highest-level status to the deepest detail.

#1. ScaledObject Status

Viewing ScaledObject status

kubectl get scaledobject -n orders
kubectl describe scaledobject order-worker -n orders

kubectl describe scaledobject order-worker -n orders shows conditions and events. Pay attention to the Ready, Active, and Fallback columns from kubectl get scaledobject. Ready = valid configuration and HPA created; Active = the scaler is triggering; Fallback = the scaler failed and fallback is active.

#2. Deep Dive into the YAML

When the description isn't enough:

Inspecting the full YAML

kubectl get scaledobject order-worker -n orders -o yaml
kubectl get scaledobject order-worker -n orders -o jsonpath="{.status.conditions[?(@.type=='Ready')].status}"

The status section holds the answers: conditions, reason, and the scaler's last error message. Many cases are immediately visible here — for example AuthSecret does not exist.

#3. HPA and Actual Replicas

HPA managed by KEDA

kubectl get hpa -n orders
kubectl get hpa order-worker -n orders -o yaml

KEDA names the HPA after the ScaledObject. If the HPA doesn't exist, KEDA hasn't managed to create it — usually an RBAC or validation issue. If the HPA exists but replicas don't rise, check kubectl get pods -n orders for pod conditions (Pending, CrashLoopBackOff).

#4. Operator Logs and Metrics

kubectl logs -n keda deploy/keda-operator --tail=100
kubectl logs -n keda deploy/keda-operator -f | grep -iE "error|failed"
kubectl port-forward -n keda deploy/keda-operator 8080:8080

The operator writes all scaler activity to stdout. Prometheus keda_scaler_* metrics are exposed on port 8080 — a lesson we started in episode 15. Grafana can visualize them:

groups:
  - name: keda.rules
    rules:
      - alert: ScaledObjectNotReady
        expr: keda_scaledobject_ready == 0
        for: 5m
        labels:
          severity: warning

#Common Problems & Their Solutions

#Scaler Not Active Due to Activation Threshold

Symptom: the queue is full, Active is false, replicas stay at zero. Most common cause: the metric value is still below activationThreshold. Check it with:

curl -s localhost:8080/metrics | grep keda_scaler_metrics_value
kubectl get scaledobject order-worker -n orders -o jsonpath="{.status.scaleMetricName}"

If keda_scaler_metrics_value is below the threshold, it's not a bug — it's by design. Raise the threshold or reduce sensitivity if that isn't what you want.

#HPA Not Being Updated

Symptom: the ScaledObject is Ready, but replicas don't move even though metrics are high. Check two things: whether the HPA reads external metrics, and whether the scaleTargetRef Deployment is correct.

kubectl describe hpa order-worker -n orders
kubectl get apiservice v1beta1.external.metrics.k8s.io

If external.metrics.k8s.io isn't registered or KEDA's metrics server isn't responding, the HPA has no metric source. Restart the metrics server: kubectl rollout restart deploy/keda-adapter -n keda.

#Authentication Failure: Secret Not Found

Symptom: the ScaledObject status shows an AuthSecret does not exist error or a 403 from the provider. Cause: the secret name in secretTargetRef doesn't match, or the secret is in another namespace while TriggerAuthentication is namespaced.

kubectl get secret -n orders
kubectl get triggerauthentication sqs-auth -n orders -o yaml
kubectl get secret sqs-creds -n orders -o jsonpath="{.data.AWS_ACCESS_KEY_ID}"

Check that the key referenced by secretTargetRef.key actually exists in the secret. For cross-namespace credentials, switch to ClusterTriggerAuthentication (episode 14).

#Webhook Error

Symptom: kubectl apply of the ScaledObject is rejected with Internal error occurred: failed calling webhook. Cause: the keda-admission pod is unhealthy, or the configuration is genuinely invalid according to the validating webhook.

kubectl get pods -n keda -l app=keda-admission
kubectl logs -n keda deploy/keda-admission --tail=50
kubectl get validatingwebhookconfigurations | grep keda

Read the full error message — the validating webhook tells you which field is violated. If the webhook itself is down, fix its pod first; without a webhook, KEDA resources can still be applied (Kubernetes fail-open behavior).

#Unknown Status

Symptom: the Ready/Active columns show Unknown. This usually means the operator can't execute the scaler — credential timeout, provider DNS failure, or an invalid query. Look at the condition message and operator logs:

kubectl get scaledobject order-worker -n orders -o jsonpath="{.status.conditions}"
kubectl logs -n keda deploy/keda-operator | grep -i "order-worker"

One classic cause: a Prometheus query containing pipe characters hasn't been escaped, so metric parsing fails. Make sure the query in metadata uses a form that's been sanitized for YAML.

#Common Diagnosis Mistakes

1. Only looking at status, not logs. Status is the symptom; operator logs are the root cause.
2. Forgetting to check TriggerAuthentication. Auth errors are often mistaken for scaler problems.
3. Ignoring keda_scaledobject_ready == 0. The autoscaler dies silently long before the queue piles up.
4. Testing the HPA without waiting for pollingInterval. Conclusions drawn too quickly lead you in the wrong direction.
5. Blaming KEDA while the Deployment is CrashLoopBackOff. Replicas may not rise because the pod itself is unhealthy, not the scaler.

#Conclusion

This episode gave you a diagnostic map: start with the ScaledObject status and full YAML, continue to the HPA and pods, then operator logs and keda_scaler_* metrics. We also closed out six common problems: activation threshold, stale HPA updates, auth failures, webhook errors, and Unknown status — each with a definitive check command.

Points you should take away:

• Diagnosis sequence: status → YAML → HPA → operator logs → metrics.
• keda_scaler_metrics_value and keda_scaledobject_ready are the two most important metrics.
• Auth errors are usually about secretTargetRef or the trigger authentication namespace.
• Unknown status = the operator can't execute the scaler, not a configuration issue.
• Always wait one polling cycle before drawing conclusions.

Now that you can solve problems, it's time to refine behavior. In the next episode, 19, we discuss Performance & Tuning: optimal pollingInterval vs cooldownPeriod, calculating replica saturation per message, dealing with provider API throttling, and measuring scale-up latency and the cost vs responsiveness trade-off. See you in episode 19!