Learn Karpenter - Troubleshooting

#Introduction

In episode 17 you learned about Karpenter's advanced features: static capacity, bare metal, and community provider support on other platforms. The more complex a system, the more important it is to be able to find out why it is not behaving as expected. Karpenter is a fast-moving system — dozens of decisions are made every second — so when something goes wrong, the symptom appears far from the root cause.

This episode equips you with a systematic diagnostic flow. We start with controller logs, then NodeClaim and NodePool conditions, webhook validation messages, and Kubernetes events. Finally, we break down common problems and their solutions one by one.

#Diagnostic Flow: Start from the Symptom

Don't jump straight into the logs. Start with the question: what is actually happening? Pods pending, nodes not created, or nodes never dismantled? These three symptoms point to different layers — scheduler, cloud provider, or disruption.

The most effective order:

Look at the pod and NodeClaim status.
Read the events connected to the pod and NodeClaim.
Trace the Karpenter controller logs.
Check the AWS resources on the other side, such as IAM, quotas, and subnets.

Tip

Note the time of the incident. Karpenter processes many things in parallel, so logs and events can be verbose. Searching for entries around the time of the incident is much faster than reading everything from the beginning.

#Karpenter Controller Logs

The Karpenter controller runs in the karpenter namespace as a deployment named karpenter. The logs there are the primary source of truth for what Karpenter is thinking.

kubectl logs -n karpenter deploy/karpenter --tail=200

The log level can be raised to debug via the karpenter-global-settings ConfigMap in the same namespace, by changing the logLevel key. Debug logs reveal scheduling simulation details, instance type choices, and the reason a node was rejected — very useful when the symptom is vague.

#NodeClaim and NodePool Conditions

The NodeClaim is the most important object to read. Every claim has conditions in its status that show the phase of its life cycle.

Inspecting NodeClaim conditions

kubectl get nodeclaims
kubectl get nodeclaim <nama-nodeclaim> -o yaml

Pay attention to the status.conditions section. Conditions such as Launching, Registered, and Initialized indicate normal progress. If Registered is false, the node cannot join the cluster — usually an IAM or bootstrap problem. NodePools have similar conditions; an unhealthy NodePool means every workload pointing to it never gets scheduled.

Important

Also check the status section for error messages from the cloud provider, such as a failed instance launch. These messages often state the cause directly, like an exceeded vCPU quota or an unavailable instance type.

#Webhook Validation

Karpenter installs an admission webhook to validate NodePools and EC2NodeClasses before they are accepted. Configuration errors usually surface here — before the resource is created, not when it runs.

Seeing webhook validation

kubectl apply -f nodepool.yaml
kubectl get validatingwebhookconfiguration -l app.kubernetes.io/name=karpenter

If the YAML is wrong, kubectl apply rejects it with a fairly explicit message, such as a duplicate requirement or an unknown field. Make sure the apiVersion matches the installed Karpenter version; mixing CRD versions is the most common cause of the no matches for kind error.

#Reading Kubernetes Events

Events record what happened to a specific object. For a pending pod, describe the pod; for a node that never gets created, filter events by the NodeClaim kind.

kubectl describe pod <nama-pod>
kubectl get events --field-selector involvedObject.name=<nama-pod>

Events on a NodeClaim show the claim's journey: launched, registered, initialized, and finally deleted. If the claim stops in the middle, the surrounding events tell you why.

#Common Problems and Their Solutions

Symptom	Most likely cause	First step
Node never created	IAM role or instance profile mismatch	Check controller logs for `AccessDenied`
Node not created, quota error	EC2 vCPU quota exceeded	Check service quotas and reduce instance size
Spot always fails	Empty Spot capacity in the zone	Add instance types and zones
Node created repeatedly	Excessive drift due to frequently changing configuration	Limit changes and use a `Drifted` budget
Node not dismantled	Disruption blocked by a PDB	Check `kubectl get pdb -A`
Pod pending forever	Scheduling constraints don't match	Describe the pod and read events

#Node Fails to Be Created: IAM and Instance Limits

When Karpenter cannot launch an instance, the controller logs contain the error from the AWS API, such as AccessDenied on the instance profile, or a quota message for the vCPU limit. For IAM: make sure the instance profile in the EC2NodeClass is correct and the role has the policies required to launch instances.

For instance limits, check the EC2 vCPU quota in the region you use. Karpenter has no try-until-succeed mechanism — it waits for the error and then emits an event. Better to raise the quota in the AWS Support Center, or widen the instance type choices in the NodePool so an available option exists.

#Empty Spot Capacity

Whenever Spot capacity thins out, errors like InsufficientInstanceCapacity or NoSpotCapacity appear on the NodeClaim. Karpenter tries other instance types within one batch, but if the NodePool restricts too narrowly — for example one instance family in one zone — the chance of success is very small.

The quickest fix: add more instance types through requirements and enable several zones. The more diverse the options, the more resilient you are to Spot capacity fluctuations.

#Excessive Drift

Drift happens when a node no longer matches the specification — for example, when the AMI or security group changes in the EC2NodeClass. If changes happen continuously, Karpenter keeps replacing nodes and the cluster feels unstable.

How to control it: reduce the frequency of changes to the EC2NodeClass, and use a disruption budget to limit how many drifted nodes can be processed at once.

Budget for limiting drift

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    budgets:
      - nodes: "20%"
        reasons:
          - Drifted

#Disruption Fails Due to PDB

Consolidation and drift respect Pod Disruption Budgets. If a PDB refuses eviction — because minAvailable is too high or maxUnavailable is zero — the NodeClaim is not dismantled and the DisruptionBlockedByPDB event appears.

Check with kubectl get pdb -A and see which application is holding things up. Possible fixes: lower minAvailable, tolerate pods that really can be disrupted, or separate the workload into a NodePool that is not consolidated.

#Pending Pods That Never Get Scheduled

A pending pod without a NodeClaim means Karpenter believes no NodePool matches. Describe the pod and read the didn't match any node or Insufficient event — the failed constraint is listed there.

Also check whether there is a taint without a toleration, an impossible topology spread, or conflicting NodePool requirements. Karpenter does not create a NodeClaim if no instance type passes all pod constraints.

#Closing

Troubleshooting Karpenter is not magic — it is the discipline of reading the signs in the right order.

Key takeaways:

Start from the symptom: determine the problem layer — scheduling, cloud provider, or disruption — before touching the logs.
The controller log is the source of truth: kubectl logs -n karpenter reveals Karpenter's internal decisions and errors.
NodeClaim conditions tell the life cycle: Registered and Initialized are healthy markers; errors in the status point directly to IAM or quotas.
PDBs can hold up consolidation: the DisruptionBlockedByPDB event and kubectl get pdb -A make sure disruption budgets are honored.

A running Karpenter does not mean a fast one. In episode 19 we cover Performance and Tuning — provisioning latency factors, tuning spec.template and consolidation windows, binpacking efficiency, and how to avoid fragmentation with correct resource requests. See you there!