Learn Karpenter - Disruption & Consolidation

#Introduction

In episode 6 you saw how Karpenter provisions: grouping unscheduled pods, binpacking, then launching the smallest sufficient instance type. Episode 7 reverses the direction: instead of adding nodes, it's about removing them. Karpenter isn't just great at adding capacity, but also at trimming inefficient capacity through two main mechanisms: disruption budgets and consolidation.

After this episode, you'll understand how to limit how many nodes may be disrupted at the same time, the difference between consolidationPolicy: WhenEmpty and WhenUnderutilized, and how both features work together to keep cluster costs low.

#Why Disruption Is Needed

Cluster capacity needs are never constant. Apps that are busy in the morning can be quiet at night. New deployments can reduce replica counts, and finished batch jobs leave nodes idle. Without a trimming mechanism, unused nodes still get billed by AWS. This is why Karpenter built a dedicated controller called the disruption controller that continuously evaluates whether any node deserves to be disrupted.

Disruption in the Karpenter context means the process of terminating a node and moving its pods to another node. This process is done gradually so it doesn't affect application availability: pods are drained gracefully, then the node is terminated.

#Disruption Budgets

#Basic Concept

Trimming nodes is necessary, but trimming too many at once is dangerous. If twelve nodes are drained and all full of pods, then all are disrupted simultaneously, the application could lose capacity drastically. Disruption budget is Karpenter's mechanism to limit how many nodes may be in the process of disruption at the same time, whether per NodePool or per cluster.

In Karpenter v1, this configuration is written in the spec.disruption.budgets section of the NodePool.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: 10%
      - nodes: "3"
        schedule: "0 9 * * *"

#Per NodePool and Per Cluster Budgets

Budgets written in a NodePool apply to the nodes created by that NodePool. To limit disruption across the whole cluster, Karpenter provides a global default budget: if a NodePool doesn't define a budget at all, Karpenter applies a default limit of 10 percent of the nodes managed by that NodePool.

The schedule field lets you create different budgets at specific times, for example loosening the limit outside working hours or tightening it during busy hours. This pattern is useful to make sure disruption doesn't coincide with traffic peaks.

#Protecting Nodes from Disruption

Sometimes a pod needs total protection from disruption, for example a pod processing a non-repeatable batch job. Karpenter honors the karpenter.sh/do-not-disrupt annotation on pods.

kubectl get pod critical-job-xyz -n apps -o jsonpath='{.metadata.annotations.karpenter\.sh/do-not-disrupt}'

#Consolidation

#Two Policies: WhenEmpty and WhenUnderutilized

If disruption budgets answer the question how many, consolidation answers when and how. Karpenter provides two consolidationPolicy options:

• WhenEmpty — removes nodes that no longer run non-DaemonSet pods once consolidateAfter is reached. This is the most conservative policy: a node is removed only when it's completely empty.
• WhenUnderutilized — in addition to removing empty nodes, also merges pods from several rarely-used nodes onto one more efficient node, or replaces an expensive node with a cheaper one.

spec:
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 10m

spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 5m

#How Consolidation Works

Consolidation runs in a repeating cycle. For every node managed by Karpenter, the controller calculates two things: whether that node is worth deleting, and whether there's a cheaper instance combination that still fits the pods on that node. If the answer is yes, Karpenter creates the new node first, waits for the new pods to be ready, then drains the old node. With this ordering, application downtime is practically imperceptible.

kubectl get nodes -l karpenter.sh/managed=true
kubectl get nodeclaims -o wide

Nodes being moved will appear to enter the drain process before being terminated.

#Disruption Priority Scheme

To keep decision-making orderly, Karpenter follows this priority order when deciding which node gets disrupted first:

The higher the priority, the sooner it's handled. This ensures the limited budget is used for the most important disruptions.

#Common Mistakes

#Setting consolidateAfter Too Aggressively

A value like 1s makes Karpenter continuously evaluate and move pods, producing unnecessary churn and adding load to the API server. Give a reasonable pause, for example a few minutes.

#Forgetting the DaemonSet Effect

DaemonSets like kube-proxy and CNI run on every node. When calculating whether a node is worth removing, Karpenter ignores DaemonSet pods. A node with a single DaemonSet pod is still considered empty by the WhenEmpty policy.

#Closing

Disruption budgets and consolidation are two sides of one goal: capacity that always fits the need. Budgets keep the trimming process safe, while consolidation ensures every remaining node is used as efficiently as possible.

Key takeaways:

• Budgets are a safety brake — spec.disruption.budgets limits how many nodes may be disrupted at the same time, per NodePool or globally.
• WhenEmpty for stability — nodes are only removed when completely empty.
• WhenUnderutilized for savings — pods are merged and instances replaced with cheaper ones.
• do-not-disrupt is a veto — one annotation is enough to protect a pod or node from disruption.

In episode 8, you'll learn how pod-side constraints like nodeSelector, toleration, and anti-affinity affect NodePool selection — and how Karpenter treats DaemonSets and critical pods during drain. See you there!