Learn Karpenter - Drift Detection

#Introduction

In episode 10 you organized many NodePools within one cluster and kept consistency across clusters with GitOps. Episode 11 covers the problem that appears precisely after everything is running: drift. A healthy cluster doesn't stop at fast provisioning — it must also keep existing nodes aligned with the desired configuration.

After this episode, you'll understand what drift is, how Karpenter detects it, and what NodeOverlay is and when it should be used.

#The Drift Concept

#What Is Drift

Drift is a condition where a running node no longer matches the template that defines it. Karpenter detects drift by comparing the NodePool and NodeClass specifications against the node's actual state on AWS. When the two differ, the node is considered drifted.

The most common drift examples:

• AMI drift — the AMI selected via amiSelectorTerms is updated, for instance the latest Amazon Linux version, while the node still runs the old AMI.
• Security group drift — the security group list on the EC2NodeClass is changed, but the node still uses the old list.
• Instance type drift — the instance type requirement on the NodePool changes, so a node with the old type is considered out of compliance.
• User data and tag drift — changes to userData or the EC2NodeClass tags aren't reflected on the node.

kubectl get nodeclaims -l karpenter.sh/drifted=true

#Why Drift Is Dangerous

Drifted nodes are a silent source of problems. Old images can carry kernels or runtimes with existing security vulnerabilities. Diverging configuration makes cluster behavior unpredictable. Without detection, drift piles up until it's only noticed once it's already an incident.

#The Drift Remediation Process

Karpenter treats drift like any other form of disruption. Drifted nodes enter the disruption queue, are drained gracefully, then replaced with new nodes following the latest template. This process is the same disruption cycle covered in episode 7.

kubectl get nodeclaim -o yaml | grep -A 8 'karpenter.sh/drifted'

#Protecting Nodes from Drift

Sometimes a node is intentionally kept in its old state, for example because it's running a process that must not be interrupted. The karpenter.sh/do-not-drift annotation can be attached to stop that node from being replaced by drift.

apiVersion: v1
kind: Pod
metadata:
  name: long-running-job
  annotations:
    karpenter.sh/do-not-drift: "true"
spec:
  containers:
    - name: job
      image: busybox

#NodeOverlay: Updates Without Full Replacement

#The Problem Behind Full Replacement

Replacing a node for every template change is expensive and risky. Each replacement means creating a new instance, pulling images, and running a drain — no small workload and cost, especially when the change is trivial like updating userData. For such cases, Karpenter provides an experimental feature called NodeOverlay.

#What Is NodeOverlay

NodeOverlay is Karpenter's ability to update parts of a node directly in place, without launching a replacement instance. This feature is enabled via the NodeOverlay feature gate and is currently alpha. With NodeOverlay, changes to userData or AMI configuration that don't need a new kernel can be applied without node replacement.

kubectl set env deployment/karpenter -n karpenter \
  --list | grep FEATURE_GATES

#When NodeOverlay Is Better

Consider NodeOverlay when the change being made is configurational, not a change to the instance structure. For example updating userData to install additional dependencies, or adjusting configuration read at boot time. Conversely, changes that alter the kernel or require a new disk are still better handled with a full replacement.

#Common Mistakes

#Ignoring Drift Annotations

Drift left to pile up makes the cluster run on a foundation that doesn't match the configuration. Monitor the karpenter.sh/drifted annotation regularly and make sure remediation runs.

#Relying Too Much on NodeOverlay

NodeOverlay is tempting because it looks cheap, but applying an overlay to a configuration that should be fully replaced actually adds risk. Use this feature selectively.

#Closing

Drift detection is Karpenter's mechanism to make sure the cluster doesn't run far from what's desired. With automatic detection, protection annotations, and NodeOverlay for the right cases, you can keep nodes aligned with the template without sacrificing availability.

Key takeaways:

• Drift is the gap between template and reality — AMI, security group, and instance type are the main triggers.
• Drift is handled like disruption — nodes are drained and replaced following the latest template.
• do-not-drift is the exception — use it for processes that can't be interrupted, with full awareness of the consequences.
• NodeOverlay saves replacements — an alpha feature for in-place updates, not a substitute for full replacement.

In episode 12, you'll learn to monitor Karpenter with Prometheus metrics like karpenter_nodes_* and karpenter_pods_*, build Grafana dashboards, and set up alerts for pods stuck Pending too long. See you there!