Learn Karpenter - Scheduling Constraints

#Introduction

In episode 7 you learned how Karpenter trims nodes via disruption budgets and consolidation. Episode 8 shifts to the other side of Kubernetes scheduling: the constraints that come from the pod itself. You might think the NodePool determines everything, but in reality most decisions are born from how a pod is written — nodeSelector, toleration, topologySpreadConstraints, and pod anti-affinity.

After this episode, you'll understand how these constraints affect NodePool selection, and how Karpenter treats DaemonSets and critical pods while a node is being drained or terminated.

#Node & Pod Constraints

#nodeSelector: Select Nodes with Specific Labels

nodeSelector is the simplest constraint: a pod may only be scheduled to nodes with a specific label. Karpenter uses it in a unique way — the labels attached to a NodePool template can be targeted by a pod's nodeSelector.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-inference
spec:
  nodeSelector:
    node-pool: gpu
  containers:
    - name: inference
      image: tensorflow/serving
      resources:
        requests:
          cpu: 500m
          memory: 1Gi

If a NodePool attaches the node-pool: gpu label to its template, only that NodePool can schedule this pod. This constraint narrows the NodePool candidates directly.

#Tolerations: The Valve for Tainted Nodes

Taints and tolerations work as a gate. Nodes with a taint reject pods without a matching toleration. Karpenter uses this to separate workloads: a NodePool for special workloads can be tainted, so only pods with the matching toleration get in.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-pool
spec:
  template:
    spec:
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.xlarge", "g5.2xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

apiVersion: v1
kind: Pod
metadata:
  name: gpu-worker
spec:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: worker
      image: nvidia/cuda:12.0-base

#topologySpreadConstraints: Spreading Across Availability Zones

When pods are scheduled, Kubernetes can balance their spread via topologySpreadConstraints. This matters for availability: if all replicas sit in one availability zone and that zone has issues, the application goes down with it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: web
          image: nginx

This constraint affects Karpenter in two ways. First, when choosing instance type and subnetwork, Karpenter considers the availability zones available. Second, during consolidation, Karpenter makes sure moving pods doesn't violate this spread constraint.

#Pod Anti-Affinity: Not on the Same Node

Pod anti-affinity forbids several pods from running on the same or nearby nodes. This is often used for workloads that interfere with each other, or to separate two components so a failure doesn't hit both at once. Rules like topologyKey: kubernetes.io/hostname force replicas to spread across different hostnames, so one failing node doesn't take down all replicas at once.

#Impact on NodePool Selection

All of the constraints above narrow the list of eligible NodePools. Karpenter evaluates pods with the same rules as the Kubernetes scheduler: a pod is only considered schedulable if there's a NodePool that produces a node satisfying all constraints. If no NodePool matches, the pod stays Pending and no node is created.

#DaemonSets & System Workloads

#How Karpenter Handles DaemonSets

DaemonSets are designed to run on every node. When Karpenter launches a new node, DaemonSets like kube-proxy and CNI are automatically scheduled onto that node by the DaemonSet controller — without Karpenter's involvement. Likewise when a node is terminated, DaemonSet pods don't need to be moved because they'll be born again on the replacement node.

#Kube-Proxy and CNI

Kube-proxy and CNI are DaemonSets that must exist before ordinary pods can run. Karpenter accounts for their resource needs when sizing instances: some node capacity is always reserved for these system components. This is why pods are never counted as using the full node capacity.

#Critical Pods and Graceful Drain

Pods with priorityClassName: system-cluster-critical or system-node-critical receive special treatment. When a node is about to be terminated, Karpenter drains gracefully: pods are given time to finish, then terminated one by one. For pods that can't be moved, Karpenter waits until conditions are safe before the node is actually shut down.

kubectl get pod -A --field-selector=spec.priorityClassName!= -o wide
kubectl drain node-xyz --ignore-daemonsets --delete-emptydir-data

#Common Mistakes

#Toleration Without a Taint

Giving a pod a toleration when there's no tainted NodePool makes that toleration useless. A toleration only matters when there's actually a node carrying that taint.

#Spread That's Too Strict

topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule and maxSkew: 0 can make pods never schedule if there aren't enough zones. Use ScheduleAnyway when possible.

#Closing

Scheduling constraints define the boundaries of the options available to Karpenter. The clearer you write nodeSelector, tolerations, spread, and anti-affinity, the more accurate the provisioning and consolidation decisions Karpenter makes.

Key takeaways:

• nodeSelector and tolerations select NodePools — labels and taints on the template determine which pods are allowed in.
• Spread and anti-affinity shape topology — both affect AZ choice and node density.
• DaemonSets are counted separately — system pods don't affect consolidation decisions.
• Drain runs automatically — Karpenter handles critical pods with graceful shutdown at termination.

In episode 9, you'll learn how Karpenter can be bent to save costs — from Spot priority, choosing efficient instance families, to automatic handling of Spot instance interruptions via the queue processor. See you there!