Learn Karpenter - Performance & Tuning

#Introduction

In episode 18 you learned troubleshooting: reading controller logs, NodeClaim and NodePool conditions, and handling common problems up to pods that never get scheduled. Knowing how to fix what's broken is half the journey. The other half is making the system run as well as possible from the start.

This episode covers two dimensions of Karpenter performance. First, provisioning latency: how quickly a new pod can run after it is marked pending. Second, binpacking efficiency: how densely pods sit on nodes so every instance is optimally used. Both can be tuned via spec.template and disruption policies.

#Measuring Provisioning Latency

Before tuning anything, measure first. Karpenter exposes Prometheus metrics on the /metrics endpoint on port 8080. The three most useful metrics:

karpenter_pods_startup_duration_seconds — end-to-end time from pod creation to running status.
karpenter_scheduler_scheduling_duration_seconds — the duration of scheduling simulation on the Karpenter side.
karpenter_cloudprovider_duration_seconds — latency of calls to the cloud provider API.

Example PromQL query for latency

sum(rate(karpenter_cloudprovider_duration_seconds_sum[5m]))
  / sum(rate(karpenter_cloudprovider_duration_seconds_count[5m]))

By comparing the three, you can see where time is being spent. If the scheduling simulation is slow, the problem is in the constraints; if the cloud provider is slow, the problem is API rate limiting or too narrow an instance selection.

#Factors That Affect Speed

Two dominant factors determine provisioning latency:

API rate. Karpenter makes EC2 calls to launch instances and performs dry-runs to test instance types. If your cluster is very active, these calls can be throttled and every attempt becomes slow. The symptom shows up as a bloated karpenter_cloudprovider_duration_seconds and rising karpenter_cloudprovider_errors_total.

Instance selection. When scheduling, Karpenter simulates pods against the list of instance types the NodePool allows. The tighter the constraints, the fewer valid options, and the more work that must be done before finding a match.

Tip

Check the karpenter_scheduler_scheduling_duration_seconds metric in Grafana. A spike in duration often means stacked constraints — affinity, topology spread, or impossible resource requests — that make the simulation explore many combinations before giving up.

#Tuning spec.template for Speed

The NodePool spec.template.spec.requirements defines the instance search space. The wider it is, the higher the chance of getting an instance and the more resilient you are to capacity fluctuations, but the heavier the scheduling simulation. Conversely, restricting it too narrowly makes the search fast but prone to failure.

A balanced approach:

Allow several instance families with sensible sizes, not just one.
Set karpenter.sh/capacity-type and zones as needed so Karpenter doesn't waste time exploring unused options.
Use a karpenter.sh/instance-type cap to prevent selecting giant instances for small workloads.

Balanced requirements

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["ap-southeast-1a", "ap-southeast-1b"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
            - "m6i.large"
            - "m6i.xlarge"
            - "m6i.2xlarge"

#Consolidation Windows

Consolidation turns rarely used nodes into savings, but too-aggressive consolidation creates and dismantles nodes repeatedly. This is where disruption.consolidationPolicy comes in.

The WhenUnderutilized pattern moves pods and dismantles nodes as quickly as possible when utilization is low — fast but reactive. The WhenEmpty pattern waits until a node is truly empty for ttlSecondsAfterEmpty before dismantling it, which is calmer for bursty workloads.

Waiting for an empty node before dismantling

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    consolidationPolicy: WhenEmpty
    ttlSecondsAfterEmpty: 600
    expireAfter: 720h

Important

expireAfter forces nodes to be replaced after a certain period — useful for absorbing AMI updates and keeping instance age bounded. But if set too short, Karpenter keeps replacing nodes and the launch cost exceeds the savings. Adjust it to the maximum acceptable workload age.

#Binpacking Efficiency

Binpacking is Karpenter's ability to pack pods into nodes as densely as possible. The first key: correct resource requests. Karpenter schedules based on requests, not limits. If a pod requests 4 vCPU while only using 1 vCPU on average, nodes grow larger and costs spike for no reason.

Correct practices:

Use VPA or observability to measure actual usage before setting requests.
Avoid setting requests that mimic limits.
Review requests whenever a new feature ships.

The second key: avoid fragmentation. Fragmentation happens when the remaining capacity on a node is too small for any pod, so the node looks partially used but can't be filled. A large instance with few pods is the classic example.

#Avoiding Fragmentation

Several ways to reduce fragmentation:

Make sure request sizes don't leave awkward remainders — a combination of 0.4, 0.6, and 1.1 vCPU on one node leaves a remainder that is hard to fill.
Limit the maximum instance size so nodes don't grow beyond aggregate needs.
Let consolidation do its job: WhenUnderutilized squeezes pods together and dismantles under-packed nodes.

Also pay attention to well-known labels such as karpenter.sh/instance-type and topology.kubernetes.io/zone that Karpenter uses to calculate spread. A topology spread forced across large pods hinders binpacking because Karpenter must spread pods across many nodes.

#Tuning disruption.budgets

Budgets limit how many nodes may undergo disruption simultaneously. A budget that is too loose makes changes fast but risks disrupting services; a budget that is too tight makes the cluster slow to respond to configuration changes.

Karpenter v1 supports budgets by reason and by schedule. For example, limit drift during working hours so the cluster stays calm during full traffic, then give more room outside working hours.

Budget with schedule and reason

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    budgets:
      - nodes: "10%"
        reasons:
          - Drifted
        schedule: "0 8 * * mon-fri"
      - nodes: "40%"
        reasons:
          - Drifted
        schedule: "0 20 * * mon-fri"
      - nodes: "10%"

Warning

Budgets with a schedule use the timezone configured on Karpenter, not your cluster's local time automatically. Write schedules assuming UTC and verify against the actual calendar — this is the mistake that most often makes a budget inactive when you expect it to be active.

#Closing

Karpenter performance is not about pressing a single button — it is a balance between provisioning speed and node density.

Key takeaways:

Measure before tuning: the karpenter_cloudprovider_duration_seconds and karpenter_scheduler_scheduling_duration_seconds metrics show where time is spent.
Balanced requirements win: too narrow causes failures, too wide causes slowness; start from a reasonable set of instance families.
Requests determine binpacking: Karpenter schedules based on requests, so oversized requests bloat nodes and costs.
Consolidation needs the right windows: WhenEmpty with ttlSecondsAfterEmpty calms bursty clusters, expireAfter absorbs AMI updates.
Disruption budgets are a safety valve: limit drift and consolidation with reasoned, scheduled budgets.

With this tuning, your Karpenter runs fast and efficient. But the ecosystem doesn't stop at the version you're currently using. In episode 20 we cover the latest stable features in Karpenter v1.14 — the evolution from v1.0, the v1 API, Static Capacity, NodeOverlay, drift detection for the CA bundle, and scheduling and disruption budget improvements. See you there!