Learn Karpenter - Multi-NodePool & Multi-Cluster

#Introduction

In episode 9 you optimized costs with Spot priority, instance filters, and automatic interruption handling. Episode 10 takes you to a larger scale: how to organize many NodePools within one cluster, and how to treat Karpenter when the clusters being managed number more than one.

After this episode, you'll understand how to separate general, GPU, spot-only, and zone-specific workloads using weight and taints, plus per-cluster policy best practices for multi-cluster deployments.

#Multiple NodePools

#Why More Than One NodePool

A single NodePool serving all workloads is simple, but quickly becomes rigid. Database workloads need on-demand and large volumes, while batch workers prefer cheap spot. ML workloads need GPU instances, and latency-sensitive services need specific zones. Combining all of them into one NodePool forces every group to compromise.

A common separation:

• general-pool — default workloads, spot with on-demand fallback
• gpu-pool — GPU instances for training and inference, tainted
• spot-only-pool — batch jobs tolerant to interruption
• az-pool — workloads that must run in a specific zone

#Weight: Scoring Preference

When a pod can be scheduled by more than one NodePool, Karpenter decides which NodePool to use. The spec.weight field sets the preference: a NodePool with a larger weight is prioritized, and its default value is zero. For example, a general NodePool with weight 10 would be chosen before a spot-only NodePool with weight 5.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  weight: 10
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 1m

#Taints: Separating Special Workloads

Weight manages preference, but taints manage permission. A GPU NodePool is tainted so ordinary pods can't get in, and only pods with the nvidia.com/gpu toleration are scheduled there.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-pool
spec:
  weight: 20
  template:
    spec:
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.xlarge", "g5.2xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

With this configuration, ordinary web workloads will never run on expensive GPU nodes.

#NodePools for Specific Zones

For workloads that need a particular location — for instance due to proximity to a database or data regulations — a NodePool can be limited to a specific zone via the topology.kubernetes.io/zone label.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: az-a
spec:
  template:
    spec:
      requirements:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

#NodePool Separation Matrix

#Multi-Cluster

#Karpenter Runs Per Cluster

Karpenter is a controller that lives inside one cluster and manages that cluster's nodes. There's no central Karpenter managing many clusters. When you have ten clusters, that means ten separately running Karpenter instances — each with its own NodePools, NodeClasses, and disruption policies.

The benefit of this isolation is a small blast radius. A configuration error in one cluster doesn't spread to another. But the consequence is that every cluster needs to be managed with the same discipline to avoid configuration drift between environments.

#Consistent Per-Cluster Policies

The first multi-cluster best practice is consistency through code. Define NodePools and NodeClasses as files managed via GitOps, for example with ArgoCD or Flux, then apply them to all clusters from a single source. This way, differences between clusters can be reduced to only the things that genuinely must differ.

kubectl apply -f clusters/staging/karpenter/
kubectl apply -f clusters/production/karpenter/

Additionally, watch out for the following:

• Use uniform resource names across clusters so linting and audit tooling works identically.
• Differentiate NodePools per environment when needed, for example staging using spot only and production with more on-demand.
• Secure the configuration by restricting who can write NodePools and NodeClasses via RBAC.
• Monitor configuration drift by comparing applied state against expected state.

#Example Policy per Environment

Staging clusters can usually run aggressively on cost savings, while production needs more safety nets. The following configuration illustrates that difference.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 15m

#Common Mistakes

#Weight Without Proper Taints

A large weight doesn't prevent misplaced workloads if taints aren't set. Use both together: weight for preference, taints for hard restrictions.

#One Configuration for All Environments

Forcing staging and production to use identical NodePools ignores the differences in cost and stability needs. Adjust policies per environment, but keep a single source of truth in Git.

#Closing

Multi-NodePool and multi-cluster are two levels of organization that must be designed, not left to grow wild. Weight, taints, and zone isolation keep one cluster organized; GitOps and per-environment policies keep many clusters in sync.

Key takeaways:

• NodePools separate interests — general, GPU, spot-only, and zone-specific pools can coexist.
• Weight for preference, taints for permission — both work together, not as replacements.
• Karpenter runs per cluster — each cluster manages its own policies and infrastructure.
• GitOps keeps consistency — a single source of truth prevents configs from diverging across clusters.

In episode 11, you'll learn about drift detection — how Karpenter detects differences between the NodePool and NodeClass templates and actual nodes, plus the NodeOverlay feature for updating nodes without a full replacement. See you there!