Learn Karpenter - Production-Ready Deployment

#Introduction

In episode 20 you learned about the latest Karpenter v1.14 features and how to upgrade safely. Features are raw materials; production needs architecture. This episode assembles all your knowledge into a deployment fit for a real team.

We start with the NodePool architecture: separating general, spot, and GPU. Then strict budgets, HPA and KEDA integration for pod scaling with Karpenter for node scaling, and disaster recovery. Finally, how to manage everything through Helm, CI/CD, GitOps, and monitoring.

#Multi-NodePool Architecture in Production

One NodePool for all workloads is rarely the best choice. Different needs — latency, cost, and special hardware — are best separated so disruption policies don't hold each other back: general uses on-demand for primary workloads, spot for interruption-tolerant loads, and gpu for workloads that truly need accelerators.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["m6i.xlarge", "m6i.2xlarge", "m6i.4xlarge"]
  disruption:
    consolidationPolicy: WhenUnderutilized
    budgets:
      - nodes: "20%"

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    spec:
      taints:
        - key: workload-type
          value: gpu
          effect: NoSchedule
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g4dn.xlarge", "g5.xlarge"]
  disruption:
    consolidationPolicy: WhenEmpty
    ttlSecondsAfterEmpty: 900

GPU workloads add a toleration for the workload-type: gpu taint. That way only pods that genuinely need GPUs sit on expensive instances, while ordinary workloads stay on the general NodePool.

#Strict Budgets

In production, disruption must be explicitly limited. Strict budgets prevent Karpenter from dismantling too many nodes at once and preserve the cluster's minimum capacity: limit the percentage of nodes that can be processed, limit drift during working hours, and give more room on weekends for AMI rotation.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    budgets:
      - nodes: "10%"
        reasons:
          - Drifted
        schedule: "0 9 * * mon-fri"
      - nodes: "50%"
        reasons:
          - Drifted

#HPA and KEDA: Scaling at the Pod Level

Karpenter manages nodes, not replicas. Replica scaling is the job of the Horizontal Pod Autoscaler and KEDA. HPA adds replicas based on metrics such as CPU and memory; KEDA extends this to event-driven metrics — SQS queue length, Kafka consumer lag, or the number of messages in a queue.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

KEDA uses a ScaledObject that points to an event source. Replicas grow when the queue piles up and shrink when the queue empties — far more responsive than waiting for CPU to rise.

#Integrating HPA and Karpenter

Both mechanisms work in a single, complementary loop. When traffic rises, HPA adds replicas. New pods may not fit on existing nodes, so Karpenter creates a new NodeClaim. When traffic falls, HPA trims replicas, nodes empty out, and consolidation dismantles them.

For this loop to stay healthy, set realistic resource requests so HPA calculations and binpacking stay accurate, give a sensible maxReplicas limit, and monitor karpenter_pods_startup_duration_seconds — the end-to-end latency from pending to running pods.

#Disaster Recovery

Karpenter itself stores no data — it reconstructs state from the cluster. Karpenter DR centers on two things: the ability to restore the cluster, and the manifests that recreate it. Store every NodePool and EC2NodeClass in Git, back up cluster state with a tool like Velero for stateful workloads, and periodically verify that a new cluster in a backup region can create nodes from the same NodePools.

#Helm Versions and Safe Upgrades

Version management starts with Helm. Karpenter is released as an OCI chart in the public.ecr.aws/karpenter/karpenter registry. Pin the version explicitly and never use the latest tag in production.

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version v1.14.0 \
  --namespace karpenter --create-namespace \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/KarpenterControllerRole \
  --wait

Upgrade procedure: read the upgrade guide and release notes, test on staging, check the CRD apiVersions used by your manifests, then upgrade with --wait and monitor pod and NodeClaim conditions.

#CI/CD and GitOps

NodePools and EC2NodeClasses should not be applied manually. With GitOps, Git becomes the single source of truth. Argo CD or Flux syncs changes to the cluster automatically: changes go through a pull request, the pipeline runs validation and dry-runs to staging, then promotes to production — all recorded in Git history for audit and instant rollback.

#Continuous Monitoring

Production monitoring is not just looking at dashboards — it is setting alerts. Karpenter exposes metrics on the /metrics endpoint, and Grafana provides an official dashboard. The most valuable alerts: a rising karpenter_cloudprovider_errors_total signals a cloud API problem, a bloated karpenter_pods_startup_duration_seconds signals slowing provisioning, and repeatedly failing NodeClaims signal a configuration or quota problem.

Routinely check cluster health with kubectl get nodeclaims -o yaml and kubectl logs -n karpenter deploy/karpenter --tail=100. With the right alerts, problems are detected before users feel the impact.

#Closing

Production-ready is an attitude, not a single configuration: separating workloads, limiting risk, automating changes, and monitoring the results.

Key takeaways:

• Separate NodePools: general, spot, and GPU with taints and tolerations keep costs and stability in check.
• Strict budgets protect the cluster: limit the percentage and reasons for disruption, and adjust based on observation.
• HPA, KEDA, and Karpenter work together: pod scaling triggers node scaling; realistic requests keep the loop accurate.
• GitOps and pinned Helm versions are the foundation: explicit versions, review, and automatic sync make changes measurable and reversible.
• Monitoring without alerts is just a dashboard: alerts on cloud provider errors and provisioning latency warn you before impact is felt.

You now have all the skills to operate Karpenter in production. In the final episode we look at the complete map: comparing Karpenter with Cluster Autoscaler, Cast AI, KubeFin, and platform-cost solutions, summarizing the journey from episode 0 to 21, and closing with a checklist and future directions. See you in the finale!