Learn Karpenter - Observability & Metrics

#Introduction

In episode 11 you learned about drift detection and how Karpenter keeps nodes aligned with the NodeClass and NodePool templates. But there is one question that always surfaces once an automated system is running: how do we know everything is working correctly? An autoscaler that is not observed is a ticking time bomb. Every provisioning and consolidation decision happens automatically without human supervision, so without metrics you will never know why a node was created, why it was removed, or why a pod is stuck as Pending.

This episode is about observability. You will learn to read the Prometheus metrics Karpenter exposes, understand what each metric group means, use a ready-made Grafana dashboard, and build alerting for two critical scenarios: pods that stay Pending too long and consolidation that fails or never happens.

#Why Karpenter Observability Is Different

A node autoscaler manages the most expensive resource in a cluster: EC2 machines. Unlike an HPA, which only changes replica counts, Karpenter creates and deletes real infrastructure, so an error here has a direct financial impact. Observability is not merely a want — it is a necessity to answer these questions:

• How quickly does Karpenter respond to a Pending pod?
• Is the chosen node genuinely efficient for the workload?
• Does consolidation produce savings, or does it cause churn instead?
• Are AWS interruption events handled on time?
• Are there pods left as Pending without a clear cause?

#Node Metric Group

Metrics prefixed with karpenter_nodes_* describe the lifecycle of nodes managed by Karpenter. The most commonly used is karpenter_nodes_allocatable — a gauge holding the amount of resources available on nodes owned by Karpenter, with a resource label such as cpu, memory, and pods.

# Kapasitas CPU total yang disediakan Karpenter
sum(karpenter_nodes_allocatable{resource="cpu"})
 
# Jumlah node yang dibuat dan dihapus dalam satu jam terakhir
sum(increase(karpenter_nodes_created[1h]))
sum(increase(karpenter_nodes_terminated[1h]))
 
# Perbandingan resource yang terpakai vs total allocatable
sum(karpenter_nodes_allocatable{resource="cpu"}) -
  sum(karpenter_node_utilization{resource="cpu"})

Notice the karpenter_nodes_terminated metric together with the termination reason label. A spike here alongside a drop in pods can indicate consolidation is working, but if it keeps happening without a pause, there may be a problem with PDBs or disruption.budgets that are too aggressive.

#Pod Metric Group

karpenter_pods_startup_time_seconds is a histogram measuring the time from pod submission until the pod is ready to receive traffic. This is the best metric for evaluating Karpenter's core promise: provisioning latency in seconds. Dividing _sum by _count gives the average startup time.

# Rata-rata startup pod dalam detik
rate(karpenter_pods_startup_time_seconds_sum[5m]) /
  rate(karpenter_pods_startup_time_seconds_count[5m])

There is also karpenter_pods_pending_duration_seconds, which measures how long pods wait in the scheduling queue. If this metric is high while nodes are not being created, the problem is not provisioning speed but a NodePool constraint that is too narrow or exhausted AWS quotas.

#Scheduling and Provisioning Metrics

Karpenter exposes karpenter_provisioner_scheduling_duration_seconds to measure the duration of scheduling decisions — from the moment a pod enters the queue until an instance type is selected. This metric matters because it separates two different sources of delay: Karpenter's internal compute time versus the AWS API time needed to launch an instance.

#Collecting Metrics with Prometheus

Karpenter exposes the /metrics endpoint on port 8080 in the karpenter namespace. Prometheus can collect them through a ServiceMonitor, or with a direct scrape config like the example below:

scrape_configs:
  - job_name: karpenter
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
            - karpenter
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
        regex: karpenter
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        regex: http
        action: keep

Once scraping is active, verify that metrics are coming in via kubectl port-forward -n karpenter service/karpenter 8080:8080 and visit the endpoint. For production scale, store data longer with remote write or Thanos, because Karpenter metrics are most useful when they can be compared across weeks to spot cost and latency trends.

#Grafana Dashboard

Building a dashboard from scratch takes time. Fortunately, the community and AWS provide ready-to-import dashboards. Karpenter ships the official JSON dashboard in its repository, and the aws-samples EKS deployment examples also include Karpenter panels alongside other node metrics.

• Official Karpenter dashboard: brings node, pod, and provisioner metric panels onto a single screen.
• EKS workshop / aws-samples dashboard: packaged together with other EKS dashboards, a good fit for real EKS cluster conditions.
• Custom panels: karpenter_nodes_allocatable compared against karpenter_pods_startup_time_seconds is often enough to monitor efficiency.

#Alerting for Pods Stuck in Pending

A pod sitting idle as Pending is the earliest signal that autoscaling has failed. Combining Kubelet and kube-state-metrics metrics produces an accurate alert:

groups:
  - name: karpenter-alerts
    rules:
      - alert: KarpenterPodPendingLong
        expr: |
          time() - kube_pod_created > 300
            and on(namespace, pod) kube_pod_status_phase{phase="Pending"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Pod pending lebih dari 5 menit"

When this alert fires, follow this diagnosis path: check NodeClaim status with kubectl get nodeclaims, inspect the controller logs with kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter, and make sure the EC2 instance limit has not been reached. Episode 18 covers this troubleshooting in depth.

#Alerting for Consolidation

Consolidation is a cost-saving feature, but it can also become a source of disruption when misconfigured. Monitor karpenter_consolidation_seconds_since_last to detect consolidation that has not run at all, and karpenter_consolidation_pods_evicted_total to see how many pods have been evicted.

# Berapa detik terakhir sejak konsolidasi terakhir berjalan
karpenter_consolidation_seconds_since_last
 
# Pod yang di-evict oleh konsolidasi dalam satu jam
sum(increase(karpenter_consolidation_pods_evicted_total[1h]))

#Closing

Observability turns Karpenter from a black box into a system that is predictable and accountable.

Key takeaways:

• Four main metric groups: node, pod, scheduling, and consolidation — each answers a different operational question and should be monitored together.
• Time histograms are gold: karpenter_pods_startup_time_seconds and the scheduling metrics show the real latency users experience, not just cluster status.
• Start scraping early: install Prometheus when Karpenter is set up, not after a problem occurs, so historical data is available for comparison.
• A Pending alert is a must: a pod idle for more than a few minutes is an autoscaling failure that must trigger an alert, not just be visible on a dashboard.
• Dashboards speed up diagnosis: use the official dashboard or the one from aws-samples as a starting point, then tailor it to your cost labels and NodePools.

The metrics are ready to be monitored, but there is one aspect that determines whether your cluster is safe for production: networking. In episode 13 we discuss Networking Integration — how Karpenter selects subnets and Availability Zones, the role of security groups, how ENI and Pod ENI work, VPC CNI, and even EFA for HPC and ML workloads. See you there!