Learn Karpenter - Networking Integration

#Introduction

In episode 12 you learned to observe Karpenter through Prometheus metrics and build proper alerting. Now we go one layer deeper, to something that determines whether a newly created node can actually serve traffic: networking. A node that launches perfectly but lands in the wrong subnet, or rejects pods because the ENI quota is exhausted, makes all that provisioning pointless.

This episode covers networking integration. You will understand how Karpenter selects subnets and Availability Zones through NodeClass, how security groups are applied, how new nodes behave with respect to ENI and Pod ENI, how VPC CNI works, and finally the use of EFA for HPC and machine learning workloads that need very low inter-node latency.

#The Role of NodeClass in Networking

All infrastructure-level networking decisions are controlled by the EC2NodeClass, not the NodePool. The NodePool selects the instance type and scheduling, while the NodeClass determines where the node is placed and how it connects. This separation keeps one concern per resource: NodePool for the workload's wishes, NodeClass for the VPC's reality.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster

Subnet selection is done through tags, not names or IDs. With this pattern, new subnets added to the VPC are automatically recognized by Karpenter without changing any configuration. Use kubectl describe ec2nodeclass default to see the subnets and AZs that match the selector.

#Selecting Subnets and Availability Zones

Karpenter selects subnets based on labels automatically added to nodes, such as topology.kubernetes.io/zone and node.kubernetes.io/instance-type. If a workload requires a specific AZ — for example to reduce cross-AZ data transfer costs — you can restrict it through the NodePool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["ap-southeast-1a", "ap-southeast-1b"]
  disruption:
    consolidationPolicy: WhenUnderutilized

#Understanding New Node Behavior

When Karpenter creates a new node, the networking sequence is: the instance launches in the selected subnet, the security group is applied from the selector, then VPC CNI attaches ENIs to the instance and allocates IPs to pods. The speed of this sequence affects karpenter_pods_startup_time_seconds, which you learned about in episode 12.

#Security Groups and Connectivity

The security groups Karpenter selects become the traffic perimeter of the node. Selection is done through securityGroupSelectorTerms, and all nodes from one NodeClass share the same security groups. For an EKS cluster, make sure the selected security groups allow traffic between nodes and from the control plane.

Don't rely on manually setting kubelet --node-ip; Karpenter manages connectivity through labels and security groups. To verify, check a newly created node with kubectl get nodes -o wide and make sure INTERNAL-IP falls within the expected subnet CIDR.

#ENI, Pod ENI, and VPC CNI

Each EC2 instance has an ENI and IP quota based on its instance type. VPC CNI attaches one primary ENI and adds more ENIs as demand grows, or uses Pod ENI mode where every pod gets its own ENI and VPC IP. Karpenter must be aware of this capacity so it does not schedule more pods than the quota allows.

• Primary ENI: attached when the instance is born, used for the node and a certain number of pods.
• Additional ENIs: added when the pod quota runs out, increasing how many pods can be hosted.
• Pod ENI (Security Groups for Pods): each pod gets a separate ENI with its own security group, but IP consumption is much higher.
• Prefix delegation: allocates /28 prefixes to multiply the number of pods per ENI.

#Custom Networking and Prefix Delegation

Some teams choose custom networking, where pods are allocated IPs from a CIDR separate from the node. This reduces the risk of exhausting IPs in the node subnet, but requires additional subnets for pods. If custom networking is active, make sure the NodeClass subnet selector picks the right subnets for nodes, and the pod subnets are configured in VPC CNI.

# Lihat jumlah ENI dan IP yang dipakai pod
kubectl get nodes -o custom-columns=NAME:.metadata.name,MAX_PODS:.status.capacity.pods
 
# Periksa log VPC CNI pada salah satu node
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-node --tail=50{:bash}

Prefix delegation raises the number of pods per ENI from 35 to around 110 for certain instance types. The benefit is more pods per node, reducing the number of instances and cost. The downside is that available IPs in the subnet run out faster, so pay attention to the CIDR limit.

#EFA for HPC and Machine Learning

For distributed HPC and ML workloads that need very low latency and high throughput between nodes, use Elastic Fabric Adapter (EFA). EFA provides direct access to the network hardware, bypassing the kernel for part of the path, producing consistent latency for communication patterns such as MPI and NCCL.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: efa
spec:
  amiFamily: Bottlerocket
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
        purpose: hpc

#Custom Network Interfaces

For advanced cases, you can assign a specific network interface through Karpenter's custom network interfaces feature, for example when a node must attach to a particular ENI or use an elastic IP. This is rarely needed and fits best with network security appliance integrations or static public IP requirements.

#Closing

Networking is the foundation that makes the nodes Karpenter launches genuinely useful to workloads.

Key takeaways:

• NodeClass owns networking: subnets, security groups, and connectivity are controlled from the EC2NodeClass, while the NodePool only expresses scheduling intent.
• Selection via tags: use tag selectors for subnets and security groups so infrastructure additions never require config changes.
• Understand ENI quotas: the number of pods per node is bounded by ENI and IP quotas, so match instance type selection to pod requirements.
• Pod ENI and prefix delegation are purpose-built tools: use them when needed, not by default, because both change IP consumption patterns significantly.
• EFA for HPC loads: choose the right instance and subnet, and test an MPI job before scheduling large workloads.

Networking is ready, but a node connected to the VPC is not necessarily secure. In episode 14 we cover Security & IAM — minimal policies for the controller, instance profiles, IRSA and Pod Identity, hardened AMIs, SSH control, userData, up to isolation between NodePools with taints and tolerations. See you there!