Learn Kata Containers - GPU & Device Passthrough

#Introduction

Up to episode 9, all our workloads only used virtual CPU and memory. Episode 10 opens a different door: physical devices — especially GPUs. AI/ML workloads are the main reason many teams start considering microVM isolation: they want to run models and agents that execute unknown input, but still need GPU access for inference.

Kata Containers answers this need through VFIO passthrough: the GPU is released from the host and given directly to the microVM. This episode covers how passthrough works, the role of NRI for device management in Kubernetes, and how KEDA and resource limits work in a microVM environment.

#VFIO Passthrough for AI/ML

#Why Passthrough, Not a Virtual Device

Virtual GPUs (vGPU) have limited compatibility and overhead. For production AI/ML, teams prefer full passthrough: the physical GPU appears in the guest as a real device. Applications in the guest use CUDA/ROCm directly, as if running on bare metal.

The VFIO principle (from episode 8): the device is bound to the vfio-pci driver on the host, then attached to the microVM. The guest sees the GPU as a PCI device with full resources — GPU memory, compute units, everything.

#Passthrough Prerequisites

Passthrough demands IOMMU support. Without IOMMU, devices can't be safely isolated for passthrough. Prepare the host:

cat /proc/cmdline | grep iommu verifies IOMMU is active on the running kernel. ls /sys/kernel/iommu_groups/ shows the IOMMU groups — devices in the same group can't be passed through separately.

After IOMMU is active, bind the device to vfio-pci:

lspci -nn | grep -i nvidia finds the GPU's vendor/device ID. That ID is registered with vfio-pci so the device can be passed through.

#Providing the GPU to a Kata Pod

Kubernetes provides GPUs through a device plugin that creates extended resources (for example nvidia.com/gpu). Pods requesting that resource are scheduled to nodes with GPUs. For Kata, the GPU is allocated to the microVM via VFIO. Example pod:

Kata pod with a GPU

apiVersion: v1
kind: Pod
metadata:
  name: kata-gpu
spec:
  runtimeClassName: kata
  containers:
    - name: ml
      image: nvidia/cuda:12.2-base
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

nvidia.com/gpu: 1 requests one GPU. After the pod runs, nvidia-smi inside the guest should display the GPU — proof the passthrough worked.

#NRI: Node Resource Interface

NRI (Node Resource Interface) is a standard that connects container runtimes with device plugins and other node components. With NRI, plugins (for example the GPU plugin) can interact with the container lifecycle through callbacks — without modifying the runtime.

For Kata, NRI means device plugins don't need to understand microVM details. The plugin tells the runtime "this container needs this GPU", and the Kata layer translates it into a VFIO configuration for the microVM. This simplifies device integration in a Kata environment.

To enable NRI in containerd, add this to /etc/containerd/config.toml:

enable_nri = true enables NRI integration. With NRI active, device plugins work seamlessly with Kata pods.

#Resource Limits Applied at the VM

From episode 5 we know pod limits are translated into microVM allocation. In the context of GPUs and large workloads, this means:

CPU and memory are allocated for the whole VM — processes in the guest compete within VM limits, not host cgroup limits.
GPUs are allocated via passthrough — one GPU per pod (or shared according to MIG/SR-IOV mode on VMM support).
Oversubscription isn't recommended: because a VM holds its resources physically, don't schedule more Kata pods than the node's capacity.

Note this philosophical difference: with regular containers, cgroups throttle processes when they exceed limits. With Kata, limits determine the VM's size. This is firmer but less flexible — that's the price of isolation.

Warning

GPU passthrough sacrifices device-level isolation: the guest gets direct access to the physical GPU. For very sensitive multi-tenant workloads, consider GPU virtualization (MIG/SR-IOV) if the VMM supports it, or restrict nodes with GPUs to trusted workloads only.

#KEDA Autoscaling Integration

KEDA (Kubernetes Event-Driven Autoscaling) is an event-based autoscaling tool. How does it interact with Kata? Because Kata pods look like regular pods in Kubernetes, KEDA works without changes — it scales Deployments based on metrics, and new pods are born as new microVMs.

What you need to watch is boot time: because every new pod waits for a microVM boot, Kata scale-up is a bit slower than runc. For workloads needing fast scaling, consider:

Keeping a minimum pod count (KEDA minReplicaCount) so microVMs are already warm.
Measuring boot latency in your environment (episode 20).
Using VMM backends with fast boot (episode 6).

A simple KEDA trigger example:

KEDA ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kata-http-scaler
spec:
  scaleTargetRef:
    name: kata-app
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: http_requests_total
        threshold: "100"

maxReplicaCount: 10 caps the maximum scale. Because every replica is a microVM, think about node capacity: ten replicas mean ten microVMs, each holding its own resources.

#Common Pitfalls

Forgotten IOMMU: passthrough fails completely without an active IOMMU.
Non-persistent VFIO binding: the vfio-pci binding is lost after a reboot — use a udev rule or modprobe config.
GPU on a node without the device plugin: the pod requests a GPU but the resource doesn't exist — make sure the device plugin is running.
Memory overcommit: scheduling too many Kata pods starves the node of memory because every VM holds its allocation.

#Conclusion

What you should take away:

VFIO passthrough provides the physical GPU to the microVM for AI/ML workloads.
IOMMU is a mandatory prerequisite for passthrough.
NRI connects device plugins with the runtime without modifying it.
Resource limits determine the VM's size, not just cgroup limits.
KEDA works with Kata pods without changes; account for boot time during scale-up.
nvidia-smi inside the guest proves GPU passthrough succeeded.

In the next episode, episode 11, we'll cover Confidential Containers (CoCo) — encrypting workloads with TDX/SEV/SEV-SNP through the kata-qemu-coco-dev runtimeclass, the attestation and measurement process, and its use cases and limitations.