Learn Kata Containers - Kata Agent & Image Management

#Introduction

In episode 2 you met the Kata agent as the first process born inside the guest. Episode 9 digs into its role in depth: what it actually does, how it manages containers inside the microVM, and how images reach the guest. This is the episode that explains why Kata feels like a "container", even though there's a virtual machine behind it.

Understanding the Kata agent is also practical: when kubectl exec behaves strangely or an image fails to be found in the guest, the root cause is usually in this agent layer.

#The Kata Agent's Role Inside the Guest

#The First Process in the Guest

When the microVM boots, the guest kernel starts the first process: the Kata agent. It's the containerd analog inside the guest — the only process authorized to manage containers inside the microVM. All requests from the host (through the shim) reach the agent via vsock.

The Kata agent's main tasks:

• Creating and deleting containers inside the guest.
• Running application processes (fork + exec inside the guest).
• Managing mounts and volumes inside the guest.
• Forwarding signals (SIGTERM, SIGKILL) from the host to guest processes.
• Reporting status and exit codes back to the host.
• Handling exec (interactive shell into containers in the guest).

#Managing Containers Inside the Guest

There are two agent styles for managing containers in the guest:

• Agent-native: the agent manages containers directly, without an additional runtime in the guest. This is the modern direction — lighter and faster, fully supported by runtime-rs.
• runc inside the guest: the old approach where the agent uses runc inside the guest to create containers. Still used in some configurations, especially for compatibility with certain features.

This difference matters to understand because it affects overhead: containers in the guest created agent-native don't stack extra processes, while runc-in-guest adds one more process layer.

#Exec and Interactive Shell

kubectl exec is an interesting flow to dissect. This command doesn't directly touch the guest process — it passes through several layers:

1. kubelet receives the kubectl exec request and forwards it to containerd.
2. containerd calls the Kata shim.
3. The shim forwards the request to the Kata agent through vsock.
4. The agent creates a new process inside the guest and connects it to the terminal.

The consequence: kubectl exec in a Kata pod is a little slower than in a runc pod because it goes through the vsock path. For an interactive terminal, make sure TTY is enabled:

kubectl exec -it kata-demo -- sh

kubectl exec -it kata-demo -- sh enters the container shell inside the guest. All commands you run execute in the guest, not on the host — you can prove this by looking at the running processes.

#Image Management: How Images Reach the Guest

#Strategy 1: Pull on the Host, Mount into the Guest (Snapshotter)

The default strategy: containerd pulls the image on the host, then Kata provides the rootfs image to the guest. Exactly as we discussed in episode 8 — the rootfs becomes a virtio-blk device or is shared via virtio-fs.

Advantages of this strategy:

• One image cache for all runtimes: an image already pulled for a runc pod can be used directly by a Kata pod.
• Safer pulling: images are verified on the host with containerd's mature mechanisms.
• More resource-efficient: the guest doesn't need to communicate with the registry.

The downside: the rootfs needs to be moved host → guest through virtio, and there's the additional snapshotter processing cost. For large images, this time is felt at pod startup.

#Strategy 2: Pull Directly in the Guest

Alternatively, the guest pulls the image directly from the registry — bypassing the host containerd entirely. The Kata agent performs the pull inside the guest and stores the image on the guest disk.

The advantage: the guest has a self-contained copy of the image, and the rootfs doesn't need to be moved. The downside: the host image cache isn't used, the pull can be slower (the guest accesses the registry), and every pod pulls the image itself if it isn't cached.

#When to Choose

Most deployments use strategy 1 because it leverages the containerd ecosystem and existing caches. Strategy 2 is relevant for environments with very strict isolation or requirements that images leave no trace on the host.

Check the images available on the node:

crictl images

crictl images shows the images containerd has pulled on the host. This is also a quick way to verify that a Kata pod's image really exists on the host before diagnosing further.

#Handling Image Pull Issues

When a Kata pod fails because of an image, the usual diagnostic sequence:

1. kubectl describe pod — look at the events and error message.
2. crictl images — make sure the image has been pulled on the host.
3. crictl inspect <container-id> — check the container status on the CRI side.
4. Logs in /var/log/kata-containers/ — if the error is in the guest layer.

We'll cover debugging thoroughly in episode 12, but one important principle can be taken now: image problems usually show up first in the containerd layer before touching the guest. Always start diagnosis there.

#Conclusion

What you should take away:

• The Kata agent is the first process in the guest and manages all containers inside it.
• The agent handles create, exec, mount, and signals — all through vsock from the shim.
• Management styles: agent-native (modern) vs runc inside the guest.
• Images are pulled on the host and the rootfs is provided to the guest (default), or pulled directly in the guest.
• crictl images verifies images on the host before further diagnosis.
• kubectl exec goes through the shim → vsock → agent → guest path.

In the next episode, episode 10, we'll cover GPU & device passthrough — VFIO passthrough for AI/ML workloads, the role of NRI (Node Resource Interface), KEDA autoscaling integration, and how resource limits are applied inside the VM.