Learn Kata Containers - Debugging & Troubleshooting

#Introduction

Up to episode 11, everything ran smoothly in the lab. Episode 12 prepares you for the times when everything doesn't go smoothly — and trust me, that will happen. Because a Kata pod has extra layers (shim, VMM, guest kernel, agent), its troubleshooting has extra layers too. Knowing how to think in a structured way will save you hours of guessing.

Episode 12 builds a diagnostic mental model: starting from Kubernetes (the outermost), going down to containerd, the shim, the VMM, and the guest. We also cover real cases: pending pods, OOM inside the VM, failed image pulls, and wrong hypervisor configuration.

#Diagnostic Strategy: From Outside In

When a Kata pod has a problem, follow this order — don't jump into the guest first:

1. Kubernetes: kubectl describe pod and kubectl get events — status and reasons at the cluster level.
2. containerd: crictl ps and crictl inspect — what's happening on the CRI side.
3. Runtime/shims: kata-runtime list and kata-runtime state — sandboxes known to the runtime.
4. Kata logs: /var/log/kata-containers/ — details from the shim, VMM, and agent.
5. Inside the guest: kata-runtime exec and the debug console — seeing directly what happens in the microVM.

Always start from the outer layer. Most problems — images, quotas, scheduling — appear in the first layer and don't require touching the guest.

#The Kata Debugging Toolkit

#kata-runtime list and state

kata-runtime list shows the microVM sandboxes the runtime is managing on the node:

kata-runtime list

kata-runtime list displays a sandbox table: ID, CRI version, status, and hypervisor PID. This is the quick answer to the question "is the microVM really there and running?".

kata-runtime state shows the status details of a specific sandbox:

kata-runtime state <sandbox-id>

kata-runtime state <sandbox-id> shows the full state: sandbox status, process PID, rootfs, and version. If the sandbox isn't found here even though the pod claims to be running, there's a mismatch between containerd and the runtime — start suspecting a shim problem.

#Kata Logs in /var/log/kata-containers/

All Kata components write logs to the /var/log/kata-containers/ directory. The main log files:

ls -la /var/log/kata-containers/
tail -f /var/log/kata-containers/shim.log

tail -f /var/log/kata-containers/shim.log shows the shim log in real time — the first place VMM and agent errors appear. For hypervisor problems, the QEMU/CH logs are also in this directory.

Note that the default debug level is low. For more detailed logs, enable debug in the configuration (episode 5):

/etc/kata-containers/configuration.toml

[runtime]
enable_debug = true
enable_debug_console = true

enable_debug = true increases the runtime logs, and enable_debug_console = true enables the console into the guest for kata-runtime exec — a very helpful pair when troubleshooting.

#kata-runtime exec and the Debug Console

To look directly into the guest:

kata-runtime list
sudo kata-runtime exec <sandbox-id> bash

kata-runtime exec <sandbox-id> bash gives you a shell inside the microVM. From here you can inspect processes, mounts, networking, and memory inside the guest — just like nsenter for containers, but for a VM.

#Case 1: Pod Pending or ContainerCreating

Symptom: the pod is stuck in ContainerCreating with no progress. Sequential diagnosis:

kubectl describe pod kata-demo
kubectl get events --sort-by=.lastTimestamp | tail -20

kubectl describe pod shows events like FailedCreatePodSandBox with a message pointing to the root cause. The most frequent causes:

• KVM not available: the microVM fails to boot. Check /dev/kvm on the node.
• Handler mismatch: the RuntimeClass names a handler containerd doesn't know.
• Node resources exhausted: the VM can't be allocated.

If the events aren't clear, check the shim log, which usually contains the QEMU/KVM error:

journalctl -u containerd -n 100 | grep -i kata
tail -50 /var/log/kata-containers/shim.log

journalctl -u containerd -n 100 | grep -i kata shows containerd's Kata-related logs. The combination of Kubernetes events + containerd logs + shim logs is usually enough to find the root cause.

#Case 2: OOM Inside the VM

Symptom: a process in the guest is killed even though the node has enough memory. This is the difference from a regular container OOM: what ran out of memory is the microVM, not the node.

Because pod limits determine the VM's size (episode 5), OOM in the guest usually means the VM is too small for the workload. Diagnosis:

kata-runtime list
sudo kata-runtime exec <sandbox-id> dmesg | grep -i "out of memory"
sudo kata-runtime exec <sandbox-id> cat /sys/fs/cgroup/memory.events

kata-runtime exec <sandbox-id> dmesg | grep -i "out of memory" shows the OOM records in the guest kernel. cat /sys/fs/cgroup/memory.events shows the oom counter — if oom_count increases, the workload is indeed exceeding the VM's limits.

The solution: raise the pod's memory limit, or make sure the request/limit matches the application's needs. Remember from episode 5 — the values in the guest reflect the pod's limits.

#Case 3: Failed Image Pull

From episode 9 we know images are pulled on the host then provided to the guest. The symptom of a failed image pull usually appears in the pod's events. Check on the host side:

crictl images | grep <nama-image>
crictl pull <nama-image>

crictl images | grep <nama-image> verifies the image in the host cache. If the pull fails in containerd, the problem is in the registry/credentials — not Kata. Expand with kubectl describe pod to see the ErrImagePull or ImagePullBackOff message.

If the image exists on the host but the pod still fails, only then suspect the Kata layer: a rootfs image that can't be provided to the guest, or an incompatible snapshotter.

#Case 4: Hypervisor Misconfiguration

Symptom: the pod fails with an error mentioning QEMU, KVM, or a device. This is usually a wrong hypervisor configuration:

• valid_hypervisor_paths doesn't include the path of an existing binary.
• kernel or image points to a file that doesn't exist.
• A backend is chosen in the RuntimeClass but its configuration file (e.g. configuration-fc.toml) doesn't exist.

Verify the currently effective configuration:

kata-runtime kata-env | grep -E "(path|kernel|image)"

kata-runtime kata-env | grep -E "(path|kernel|image)" shows the hypervisor binary path, guest kernel, and guest image currently in use. If any of them doesn't exist on the filesystem, that's the root cause.

#Conclusion

What you should take away:

• Diagnose from outside in: Kubernetes → containerd → shim → VMM → guest.
• kata-runtime list and kata-runtime state inspect sandboxes from the runtime's side.
• Kata logs in /var/log/kata-containers/ — enable debug for more detail.
• kata-runtime exec opens a shell inside the guest for direct inspection.
• OOM in the guest differs from cgroup OOM: the VM ran out of memory, not the node.
• kata-runtime kata-env verifies the effective hypervisor configuration.

In the next episode, episode 13, we'll cover security model & trust boundary — the question "do you trust the guest?", an analysis of CVE-2026-24834 which attacks the guest root through pmem/DAX, and hardening practices: minimal devices, non-root, seccomp, and routine updates. This is the episode that explains Kata's real security boundary.