Learn Cloud Hypervisor - Performance & Observability

#Introduction

So far our VMs "run". In episode 20 we ask: how well do they run? Performance and observability are two sides of the same coin: without measurement there's no optimization, and without observation there's no diagnosis. In this episode we tune performance — hugepages, CPU pinning, NUMA awareness — then build observability — logs, metrics, tracing, and debugging with gdb.

Think of performance like tuning a car engine: hugepages enlarge the "fast lane tank" (TLB), pinning locks workers in fixed positions so they don't bounce around, and NUMA awareness ensures data sits close to its processor. Observability is the dashboard showing whether all of that works or not.

#Performance Tuning

#Hugepages

We touched hugepages in episode 5. Now we optimize them. Allocating 2 MB pages reduces the number of TLB entries, making memory access faster:

echo 1024 | sudo tee /proc/sys/vm/nr_hugepages
grep HugePages_Total /proc/meminfo

Then run the VM with hugepages:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --cpus boot=4 \
  --memory size=4G,hugepages=on,shared=on

hugepages=on builds guest RAM from the host's hugepage pool; shared=on is needed for snapshot/migration (episodes 10-11). Verify inside the guest that 2 MB pages are really in use:

grep -i huge /proc/meminfo

#CPU Pinning and Affinity

By default, vCPU threads can move between physical CPUs. Pinning locks each vCPU to a specific physical CPU — reducing cache thrashing and context switching, very noticeable for latency-sensitive workloads:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --cpus boot=4 \
  --memory size=4G \
  --cpu-affinity vcpu:0-3

--cpu-affinity vcpu:0-3 maps all four vCPUs to physical CPUs 0-3. Combine it with isolcpus on the host so other workloads don't fight over those cores:

isolcpus=2,3,6,7 nohz_full=2,3,6,7 rcu_nocbs=2,3,6,7

#NUMA Awareness

On multi-socket hosts, memory close to a CPU (its NUMA node) is much faster to access than far memory. For memory-intensive workloads, make sure the VM sticks to one NUMA node, and map the topology into the guest so applications can tune themselves:

numactl --hardware

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --cpus boot=8,topology="1 socket,8 cores,1 thread" \
  --memory size=8G \
  --numa guest_numa_id=0,cpus=[0-7],distances=[10,20]

Inside the guest, applications can see the topology and tune their allocations:

numactl --hardware
numactl --show

#Observability

#VMM Logs

Cloud Hypervisor writes logs to stderr by default. Enable --log-file to save the log and control its level:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --cpus boot=4 \
  --memory size=4G \
  --log-file /var/log/ch-vm.log \
  --log-level info

Available levels: error, warning, info, debug, trace. trace is very detailed (every KVM ioctl and device event) — turn it on only while debugging.

When there's a problem, work through the log:

grep -iE "error|fatal" /var/log/ch-vm.log | tail -20

#Metrics

The VMM produces metrics (VM-exit counts, disk I/O, network traffic, etc.) that can be exported for monitoring. Set up a metrics endpoint:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --cpus boot=4 \
  --memory size=4G \
  --metrics path=http://localhost:9091/metrics

Prometheus can scrape this endpoint. Useful metrics include cloud_hypervisor_vm_exits (VM-exit frequency) and cloud_hypervisor_disk_read_bytes. Watch trends, not just absolute values — a VM-exit spike often signals a problematic device.

#Tracing

For deep analysis, enable tracing (e.g., using perf on the host for the vCPU threads):

perf record -e kvm:kvm_exit -g -- pid=$(pgrep -f cloud-hypervisor)
perf report

perf kvm shows the distribution of VM-exit causes — whether I/O, timers, or something else dominates. This is valuable data for deciding whether you need vhost-user, hugepages, or a different device configuration.

#Debugging with gdb

For the hardest problems (hangs, VMM crashes), attach gdb to the process:

sudo gdb -p $(pgrep -f 'cloud-hypervisor' | head -1)

Inside gdb, take a backtrace to see where the process is stuck:

(gdb) thread apply all bt
(gdb) bt

For consistently reproducible crashes, run under gdb from the start:

gdb --args cloud-hypervisor --kernel kernel-vmlinux --disk path=os.raw
(gdb) run

When the crash happens, gdb holds the process — from there you can see the call stack and diagnose. If you find a bug, report it with a full backtrace to the official repository (episode 21).

#Recommended Optimization Workflow

1. Measure the baseline: boot time, memory RSS, throughput (episode 16).
2. Identify the bottleneck: from metrics and perf — CPU, memory, or I/O.
3. Apply one change: hugepages, pinning, or NUMA — one at a time.
4. Measure again: compare with the baseline.
5. Repeat: keep what gives real gains, revert what doesn't.

#Common Pitfalls

• Hugepages without a sufficient pool: the VM fails to start. Match nr_hugepages to --memory size.
• Pinning to busy CPUs: affinity is useless if the core is also used by other workloads — combine it with isolcpus.
• Wrong NUMA settings: a topology that doesn't match the host makes the guest place data on the wrong node.
• trace log level in production: floods the disk — use it only while debugging.
• gdb without ptrace permission: run with sudo or set ptrace_scope as needed.

#Conclusion

Key takeaways:

• Hugepages reduce TLB misses; enable with hugepages=on.
• CPU pinning (--cpu-affinity) reduces cache thrashing for latency-sensitive workloads.
• NUMA awareness keeps memory close to its processor.
• VMM logs (--log-file, --log-level) are your first diagnosis source.
• Metrics via an HTTP endpoint are ready for Prometheus scraping.
• perf kvm for VM-exit analysis; gdb for hangs and crashes.
• Optimization must always be measured: baseline → change → measure again.

In the next episode, episode 21, we'll cover ecosystem & governance — the cloud-hypervisor GitHub, roadmap, Slack, and mailing list, the founding charter with open governance (Intel, AMD, Arm, Microsoft), and the rust-vmm project sharing crates with Firecracker and crosvm. The community behind this VMM is part of the technology too.