Learn Cloud Hypervisor - Testing & CI

#Introduction

After exploring nearly every Cloud Hypervisor feature, now we look at how this project keeps itself reliable. A VMM without rigorous testing is a disaster — one bug in the device model can mean broken isolation between tenants (remember CVE-2026-27211 in episode 13). In episode 16 we dissect Cloud Hypervisor's testing strategy and how to build benchmarks to compare it with other VMMs.

There are two sides we cover: as a contributor (running cargo test and the integration tests) and as an operator (measuring VMM performance in your own environment before deciding to adopt it).

#Unit Tests and cargo test

#Project Structure

Cloud Hypervisor is a Rust workspace: modular crates under crates/ (vmm, devices, vm-memory, etc.). Unit tests are written alongside the code, testing pure logic — config parsing, device state machines, snapshot format — without KVM:

cargo test --workspace

cargo test --workspace compiles every crate and runs thousands of unit tests. Because they don't need hardware virtualization, these tests can run on any machine with a Rust toolchain — this is the fastest quality foundation.

#Filter and Focus

When developing one area, focus tests on a specific module:

cargo test -p vmm

And run one specific test with a name filter:

cargo test -p vmm snapshot

snapshot filters to tests whose names contain "snapshot" — practical when verifying changes to the snapshot format.

#Integration Tests: integration.rs

#What They Test

Unit tests aren't enough — components that are individually correct aren't necessarily correct together. That's why Cloud Hypervisor has integration tests that run a real VMM (needs KVM) and verify end-to-end behavior. The main file is tests/integration.rs: booting real VMs, testing hotplug, snapshot/restore, network traffic, live migration, and more.

cargo test --test integration

Each test sets up a VM from the freshly built binary, runs a scenario, and verifies the result. Example scenarios tested:

• Boot a kernel until the console prompt appears within a certain time.
• Hotplug CPU/memory, then verify inside the guest via an agent.
• Snapshot → restore → verify guest processes continue.
• Live migration between two VMM instances on the same host.
• Network: send host→guest→host packets and make sure they arrive intact.

#CI: Targeting Kernel 5.15

#Kernel Baseline

The project runs its CI against kernel 5.15 — an LTS kernel that serves as the support baseline. This is an important decision: supported features must run on the LTS kernel widely used in production, not just the newest kernel. CI flags failures that only appear on new kernels as regressions.

strategy:
  matrix:
    kernel:
      - 5.15
      - latest
    arch:
      - x86_64
      - aarch64
steps:
  - name: Run integration tests
    run: cargo test --test integration

The matrix above tests kernel and architecture combinations — ensuring features keep working on the baseline while following the latest kernel.

#Quality Gates

The typical CI flow: lint → unit test → build → integration test → basic benchmark. Only PRs that pass all gates can be merged. If you contribute, CONTRIBUTING.md explains how to run the same set locally — a practice you should copy in your own projects: automated gates before merge, not reliance on manual review.

#Benchmark: Cloud Hypervisor vs QEMU

#Why Benchmark Yourself

Claims like "fast boot" and "small footprint" are meaningless without numbers in your environment. A good benchmark answers three questions: how long does boot take, how much memory is used, and what I/O throughput can be reached.

#Boot Time

Measure the time from VMM start until the prompt is available:

time cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=ubuntu.raw \
  --cpus boot=2 \
  --memory size=1G \
  --cmdline "console=ttyS0 root=/dev/vda1 quiet"

Do the same with QEMU (identical kernel and image):

time qemu-system-x86_64 \
  -kernel vmlinuz -initrd initrd.img -append "console=ttyS0 root=/dev/vda1 quiet" \
  -drive file=ubuntu.raw,format=raw,if=virtio \
  -enable-kvm -m 1G -smp 2 -nographic

Compare the time output: the boot time difference in your environment is the real number you need — not claims from a blog.

#Memory Overhead

Compare the VMM process RSS under identical loads:

pgrep -f cloud-hypervisor | head -1 | xargs -I{} cat /proc/{}/status | grep VmRSS

Do the same for qemu-system-x86_64. Memory overhead = RSS minus guest memory (which you set). This is the actual per-VM cost of the VMM — the key number for calculating per-host density.

#I/O Throughput

Use benchmark tools inside the guest, e.g., fio for disk and iperf3 for network:

fio --name=randwrite --rw=randwrite --bs=4k --size=1G \
  --numjobs=4 --runtime=30 --ioengine=libaio --direct=1

iperf3 -c 192.168.100.1

Run identical scenarios on QEMU and Cloud Hypervisor, then compare: latency, IOPS, and throughput. Results often depend on the setup (hugepages, vhost-user, etc.) — that's why benchmarking in your own environment is more valuable than other people's numbers.

#Benchmark Pitfalls

• Loads aren't identical: use exactly the same kernel, image, and workload for both VMMs.
• Warm-up ignored: run the workload several times and take stable values, not the first attempt.
• Host memory cache: run benchmarks alternately and let the host "cool down" for consistent results.
• Config differences: QEMU defaults vs Cloud Hypervisor defaults can be unfair — document the options used.

#Conclusion

Key takeaways:

• cargo test runs thousands of unit tests without KVM; cargo test --test integration needs KVM.
• Integration tests verify boot, hotplug, snapshot/restore, migration, and networking end-to-end.
• CI targets kernel 5.15 as the LTS baseline plus the latest kernel.
• Honest benchmarks measure boot time, memory overhead (RSS), and I/O throughput.
• Benchmarks in your own environment beat claims from documentation.
• Make benchmarks a script you can re-run on every upgrade.

In the next episode, episode 17, we'll cover v53.0 & roadmap — the newly introduced offloaded snapshot/restore daemon and live migration with page-faults served from the source, then the development direction: cross-version live migration stability, RISC-V, and more mature device passthrough. The future of Cloud Hypervisor starts here.