Learn Cloud Hypervisor - Hotplug: CPU, Memory & Device

#Introduction

With networking ready in episode 7, now we talk operational agility: the ability to change VM resources without shutting it down. In the cloud world, "restart to add RAM" is a luxury you can't always afford — especially when a production workload is serving thousands of requests.

Hotplug in Cloud Hypervisor is built on kernel mechanisms (ACPI) and operated through the API socket. In episode 8 we add vCPUs and memory live, verify them inside the guest, then cover the highest level of flexibility: device passthrough with VFIO to give physical devices directly to a VM.

#API Socket: Remote Control for the VM

To operate a running VM, enable API control at startup:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --cpus boot=2,max=16 \
  --memory size=2G,hotplug_size=12G,hotplugged_size=0 \
  --api-socket /tmp/ch.sock \
  --serial tty

--api-socket /tmp/ch.sock opens a Unix socket as the remote control. All hotplug, snapshot, and live migration operations are done through this socket using curl (Cloud Hypervisor uses HTTP over the Unix socket).

curl --unix-socket /tmp/ch.sock http://localhost/api/v1/vm.info

vm.info shows the VM's current configuration — from vCPU and memory to attached devices. It's the starting point for all runtime operations.

#CPU Hotplug

#Adding vCPUs Live

curl --unix-socket /tmp/ch.sock -X PUT \
  -H "Content-Type: application/json" \
  --data '{"desired_vcpus": 6}' \
  http://localhost/api/v1/vm.resize

desired_vcpus: 6 raises the vCPU count from 2 to 6 without stopping the VM. The upper bound is the max set at startup (max=16 in the example above). To scale back down, set desired_vcpus to a smaller number.

#Reflection in the Guest

Inside the guest, the new vCPUs appear as online CPUs after the OS processes the ACPI notification. Verify:

nproc
lscpu | grep "^CPU(s):"
cat /sys/devices/system/cpu/online

Sometimes new CPUs need to be activated explicitly (depending on distro and kernel config). If nproc hasn't changed, check whether the CPU is detected but offline:

ls /sys/devices/system/cpu/cpu6/
echo 1 | sudo tee /sys/devices/system/cpu/cpu6/online

#Memory Hotplug

#Adding Memory Live

Memory is hotplugged through an ACPI device that can be "plugged in" to the guest. With hotplug_size=12G in the config, we have room to add:

curl --unix-socket /tmp/ch.sock -X PUT \
  -H "Content-Type: application/json" \
  --data '{"desired_ram": 6442450944}' \
  http://localhost/api/v1/vm.resize

desired_ram: 6442450944 (6 GB) raises RAM from 2 GB to 6 GB. The number is in bytes — the same trap as the JSON config in episode 5.

#Reflection in the Guest

Inside the guest, the additional memory appears as new memory blocks. Verify with:

free -h
grep -c "online" /sys/devices/system/memory/memory*/state

If the total memory in free -h hasn't changed, check whether the new memory block is still offline:

echo online | sudo tee /sys/devices/system/memory/memory*/state

#Balloon: Reclaiming Memory

The opposite of adding, virtio-balloon lets the host reclaim memory the guest isn't using:

curl --unix-socket /tmp/ch.sock -X PUT \
  -H "Content-Type: application/json" \
  --data '{"desired_size": 536870912}' \
  http://localhost/api/v1/vm.resize

desired_size: 536870912 (512 MB) pressures the guest to return memory to the host. This is the basis of memory oversubscription: run many VMs on limited physical memory, then reclaim when load drops.

#Device Passthrough: VFIO

#The VFIO Concept

Sometimes virtio isn't enough — a workload needs physical devices directly: GPUs for rendering/AI, special NICs, or FPGAs. VFIO (Virtual Function I/O) lets a host PCI device be detached from the host and handed to the VM whole. The guest talks directly to the hardware, without emulation, with the help of the IOMMU.

First, detach the device from its host driver and bind it to vfio-pci:

echo 0000:01:00.0 | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo vfio-pci | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver_override
echo 0000:01:00.0 | sudo tee /sys/bus/pci/drivers_probe

Then present it to the VM with --device:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --device path=/sys/bus/pci/devices/0000:01:00.0 \
  --cpus boot=4 \
  --memory size=8G

--device path=... hands the vfio device to the guest. Inside the guest, the device appears as a normal PCI device with the appropriate driver.

#VFIO Prerequisites and Pitfalls

• IOMMU active: without a host IOMMU, VFIO refuses to operate (intel_iommu=on or amd_iommu=on in the host kernel cmdline).
• Isolation: it's easier if the device is in its own IOMMU group; devices grouped with others force all of them to be detached.
• Memory: guest memory must be shared=on for passthrough, because the physical device reads guest memory via DMA.

ls -l /sys/kernel/iommu_groups/ | head

#vfio-user: Passthrough Without Bare-metal

vfio-user is an experimental protocol that provides a virtual PCI device from another process — like VFIO, but the device is simulated in userspace, not physical hardware. This opens the door to software-defined "passthrough": a device (e.g., a virtual NVMe or GPU) runs in a separate process and is connected to the VM.

The advantages: better isolation (an error in the device process doesn't bring down the VMM) and safer memory handling. Because it's still experimental, use it carefully and watch its status on the official roadmap (episode 17).

#Common Hotplug Pitfalls

• Exceeding max: adding vCPUs/memory above the limit set at startup is rejected by the API. Set max and hotplug_size from the start.
• Guest doesn't respond to ACPI: check that the guest kernel has CONFIG_ACPI_HOTPLUG_CPU/CONFIG_ACPI_HOTPLUG_MEMORY and that acpid is running.
• Balloon pressing the guest's memory: a balloon that's too small can make the guest OOM — monitor free in the guest.
• vm.info is the source of truth: before blaming the guest, check vm.info to make sure the hotplug request was actually accepted by the VMM.

#Conclusion

Key takeaways:

• The API socket (--api-socket) is the remote control for runtime operations via HTTP.
• desired_vcpus and desired_ram in /api/v1/vm.resize add resources live.
• The guest reflects hotplug via ACPI; verify with nproc, free, and sysfs.
• Balloon (desired_size) reclaims idle memory back to the host.
• VFIO passthrough hands a physical device to the guest; it needs an IOMMU and shared=on.
• vfio-user is an experimental userspace-based virtual passthrough.

In the next episode, episode 9, we'll prepare cloud images & cloud-init — downloading Ubuntu/Fedora/Debian cloud images, converting them to raw, booting via direct or UEFI, and doing automatic bootstrap (password, SSH key, packages) with cloud-init user-data and metadata. Your VMs will be ready to use without manual interaction!