Learn Cloud Hypervisor - Live Migration

#Introduction

The snapshot we learned in episode 10 does capture VM state, but there's a brief pause while the VM is stopped. To maintain full availability — a server serving requests must not stop for even a second — we need live migration: moving a VM from one host to another with no downtime perceived by the guest.

The best analogy is moving a busy shop to a new building: all goods are moved bit by bit while customers keep being served, and only when the move is nearly complete is there a final, instantaneous switch. In episode 11 we dissect how Cloud Hypervisor does this, what the prerequisites are, and the constraints you must respect in production.

#Live Migration Basics

#How It Works

Migration runs in several phases:

1. Preparation: the destination (target) host prepares a new VM with the same configuration, but in a "waiting" state.
2. Pre-copy: guest memory is copied from source to target incrementally while the VM keeps running on the source. Pages that change during copying (dirty pages) are tracked and copied again in the next iteration.
3. Stop-and-copy: when the remaining dirty pages are small enough, the VM is paused briefly, the remaining memory and vCPU/device state are copied, and execution switches to the target.
4. Post-copy (expanded in v53): the VM is already running on the target, but memory pages not yet copied (page faults) are served from the source on demand.

Migration success depends on how fast dirty pages are copied relative to the memory change rate — the slower the change rate, the shorter the stop-and-copy pause.

#Page-Faults from the Source (v53.0)

Since v53.0 (12 July 2026), live migration uses a post-copy approach with page-faults served from the source. In other words: instead of waiting for all memory to be copied (which can take a long time for VMs with large memory), the target starts running the VM right away and requests the pages it needs from the source as page faults occur. The effect: total migration time is much shorter, at the cost of depending on the source's availability during the transition.

#Prerequisites: Shared Storage and Networking

#Shared Storage

Migration moves machine state, not disk contents. That's why both hosts must see the same disk — shared storage (NFS, iSCSI, Ceph RBD, or distributed storage). In our practice, set up NFS:

sudo mkdir -p /srv/vms
sudo mount -t nfs storage.internal:/exports/vms /srv/vms

Both hosts access ubuntu.raw and the snapshot files from the same location. Otherwise, the target can't read the disk when the VM is moved.

#Networking

Source and target must be connected, and ideally on the same network bridge so the guest's MAC and IP remain valid without network reconfiguration:

sudo ip link add name br0 type bridge
sudo ip link set br0 up
sudo ip link set eth0 master br0

If the networks differ, the guest must be reconfigured after migration — which defeats the "no downtime" advantage.

#Running a Migration

#Prepare the Target

The destination host starts the VM in "receive" mode, waiting for the migration:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=/srv/vms/ubuntu.raw \
  --cpus boot=2 \
  --memory size=2G,shared=on \
  --net tap=ch0,ip=192.168.100.1,mac=a8:21:95:80:35:e6 \
  --api-socket /tmp/target.sock

Note that this is a normal VM — the migration later "overwrites" its state. shared=on on both sides is mandatory (just like snapshot).

#Start Migration from the Source

From the source host, call the migration endpoint via the API:

curl --unix-socket /tmp/source.sock -X PUT \
  -H "Content-Type: application/json" \
  --data '{
    "receiver_url": "http://192.168.100.2:5000",
    "protocol": "extended"
  }' \
  http://localhost/api/v1/vm.migrate

• receiver_url: the address the target opens to receive migration data.
• protocol: "extended": uses the expanded live migration protocol — including the post-copy page-faults served by the source in v53.

The target listens on that port using its API socket:

curl --unix-socket /tmp/target.sock -X PUT \
  -H "Content-Type: application/json" \
  --data '{
    "receiver_url": "http://192.168.100.2:5000",
    "protocol": "extended"
  }' \
  http://localhost/api/v1/vm.migrate

When the migration completes, the VM on the source stops automatically and execution continues fully on the target. From the guest's perspective, no process realizes it changed hosts.

#Verify in the Guest

Inside the guest, confirm there's no significant disruption:

uptime -s
cat /proc/uptime

uptime -s still shows the original boot time — if it shows the migration time, the VM was rebooted, not migrated. For a more convincing test, run a continuous ping from the host during migration and observe only millisecond gaps (not tens of seconds).

#Constraints and Limitations

#Not Cross-Version

The same rule as snapshots applies: live migration is not supported between different Cloud Hypervisor versions. Both hosts must run an identical version (or within a guaranteed LTS compatibility window). This is a consequence of the state format being tied to a version. In production, separate your migration "release trains": upgrade all hosts in one wave, not gradually across major versions.

#Device Compatibility

Not every device can be migrated easily:

• VFIO passthrough (episode 8): a physical device can't be "moved" to another host — the target must have an identical, available device. This is often why VMs with passthrough aren't migration-eligible.
• vhost-user: the source daemon doesn't move automatically; the target needs a daemon with the same socket.
• TAP names and MACs: must be consistent between source and target.

Before migrating, check that both hosts' device configurations are truly aligned:

diff <(cat /srv/vms/guest.json) <(ssh target "cat /srv/vms/guest.json")

#Windows Guests

Migrating Windows has additional challenges around timers (TSC) and devices — make sure both hosts use CPUs with identical features, because Windows guests are very sensitive to differences in exposed CPU features.

#Common Pitfalls

• Shared storage isn't really shared: both hosts read different disk copies → corrupted data. Verify by writing from one host and reading from the other.
• Different VMM versions: migration fails or state is incompatible. Always pin versions.
• shared=on on only one side: the target fails to receive memory. Set it on both sides.
• Different bridges: the guest loses connectivity after the move because it's on a different L2 network.
• Forgetting to disable passthrough: physical devices don't migrate.

#Conclusion

Key takeaways:

• Live migration moves a VM between hosts without downtime: pre-copy → stop-and-copy → post-copy.
• v53.0 expands post-copy with page-faults served from the source — much faster migration.
• Shared storage is mandatory: both hosts see the same disk.
• Both hosts' networking must be consistent (bridge, MAC) so the guest doesn't lose connectivity.
• Not supported cross-version; device compatibility (especially passthrough) must be checked.
• Migration is an operation that needs regular testing, not improvisation.

In the next episode, episode 12, we'll dissect virtual I/O: virtio, IOMMU, and security — how virtio ring buffers and packed virtio work, virtio-iommu for paravirtualized IOMMU, and Landlock and seccomp sandboxing that protect the host from the VMM itself. VM security starts from the bottom.