Learn Cloud Hypervisor - Storage: Block, pmem & virtio-fs

#Introduction

After configuring CPU and memory in episode 5, now we handle where data is stored. Storage is one of the most varied areas of Cloud Hypervisor because it offers three mechanisms serving different needs: virtio-blk for classic disks, virtio-pmem for persistent memory, and virtio-fs for sharing the filesystem with the host.

A simple analogy: virtio-blk is like an external hard drive, pmem is like memory that doesn't disappear when power is off (it persists, but is very fast), and virtio-fs is like a shared folder on the network accessible to host and guest alike. The right choice depends on the workload: databases need blk, caching needs pmem, and development tooling needs fs sharing.

#Virtio-blk: The Classic Disk

#Image Formats: raw vs qcow2

Virtio-blk presents an image file as the block device /dev/vda. Supported formats include raw and qcow2. The difference is fundamental:

• raw: a byte-for-byte image. Fastest (no metadata parse overhead) but uses full space according to the file size.
• qcow2: QEMU Copy-on-Write format. File size tracks used data, supports snapshots and backing files, but has a small access overhead.

Conversion is done with qemu-img:

qemu-img convert -O raw ubuntu-24.04.qcow2 ubuntu.raw

Conversely, to save space:

qemu-img convert -O qcow2 ubuntu.raw ubuntu.qcow2
qemu-img info ubuntu.qcow2

qemu-img info shows the virtual size and actual size — useful for understanding how much space is really being used.

#Presenting Disks to the VM

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=ubuntu.raw \
  --disk path=data.qcow2 \
  --cpus boot=4 \
  --memory size=4G

Each --disk becomes one virtio-blk device: ubuntu.raw → /dev/vda, data.qcow2 → /dev/vdb. The path= option can also use a URL (path=http://...) for remote images, and readonly=on for disks that must not be written (a security practice we cover in episode 13).

#Multi-disk: Separating OS and Data

A common production practice: one small disk for the OS, one large disk for data. This makes backup, resize, and disk replacement easier without touching the system.

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --disk path=data.raw \
  --cpus boot=2 \
  --memory size=2G

#Virtio-pmem: Persistent Memory

Virtio-pmem presents a non-volatile memory region (NVDIMM) to the guest. The guest uses it as a very fast filesystem — data survives VM shutdown without disk round-trip overhead.

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --pmem file=metadata.raw,size=1G \
  --cpus boot=2 \
  --memory size=2G

Inside the guest, pmem appears as /dev/pmem0 and can be formatted as a filesystem:

mkfs.ext4 /dev/pmem0
mkdir -p /mnt/pmem
mount -o dax /dev/pmem0 /mnt/pmem

-o dax enables Direct Access: the guest accesses memory pages directly, bypassing the page cache — latency nearly as low as RAM. Typical use cases: database metadata, cache layers, and journals.

#Virtio-fs: Sharing the Filesystem with the Host

#Concept and Host Setup

Virtio-fs lets a host directory be mounted directly into the guest. The data plane is handled by the virtiofsd daemon on the host:

virtiofsd --socket-path /tmp/virtiofs.sock \
  --shared-dir /srv/shared \
  --cache always

Then connect its socket to the VM:

cloud-hypervisor \
  --kernel kernel-vmlinux \
  --disk path=os.raw \
  --fs tag=shared,socket=/tmp/virtiofs.sock \
  --cpus boot=2 \
  --memory size=2G

--fs tag=shared,socket=... connects the vhost-user device to the daemon. Inside the guest, mount that tag:

mount -t virtiofs shared /mnt/shared

#When to Use virtio-fs

• Development: source code and tooling are shared between host and VM without syncing.
• Container runtime: Kata Containers uses virtio-fs to share container rootfs and volumes into the guest VM — we see this in episode 18.
• Not suitable for large databases that need deterministic fsync — for that, stick with virtio-blk.

#Choosing the Right Storage

+------------+----------------+-------------+----------------+
| Tipe       | Kecepatan      | Persistensi | Kasus khas     |
+------------+----------------+-------------+----------------+
| virtio-blk | sedang (disk)  | ya          | OS, data, DB   |
| virtio-pmem| sangat cepat   | ya          | cache, journal |
| virtio-fs  | tinggi (sharing)| ikut host  | dev, container |
+------------+----------------+-------------+----------------+

#Common Pitfalls

• Wrong format: older virtio-blk requires images compatible with O_DIRECT; if boot fails with unexpected end of file, check image integrity with qemu-img check.
• pmem without -o dax: without DAX, pmem becomes just a regular block device and loses its latency advantage.
• Wrong virtiofsd path: --socket-path must match socket= in --fs; make sure the daemon is running before the VM starts.
• Disk "full" without the guest knowing: qcow2 using a sparse file can look "roomy" on the host until a large write happens — monitor physical usage with du -h.

#Conclusion

Key takeaways:

• virtio-blk presents disks (raw/qcow2) as /dev/vda; conversion uses qemu-img.
• Multi-disk setups separate OS and data to ease backup and resize.
• virtio-pmem offers persistent memory with low-latency DAX access.
• virtio-fs shares host directories via virtiofsd and is mounted with mount -t virtiofs.
• Choose storage based on needs: capacity (blk), speed (pmem), or sharing (fs).

In the next episode, episode 7, we'll build networking: TAP, vhost-user, and vDPA — setting up TAP/bridge on the host, connecting a NIC to the guest with --net tap=...,mac=...,ip=..., offloading the data plane to a vhost-user daemon, and getting to know the still-experimental vDPA. This is where your VM starts talking to the outside world.