Learn Firecracker - Storage: Virtio-block & Virtio-fs

#Introduction

In episode 5 your microVM already had networking. Episode 6 completes the I/O foundation with two things: storage — how data lives and flows to the guest — and metadata — how the host talks to the guest without fragile networking.

Why is this episode important? Storage determines cost and speed: a read-only rootfs enables page cache sharing that saves host memory massively, while virtio-fs allows sharing directories between host and guest without copying files. And MMDS is Firecracker's answer to the classic orchestration question: how does a VM know its own configuration and secrets?

#Virtio-block: Attaching the Rootfs and Additional Disks

PUT /drives is the door to all block storage. In episode 4 we attached the rootfs; here we add a second disk — for example, application data storage:

dd if=/dev/zero of=~/fc-demo/data.ext4 bs=1M count=1024
mkfs.ext4 ~/fc-demo/data.ext4

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/drives/data \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "drive_id": "data",
    "path_on_host": "/home/user/fc-demo/data.ext4",
    "is_root_device": false,
    "is_read_only": false
  }'

After boot, the guest will see a new block device (usually /dev/vda for the rootfs, /dev/vdb for the second disk), which you can format and mount from inside the guest:

mkfs.ext4 /dev/vdb
mkdir -p /mnt/data
mount /dev/vdb /mnt/data

#Read-Only vs Read-Write: An Architectural Choice

The is_read_only field isn't just a technical flag — it's an architectural decision with major consequences:

Read-only rootfs (is_read_only: true):

• The rootfs never changes → file pages can be shared between microVMs on the host (page cache sharing). Ten microVMs with the same rootfs can share the same blocks in host memory, saving hundreds of MiB.
• Great for ephemeral workloads: every VM starts from the same image, with state stored elsewhere (data drive, snapshot, or external storage).

Read-write rootfs (is_read_only: false):

• The guest can write freely; convenient for development, but every VM consumes host memory for its own page cache.
• Cannot be shared; suits VMs that genuinely have their own persistent data.

The common production pattern: read-only rootfs + read-write data drive. State that must survive is kept separate from the operating system.

#Virtio-fs: Sharing Host Directories → Guest

Sometimes you don't need a new disk — you need to share a directory with the host directly: source code for development, caches, or sockets. Virtio-fs is Firecracker's mechanism for this, with far lower overhead than 9p or NFS.

Virtio-fs in Firecracker uses the virtiofsd daemon on the host side to serve filesystem requests from the guest:

virtiofsd \
  --socket-path /tmp/virtiofs.sock \
  --shared-dir /home/user/shared \
  --cache always &

Then register the virtio-fs device to the microVM before boot:

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/fs/fs0 \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "fs_id": "fs0",
    "virtio_fs": {
      "cache_type": "Always",
      "source_dir": "/home/user/shared",
      "tag": "shared"
    }
  }'

After boot, mount from inside the guest:

mkdir -p /mnt/shared
mount -t virtiofs shared /mnt/shared
ls /mnt/shared

Note: source_dir still points to the host, and the tag (shared) is the name the guest uses when mounting. With Always caching, the guest reads data directly from the host's page cache — performance nearly on par with a local filesystem.

#MMDS: The Metadata Service

MicroVMs often need to know who they are: IP address, hostname, credentials, or the config server endpoint. Firecracker provides MMDS (MicroVM Metadata Service) — a metadata server exposed to the guest at 169.254.169.254, following the same pattern as the EC2 metadata service.

The host writes metadata via the API:

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/mmds \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "latest": {
      "meta-data": {
        "instance-id": "i-1234567890",
        "local-ipv4": "172.16.0.2"
      },
      "credentials": {
        "token": "supersecret"
      }
    }
  }'

The guest reads it over HTTP to 169.254.169.254:

curl http://169.254.169.254/latest/meta-data/instance-id
curl http://169.254.169.254/latest/meta-data/local-ipv4

MMDS advantages:

• No IP networking: this communication path needs no route to the host; the link-local address is automatically available.
• Locked from outside: 169.254.169.254 is not routable from other networks, so other VMs can't read the metadata.
• Available at boot: metadata can be read as soon as the network is up — no need to wait for a config server.

This is the pattern production serverless uses: applications inside the microVM read their own configuration from MMDS, rather than from files baked into the image or leaked environment variables.

#MMDS on Dedicated Networks

MMDS works through a path separate from the network data plane — Firecracker intercepts requests to 169.254.169.254 at the device level. That's why it can be disabled or directed to a specific interface via network_interfaces in the MMDS config. For most cases the default is enough: enable MMDS with PUT /mmds, and the guest reads via 169.254.169.254.

Best practice for secrets: store only a token or reference in MMDS, and pull the real values over TLS to Vault or Secrets Manager. MMDS is a fast delivery path, not a substitute for a secret store — but without MMDS, the guest wouldn't even know which secret store to contact.

#Common Pitfalls

• is_root_device double true: two drives claiming to be root → boot fails. Only one may be root.
• qcow2 image files: Firecracker only supports raw format (qemu-img convert -O raw to convert). Attaching qcow2 directly will error.
• Virtio-fs mount doesn't work: make sure the guest kernel has virtio-fs support (CONFIG_VIRTIO_FS), and virtiofsd is alive before InstanceStart.
• MMDS not showing up: make sure the guest uses a supporting kernel, and requests go to 169.254.169.254 (not some other IP). Check with curl -v to see whether the response comes from Firecracker.
• Data drive without a format: a new drive is empty; the guest must format it before mounting.

#Closing

The key takeaways:

• PUT /drives attaches block devices; only one may be is_root_device.
• A read-only rootfs enables page cache sharing between microVMs — massive memory savings.
• Additional disks (e.g. /dev/vdb) are used for persistent state.
• Virtio-fs shares host→guest directories with high-performance Always caching.
• MMDS (169.254.169.254) is a safe host→guest metadata path that stays offline from the network.

In the next episode 7 we'll lock the microVM away from the outside world: Jailer & Security Isolation — wrapping Firecracker in namespaces, cgroups, seccomp, a read-only rootfs, and a non-root user, and understanding the --id, --chroot-base-dir, and --exec-file flags that form the last line of defense if the hypervisor is breached.