Learn Firecracker - Testing & Performance Benchmark

#Introduction

Firecracker claims ~125 ms boot and < 5 MiB overhead. Claims like that must be tested — on your host, with your workloads. Episode 16 turns you from a Firecracker user into an engineer who measures Firecracker: proper benchmark methodology, the numbers to measure, and how to validate platform resilience under pressure.

Why is this episode important? Every production decision — how many microVMs per host, when to use snapshots, how big the overhead is — must be based on numbers, not intuition. This episode gives you the framework for getting those numbers in a defensible way.

#Measuring Boot Time: Cold vs Snapshot

Boot time is the most iconic metric. What you must distinguish clearly:

• Cold boot: from zero — the Firecracker process starts, the kernel loads, init runs, the application is ready.
• Snapshot restore: from an already-initialized snapshot — a new process, load state, resume.

A practical measuring method — use timestamps inside the guest and compare with host time:

# In the guest: note the kernel load time (from dmesg) and when the app is ready
cat /proc/uptime

For precise measurement from the host side, measure the difference between InstanceStart and when the guest output is "ready":

time curl --unix-socket /tmp/firecracker.sock -X PUT http://localhost/actions \
  -d '{ "action_type": "InstanceStart" }'

The correct interpretation:

• Cold boot is expected in the 100-300 ms range for a lightweight image — anything above 1 second signals an image that's too fat or a kernel doing too much work.
• Snapshot restore is expected to be far below cold boot — tens of milliseconds for a small VM — which is what makes serverless cold starts feel instant.

Watch out for variability: measure 10+ times, take the median and p95, not a single run. Boot time is affected by host load, NUMA, and page cache.

#Memory Overhead

Firecracker's overhead (< 5 MiB per VM above guest memory) can be measured with cgroup or /proc:

ps -o pid,rss,comm -C firecracker

Read it correctly:

• RSS is the memory used by the VMM process — on top of the memory allocated to the guest (mem_size_mib).
• Guest memory is separate: mem_size_mib + overhead = total VM footprint.

For a thorough measurement, also add the page cache from a shared rootfs — this is why a read-only rootfs lets 10 VMs with the same image use the same page cache. An honest measurement: measure the host before and after adding N VMs, divide the difference by N — not just summing RSS.

#IOPS and Bandwidth

Storage and networking need throughput numbers. Two standard tools:

fio --name=test --rw=randread --size=64m --bs=4k \
  --direct=1 --numjobs=4 --runtime=30 --group_reporting

iperf3 -s  # on one guest
iperf3 -c <ip-other-guest>  # on another guest

Interpretation: storage IOPS and network bandwidth are measured after the rate limiter and cgroup are active — those are the numbers relevant to production. Also compare numbers without the rate limiter to understand the available margin. Measure multiple directions (rx/tx) and multiple block sizes; a single number doesn't tell the whole story.

#Load Testing: Thousands of MicroVMs per Host

The biggest architectural decision — how many microVMs per host — must be verified with real load testing. A healthy methodology:

1. Scale gradually: start at 50 VMs, go up to 500, then to the target (hundreds to thousands depending on host capacity).
2. Monitor host metrics while scaling up: CPU, memory, page cache, load average, and boot time.
3. Find the saturation point: when boot time degrades significantly or the host starts thrashing — that's your realistic capacity.

A simple script for scale testing using the jailer + API:

for i in $(seq 1 200); do
  jailer --id "vm-$i" --exec-file /usr/local/bin/firecracker \
    --uid 123 --gid 100 --chroot-base-dir /srv/jailer &
done

Watch the boot time curve against the number of VMs. Healthy density keeps boot stable; too-high density makes boot slow down exponentially. This data is what determines the capacity numbers in episode 19.

#Failure Injection: Testing Resilience

A healthy platform isn't visible when things are normal — it's visible when things fail. Failure injection simulates failures to validate that the system recovers:

• Force-kill the Firecracker process — does the orchestrator detect, restart, and clean up resources (TAP, chroot)?
• Take down one host — is the workload moved from stored snapshots?
• Fill the snapshot disk — does the snapshot pipeline fail with a clear error?
• Slow down I/O (e.g. with tc netem or throttling) — does timeout handling work?

kill -9 <firecracker-pid>

Observe: is the socket cleaned up? Is the TAP released? Is the cgroup empty? A good system leaves little garbage behind; a bad system accumulates leaked resources until the host runs out.

#Testing: Unit, Integration, and E2E

For your own platform on top of Firecracker, three levels of testing:

• Unit tests — internal logic (configuration, image mapping).
• Integration tests — Firecracker + jailer + networking together; boot a VM and run assertions inside it.
• E2E — the full user flow: build image → create VM → boot → run workload → snapshot → restore → cleanup.

A key pattern for stable tests: always clean up resources at the end of a test (remove chroot, TAP, cgroup), and don't run parallel tests that share the same socket or port. Good Firecracker tests can be repeated thousands of times without degrading the host.

#Common Pitfalls

• Measuring once: boot time is noisy; measure many times and take the distribution.
• Benchmarking without rate limiter/cgroup: numbers that don't represent production.
• Ignoring page cache: RSS alone isn't enough; measure the total footprint via host difference.
• Load tests changing many variables: the results become uninterpretable.
• Not cleaning up resources during tests: the host runs out of TAPs/sockets after thousands of iterations.
• Failure tests covering only one scenario: test kill, crash, and disk errors together.

#Closing

The key takeaways:

• Cold boot ~100-300 ms; snapshot restore is far faster — measure both separately.
• Overhead < 5 MiB per VM above guest memory; measure by total difference, not RSS alone.
• IOPS and bandwidth are measured after the rate limiter and cgroup are active.
• Gradual load testing determines the real density per host.
• Failure injection validates that the system recovers — not just that it runs when healthy.
• Layered testing: unit, integration, E2E, with disciplined resource cleanup.

In the next episode 17 we'll look at the latest features: Firecracker 1.16 & The Latest Features — the CVE-2026-5747 fix in v1.16.1, VMClock device support, testing on Intel Granite Rapids, snapshot improvements, experimental VFIO device passthrough, and the rust-vmm integration direction.