Learn Firecracker - Snapshot & Restore

#Introduction

A 125 ms boot feels fast — until you imagine thousands of cold starts per second. Episode 9 answers the open question: how do you bring a "warm" microVM back to life in milliseconds, not seconds? The answer is snapshotting: freeze a running microVM, save its state, and restore it exactly where it left off.

Why is this episode important? Snapshotting is the technology that transforms serverless economics. Instant cold starts, scale-to-zero, and suspend/resume up to 8 hours — all the features used by AWS Lambda MicroVMs — stand on Firecracker snapshots. Understanding snapshots means understanding how modern serverless platforms really work.

#The Concept: Freezing Time

A running microVM has a lot of state: memory contents, vCPU registers, and device conditions. A snapshot captures all of it into files, so that state can be restored in another Firecracker process on the same or a different host.

Two files are produced:

• Snapshot file — device state, vCPUs, and VM configuration (not memory contents).
• Memory file — guest memory contents, written as a file on the host.

Restore is the reverse: a new Firecracker process reads both files, loads the state, and resumes execution from the frozen point — as if it never stopped. The guest notices nothing.

#Creating a Snapshot

Before snapshotting, the microVM must be paused so the state is stable. The full flow:

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/actions \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{ "action_type": "Pause" }'

Then create the snapshot:

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/snapshot/create \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "snapshot_type": "Full",
    "snapshot_path": "/snapshot/base",
    "mem_file_path": "/snapshot/base.mem"
  }'

Important fields:

• snapshot_type — Full (entire memory) or Diff (only pages changed since the base; episode 19).
• snapshot_path — the VM state file.
• mem_file_path — the memory contents file.

After the snapshot is created, resume or stop the VM:

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/actions \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{ "action_type": "Resume" }'

#Restore: Bringing It Back

Restore happens in a new Firecracker process (not by continuing the old process). Start a new process, then send PUT /snapshot/load:

firecracker --api-sock /tmp/firecracker-restored.sock --config-file /dev/null &

curl --unix-socket /tmp/firecracker-restored.sock -i \
  -X PUT http://localhost/snapshot/load \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "snapshot_path": "/snapshot/base",
    "mem_backend": {
      "backend_type": "File",
      "path": "/snapshot/base.mem"
    },
    "enable_diff_snapshots": false,
    "resume_vm": false
  }'

Then resume:

curl --unix-socket /tmp/firecracker-restored.sock -i \
  -X PUT http://localhost/actions \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{ "action_type": "Resume" }'

Notice: resume_vm: false at load time gives you the chance to reconfigure networking (MAC, rate limiter) before execution continues — a mandatory step if the VM is restored on a different host or network.

#The Key to CPU Compatibility: cpu_template

A snapshot is captured on one host and restored on another. The problem: hosts can have different CPU microarchitectures, and instructions saved in vCPU registers may not be valid on a new CPU. The solution is cpu_template in machine-config: Firecracker hides microarchitectural differences by exposing the guest CPU as a standard model (e.g. T2, T2S, T3).

The practical consequence: the host producing the snapshot and the host restoring it must have the same cpu_template and compatible vcpu_count/memory. This is why serverless platforms apply snapshots to hosts of a homogeneous type — and why you should set the template from the start, not after a snapshot exists.

#Real-World Patterns: Instant Cold Start, Scale-to-Zero, Suspend/Resume

Snapshots open three patterns that change system design:

#Instant Cold Start

Instead of booting kernel + init + application (hundreds of milliseconds to seconds), restore a snapshot of an already-initialized application: open a process, load state, resume — the application is immediately ready in milliseconds. This is the pattern serverless uses to bring functions up without waiting for a boot.

#Scale-to-Zero

When there's no traffic, the VM can be stopped entirely — no memory cost, no CPU. When a request arrives, a snapshot is restored in milliseconds. Full scale-to-zero cost efficiency is achieved because restarting is no longer expensive.

#Suspend/Resume

A VM is frozen and its state stored for a long duration — AWS Lambda MicroVMs allow suspend up to 8 hours, then resume exactly where it stopped. This lets workloads that need to keep connections and state "alive" stay that way even while inactive.

#The Full Snapshot Workflow in an Orchestrator

Here's the pattern microVM orchestrators (firecracker-containerd, Flintlock, Lambda) use in practice:

1. Warm pool — several microVMs are booted from a base image with their applications initialized, then paused.
2. Base snapshot — create a Full snapshot from a ready VM.
3. Restore on demand — when a request arrives, restore the snapshot on a matching host (same cpu_template), set up networking, then resume.
4. Diff snapshot (optional) — after use, save only the changes since the base for auditing or replay.

This pattern removes boot time from the critical path: applications are restored, not booted.

#Common Pitfalls

• Read-write drives: VMs with read-write drives can't be snapshotted — use a read-only rootfs.
• Inconsistent cpu_template: restoring on a host with a different template → a crash candidate. Standardize the template.
• Forgetting to resume: after load, the VM stays paused until Resume is sent.
• Snapshotting on the same host while the VM is running: the snapshot must come from a paused VM, and restore happens in a separate process — not the same one.
• Snapshot files not synced: make sure the snapshot and mem files are copied intact to the destination host before loading.
• Networking not reset after load: MAC and rate limiter must be reconfigured before resuming if you move hosts.

#Closing

The key takeaways:

• A snapshot = complete VM state (vCPUs, memory, devices) in two files.
• Pause → PUT /snapshot/create → (optionally) resume/stop.
• Restore in a new Firecracker process: PUT /snapshot/load → Resume.
• A consistent cpu_template is a prerequisite for snapshot migration between hosts.
• Full vs Diff snapshots determine storage cost and speed.
• Production pattern: warm pool → base snapshot → restore on demand → resume.

In the next episode 10 we'll tune resources so microVMs coexist healthily: Balloon, Entropy & Resource Control — using the balloon device to reclaim memory from the guest, reading its statistics, ensuring entropy via virtio-rng, and enforcing CPU/memory limits with cgroup v2.