Learn Firecracker - Observability & Troubleshooting

#Introduction

A healthy system at the thousands-of-microVMs scale is a system that's visible. Episode 20 covers observability and troubleshooting: how Firecracker reports its condition, and how you debug failures — from boot errors to failed snapshot restores.

Why is this episode important? In the container world, debugging has a mature ecosystem. In the microVM world, it all has to be built on top of Firecracker's logs, metrics, and serial console — and most serverless platform incidents are rooted in debugging that took too long. Mastering observability means compressing recovery time from hours to minutes.

#Firecracker Logs

Firecracker logs go to stderr (or a redirected file). Level and format can be configured via the API:

Set the log level

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/logger \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "level": "Info",
    "log_path": "/logs/firecracker.log"
  }'

Key points about reading logs:

Levels: Error for errors only, Info for the normal lifecycle, Debug for deep troubleshooting. Raise the level only while debugging — Debug is noisy and expensive.
Ratelimit: Firecracker limits log rate so a single VM can't flood storage. Configure log_path, level, and the ratelimit interval via PUT /logger.
Format: every line carries a timestamp, level, and context — ideal for parsing into a centralized log system.

An example of the lines you'll see while debugging:

A healthy habit: direct all VMs' logs to a central system (via log_path to a file, then tailed by an agent), and don't leave logs on ephemeral hosts.

#Metrics: The /metrics API

Firecracker exposes metrics in Prometheus format at GET /metrics — lifecycle data you can scrape and alert on. Grab an example:

Fetch metrics

curl --unix-socket /tmp/firecracker.sock http://localhost/metrics

Available metrics include:

VM lifecycle: fc_api_requests, vmm_* states, start_time/end_time.
Devices: packet counts and errors on virtio-net, I/O on virtio-block, balloon statistics.
API: successful and failed request counts per endpoint.

Production usage patterns:

Scrape periodically into Prometheus from all hosts, with per-host and per-VM labels.
Alert on danger signals: rising API errors, device errors, excessive VM restarts.
Trend for capacity: boot time and restore time metrics aggregated per host.

Metrics are the objective data answering "is the system healthy" — the counterpart to logs answering "why isn't it."

#Tracing

To trace a request flow end-to-end (from incoming request → orchestrator → restore → resume), add tracing context to the Firecracker API:

Set tracing context

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/metrics/config \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{ "metrics_path": "/metrics", "res_metrics": true }'

Tracing here gives you correlation between API requests and Firecracker's internal operations — useful when you want to know why a restore is slow: is it snapshot disk I/O, CPU contention, or API queueing? Combine it with orchestrator tracing for a complete picture.

Tip

Start observability from three minimum sources: logs (what happened), metrics (how bad), and health checks (is the VM alive). Add tracing when you need to find causes across components. Don't build expensive tracing before logs and metrics are running.

#Serial Console: A Window into the Guest

When the guest is in trouble — boot hang, kernel panic, app not running — the serial console is the most honest window:

Redirect Firecracker's output to a pty (episode 4), or
Access directly from the running process.

Inside the console you can see: kernel boot logs, where the process stopped, and what the guest says when it fails. For a guest with getty on serial, you can even log in and investigate deeper. The serial console is irreplaceable for boot debugging — no dashboard can substitute for it.

#Common Troubleshooting

#KVM Errors

Symptom: Error: KVM not available at start.
Check order: does /dev/kvm exist? What does kvm-ok say? Is grep -cE 'vmx|svm' /proc/cpuinfo non-zero? Does the kvm group include the non-root user?
Solution: enable virtualization in the BIOS, or run on a host with nested virtualization.

#Snapshot Restore Failures

Symptom: PUT /snapshot/load errors.
First check: cpu_template and vcpu_count between the snapshot creator and the destination host. Mismatch is the number one cause.
Second check: the drives used are consistent (read-only, same image).
Third check: snapshot and mem files are complete and uncorrupted (match hashes when transferring).

#Boot Never Finishes

Symptom: the process runs, but the guest produces no output.
Check: is console=ttyS0 in the boot args? Is the kernel uncompressed? Does the rootfs have an init?

#API Rejections

Symptom: 409 Conflict after start.
Meaning: configuration is locked after boot (episode 4) — create a new VM or use a snapshot.

#Last-Resort Tools: `firecracker --help` and gdb

When logs aren't enough, two final tools:

See all firecracker options

firecracker --help

firecracker --help prints every process flag — including ones not in the main documentation. It's the source of truth for what the process can be configured with.

For the hardest crash cases, gdb can attach to the Firecracker process to inspect a backtrace:

Attach gdb to the Firecracker process

sudo gdb -p $(pgrep -f "firecracker" | head -1)

Inside gdb, inspect threads and backtraces to understand the crash point. This is the final, deepest step — usually needed when facing an unknown VMM bug.

#Common Pitfalls

Debug-level logs in production: noisy and expensive; switch back to Info/Error when done.
Metrics without alerts: data without action is just an archive; set alerts on critical signals.
Relying on host logs for the guest: guest boot logs are only visible in the serial console — don't miss them.
Snapshot restore without template checks: the most common cause of restore failure.
Forgetting to route logs to a central system: when a VM dies, logs on the ephemeral host die with it.

#Closing

The key takeaways:

Logs via PUT /logger: configurable level, log_path, and ratelimit.
Prometheus metrics at GET /metrics for alerting and capacity trends.
The serial console is a window into the guest — mandatory for boot debugging.
KVM errors, failed restores, and boot hangs have clear check orders.
firecracker --help and gdb are the last-resort tools when logs aren't enough.

In the next episode 21 we'll look behind the scenes of the project: The rust-vmm Ecosystem & Collaboration — the shared crates like kvm-ioctls, vm-memory, and virtio-devices used by Firecracker, Cloud Hypervisor, and crosvm; the monorepo evolution and RISC-V support; and the GitHub community, Slack, roadmap, and Firecracker governance.