Learn Firecracker - Networking: Virtio-net & TAP

#Introduction

In episode 4 your microVM booted successfully — but it's still locked away from the outside world. Episode 5 opens the gate: networking. We'll create a TAP interface on the host, connect it to the microVM via PUT /network-interfaces, assign an IP address inside the guest, and wrap everything with a rate limiter.

Why is this episode important? Almost every Firecracker workload needs networking — from running servers inside the microVM, accessing registries, to building meshes between microVMs. And the rate limiter we learn about here isn't an afterthought: it's the mechanism that lets a single host serve thousands of tenants without a single VM monopolizing bandwidth.

#The Concept: How Packets Flow

The basic Firecracker networking pattern:

The host creates a TAP interface — a virtual device that, in the host kernel's eyes, behaves like a real Ethernet interface.
Firecracker attaches the microVM's virtio-net device to this TAP via PUT /network-interfaces.
Every packet from the guest exits through virtio-net → written to the TAP → processed by the host kernel (routed, bridged, or filtered).
Incoming packets flow in the opposite direction.

The TAP lives on the host side; virtio-net is the door on the guest side. Firecracker bridges the two, moving frames between these two worlds. Because the TAP is a real interface in the host kernel, all standard Linux tooling — bridge, iptables, nftables, tc — can be applied at this point.

#Creating a TAP Interface on the Host

Create a TAP with ip:

Create a TAP interface

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up

ip tuntap add ... mode tap creates the tap0 device. Bringing it up with ip link set tap0 up is essential — a down TAP won't process packets. Check the result:

View the TAP interface

ip link show tap0

Note: at this stage the TAP has no IP address — it works at layer 2. For the guest to reach the host and the internet, you'll need to bridge or add routes on the host. We build the complete pattern (bridge and isolation) in episode 13; for this experiment, connecting the microVM to the TAP and assigning an IP inside the guest is enough.

Warning

Creating a TAP requires CAP_NET_ADMIN. In production, this is isolated in a dedicated network namespace per VM (covered in episode 13). For a lab, plain sudo as above is fine.

#PUT /network-interfaces

Now attach the TAP to the microVM. This is done before InstanceStart:

Attach a network interface

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/network-interfaces/eth0 \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "iface_id": "eth0",
    "host_dev_name": "tap0",
    "guest_mac": "06:00:00:00:00:01"
  }'

Important fields:

iface_id — interface identity inside the guest (free choice, usually eth0).
host_dev_name — the name of the TAP on the host to connect to.
guest_mac — the guest's MAC address. Use an address with the local bit set (06:00:...) to avoid conflicts with other interfaces on the host.

Notice that we do not assign an IP address in this payload — the IP address is the guest's business, not the Firecracker API's. After InstanceStart, configure the IP inside the guest with ip addr add 172.16.0.2/24 dev eth0 and ip link set eth0 up. The Firecracker API only connects the pipe; IP configuration stays in your hands.

#Testing Connectivity

After the guest boots and the IP is set, test from the guest to the host and back. From inside the guest:

Check the IP inside the guest

ip addr show eth0
ping -c 3 172.16.0.1

If the ping to the host succeeds, the virtio-net → TAP pipe works. From the host side, you can verify that frames really pass through the TAP:

Check traffic on the TAP

sudo tcpdump -i tap0 -c 10

tcpdump -i tap0 displays the passing packets — visual proof that the microVM's network really flows through the host interface.

#Rate Limiter: Token Bucket for Multi-Tenancy

Firecracker's rate limiter applies the token bucket algorithm on two dimensions per device: bandwidth (bytes per second) and ops (packets per second). The concept: a bucket holds tokens; each byte/packet drains one token; the bucket is refilled every refill_time milliseconds with size tokens.

Network interface with a rate limiter

curl --unix-socket /tmp/firecracker.sock -i \
  -X PUT http://localhost/network-interfaces/eth0 \
  -H 'Accept: application/json' -H 'Content-Type: application/json' \
  -d '{
    "iface_id": "eth0",
    "host_dev_name": "tap0",
    "rx_rate_limiter": {
      "bandwidth": { "size": 1048576, "one_time_burst": 1048576, "refill_time": 100 },
      "ops": { "size": 1000, "one_time_burst": 1000, "refill_time": 100 }
    },
    "tx_rate_limiter": {
      "bandwidth": { "size": 1048576, "refill_time": 100 }
    }
  }'

Reading this payload:

rx_rate_limiter vs tx_rate_limiter — direction from the device's point of view (rx = from guest to host / incoming packets to the microVM).
bandwidth — limits bytes/second. size = bucket capacity (maximum burst per refill), refill_time = refill interval in milliseconds, one_time_burst = one-time extra tokens at the start.
ops — limits packets/second, protecting the host CPU from floods of small packets.

With a rate limiter, you can promise bandwidth SLAs per VM contractually: one VM may draw 1 MB/second, its neighbor may too — and neither can exceed its limit even if it tries.

Important

A token bucket allows short bursts above the average rate (that's what size and one_time_burst are for), then smooths the flow afterward. This is good behavior for real workloads — network traffic is almost never perfectly flat — but remember that sustained bursts will still be capped by the refill rate.

#Virtio-net: Modes and Basic Tuning

Firecracker supports several modes for virtio-net that affect how packets are processed:

Default (virtio-mmio) — the device is memory-mapped to MMIO, with the standard guest driver.
virtio-pci — a PCI device; enabled via the kernel with PCI enabled. Fits well with the modern Linux ecosystem.
Tap mode — uses a TAP directly (what we used above), enabling full host-side filtering.

For high-throughput workloads, consider tuning on the guest and host: virtio ring sizes, TSO/GSO offload, and MTU size. We cover detailed tuning in episode 13; here it's enough to understand that everything starts from a healthy TAP connection.

#Common Pitfalls

TAP not up: packets won't flow; always ip link set tap0 up.
Non-unique MAC: if two microVMs use the same MAC on the same bridge, frames will collide. Use the local bit and different values.
IP set on the host, not the guest: the Firecracker API doesn't set IPs — the IP must be set inside the guest or via cloud-init/MMDS (episode 6).
Rate limiter size: 0: an empty bucket means no tokens → traffic stops. Always give size a positive value.
Forgetting namespaces: the TAP is created in the host namespace, but the guest is in the same process — if the TAP is in another namespace, the interface won't be visible.

#Closing

The key takeaways:

TAP interface on the host + virtio-net in the guest = the microVM's network pipe.
ip tuntap add dev tap0 mode tap then ip link set tap0 up.
PUT /network-interfaces connects the TAP; IPs are configured inside the guest.
The token bucket rate limiter caps bandwidth and ops per direction per device.
tcpdump -i tap0 is your window for seeing microVM packets on the host.
Fair multi-tenancy starts with a rate limiter installed from the very beginning.

In the next episode 6 we'll organize storage and metadata: Storage — Virtio-block & Virtio-fs — attaching the rootfs and additional disks, understanding read-only vs read-write drives, sharing host directories to the guest efficiently with virtio-fs, and introducing MMDS for distributing metadata from the host to the guest.