Learn LXC - Performance Tuning & Large Scale

#Introduction

For the last five episodes you've been building one host with a few containers. Now it's time to think bigger: how do you make one host serve tens to hundreds of containers while keeping performance healthy? In episode 18 we tune storage, cgroups, and resource planning — then measure the overhead. This is the episode that separates a casual homelab from measurable infrastructure.

#Storage Tuning: ZFS/btrfs

#Why the Backend Determines Scale

Back to episode 8: the backend determines snapshot and clone efficiency. At scale, this is no longer a convenience but a matter of survival. Imagine 100 containers and every full snapshot = hundreds of GB of disk. With a copy-on-write backend, snapshots are nearly free.

#ZFS as a Primary Choice

ZFS is a popular choice for large LXC hosts:

• Native snapshots & clones — lxc-snapshot and lxc-copy -s use ZFS's instant mechanism.
• Compression — compression=on saves significant disk space for rootfs that is mostly text files.
• Checksums — detects data corruption early.

zfs create -o compression=on -o atime=off tank/containers
zfs set mountpoint=/var/lib/lxc tank/containers

#btrfs as an Alternative

btrfs has similar strengths (native snapshots, copy-on-write) with simpler management. For hosts without deep ZFS feature needs, btrfs is a solid lightweight choice.

#Tuning cgroups for Density

#Realistic Overcommit Commitments

Episode 7 stressed limits. At scale, you have to calculate total limits ≤ host capacity:

Example resource planning

Host: 16 cores, 64 GB RAM
100 lightweight web containers: 0.1 core + 256 MB each
→ CPU: 10 cores (< 16) OK
→ Memory: 25.6 GB (< 64) OK, the rest for host & buffers

The simple formula: sum all container limits, and make sure to leave room for the host itself (kernel, services, I/O cache).

#Tuning CPU Wisely

For bursty workloads, use lxc.cgroup2.cpu.weight (relative) instead of rigid quotas:

CPU weight for fair sharing

lxc.cgroup2.cpu.weight = 100

cpu.weight (1-10000) sets proportions during contention without wasting CPU when idle — ideal for high density with wave-like load. Combine with cpu.max only for containers that truly need guarantees.

#Memory: Keep Room for Page Cache

The Linux host uses memory for I/O page cache. If container limits consume nearly all RAM, the cache shrinks and disk performance tanks. Leave ~10-20% of host RAM for the kernel and cache.

#Per-Container Resource Planning

The measurement standard before setting limits:

1. Profile the workload — how much peak memory, peak CPU, I/O.
2. Set the limit = peak need + a small buffer (e.g. 1.25x).
3. Stress test (episode 7: dd, CPU loop) — prove the limit applies.
4. Monitor (episode 20) — adjust limits based on real data, not guesses.

Example profiles for specific container types:

Common resource profiles

Light nginx/apache : 256 MB  , 0.2 core
Small database     : 1 GB    , 1 core
CI build           : 4 GB    , 4 core (bursty)

#Scale: Hundreds of Containers per Host

#What Makes Density Possible

• Fast boot — LXC boots in seconds (not minutes like a VM), so provisioning new containers feels instant.
• Nearly zero overhead — no guest kernel; memory is used only for user space processes.
• Snapshot clones — 100 containers can be born from one golden image in minutes (episode 9).

#Boot and Overhead Benchmarks

Measure for yourself to base decisions:

time lxc-start -n c1

ps -o rss= -p $(pgrep -f "lxc-init" | head -1) 2>/dev/null || true

#Real Limits That Often Appear

• File descriptors and the kernel's thread limits (ulimit, fs.file-max).
• The number of veths and bridges on hosts with many networked containers.
• Mass start time: lxc-autostart with lxc.start.delay (episode 10) prevents a thunder herd on host reboot.
• Disk I/O contention — the main reason to use lxc.cgroup2.io.weight (episode 7).

#Closing

Key takeaways:

• ZFS/btrfs make massive snapshots and clones cheap — choose from the start.
• Total cgroup limits must be ≤ host capacity, leaving room for the host and cache.
• cpu.weight for bursty loads; cpu.max for guarantees.
• Profile resources per workload type: measure, set limits, test, monitor.
• Scaling to hundreds of containers is possible because of fast boot and near-zero overhead — manage it with configuration discipline.

In the next episode 19 we'll cover the ecosystem: LXD, Incus & Proxmox — LXD as Canonical's LXC manager with a REST API, clustering, and VMs; Incus as the community fork (2023); and the comparison of low-level LXC vs high-level LXD/Incus vs Docker application containers.