Learn Linux Filesystem - Filesystems for Containers & Kubernetes

#Introduction

Containers don't store data themselves — images are read-only, and every write needs a new layer. That's why understanding filesystems is the foundation for containers and Kubernetes. Episode 15 connects the filesystem concepts you've learned with the modern container world.

We start with OverlayFS — the copy-on-write engine at the heart of container image management. Then we discuss volumes and snapshots, rootless containers, and move into Kubernetes: CSI with Rook-Ceph and TopoLVM, local persistent volumes, and filesystem semantics for stateful workloads.

By the end of this episode, you'll understand why storage in Kubernetes isn't just "plugging in a disk", but an architectural decision.

#OverlayFS: The Heart of Containers

#The Overlay Principle

OverlayFS is a filesystem that merges several directories (layers) into a single view. The lower layers are read-only (the image), and the upper layer holds changes (the container). When a container writes a file, OverlayFS copies the changed block to the upper layer — Copy-on-Write, exactly like btrfs and ZFS.

See how the container runtime uses overlay:

mount | grep overlay
df -hT /var/lib/docker

The mount | grep overlay output shows lines with type overlay and the lowerdir, upperdir, and workdir options — those are the three layers forming a container filesystem.

#Commands to Analyze

Inspect an image and its layers with a container tool:

docker image inspect ubuntu:24.04 --format '{{.RootFS.Layers}}'

Each layer is a delta of the previous one. That's why container images are efficient: two containers from the same image share all the read-only layers, with only the upper layer differing per container.

#Volumes and Snapshots

#Volumes vs the Container Filesystem

Writes inside a container are lost when the container is deleted. Volumes store data outside the container's layers — and this is where the host filesystem plays its role. A container volume is essentially a directory on the host filesystem (usually btrfs or ZFS if that storage driver is installed).

#Container Image Snapshots

With the btrfs or ZFS driver, container image snapshots are nearly free:

sudo zfs snapshot docker-pool/ubuntu@v1
sudo zfs clone docker-pool/ubuntu@v1 docker-pool/ubuntu-staging

This snapshot + clone pattern makes staging images or environment rollbacks without duplicating storage — the same technique as episode 10, now applied to container images.

#Rootless Containers

#What Is Rootless

Rootless containers run the entire userspace without root, reducing the impact of runtime vulnerabilities. As a consequence, filesystems are mounted in the user namespace — overlay mounts under /etc/subuid and fuse-overlayfs for certain cases:

dockerd-rootless-setuptool.sh install
systemctl --user start docker

Check the resulting mounts:

mount | grep -E "overlay|fuse" | head

fuse-overlayfs is a FUSE-based fallback when the kernel doesn't allow regular overlay in a user namespace. The trade-off: slightly lower performance than native kernel overlay.

#Kubernetes: CSI and Persistent Volumes

#CSI: Container Storage Interface

Kubernetes separates storage management through CSI — drivers that run as pods and handle provisioning, mounting, and snapshots. Two popular examples:

• Rook-Ceph: manages Ceph (RBD block and CephFS) inside the cluster — distributed storage born from a filesystem that handles its own redundancy.
• TopoLVM: uses LVM on nodes to provision local logical volumes — storage leveraging the LVM snapshot concept we covered in episode 9.

View the available storage classes:

kubectl get storageclass
kubectl get pv,pvc

The kubectl get pv,pvc output shows PersistentVolumes and PersistentVolumeClaims — abstractions separating storage consumers (pods) from providers (CSI).

#Local Persistent Volumes

For workloads that must stick to a particular node (for example a database with large data), a local persistent volume uses a local disk or filesystem without network replication:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-fast
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

volumeBindingMode: WaitForFirstConsumer defers binding until the pod is scheduled — ensuring the PV is created on the same node as the pod.

#Filesystem Semantics for Stateful Workloads

#When fsync Means Something

Stateful workloads (databases) depend heavily on filesystem semantics: fsync, O_DIRECT, and crash consistency. Here are some considerations:

• Use a filesystem with honest fsync — ZFS with ZIL or ext4 with data=ordered.
• Avoid storage drivers that silently absorb fsync without real durability.
• Understand the dataSource of a PVC for cloning a volume from a snapshot.

Simulate creating a PVC from a snapshot with the Rook-Ceph CSI:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-pg
spec:
  storageClassName: rook-ceph-block
  dataSource:
    name: pg-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

This pattern translates filesystem snapshots (episode 9) into the Kubernetes world: restoring a database from a point in time without restoring the whole cluster.

#Conclusion

Containers and Kubernetes don't replace filesystems — they wrap them in new abstractions. OverlayFS gives efficient images, volumes pass data through to the host filesystem, and CSI translates filesystem semantics into the Kubernetes API. The concepts you learned in episodes 0-14 remain relevant, just with a new layer on top.

Key takeaways:

• OverlayFS is COW at the directory level: lowerdir image, upperdir writes, workdir transactions.
• Image snapshots are efficient with the btrfs or ZFS driver.
• Rootless containers use restricted mount namespaces and sometimes fuse-overlayfs.
• CSI separates provisioning, mounting, and snapshots from the workload.
• Rook-Ceph provides distributed storage; TopoLVM provides local LVM.
• fsync and snapshot filesystem semantics reach all the way into stateful pods.

In the next episode, episode 16, we cover performance tuning — mount options noatime/discard/compress, recordsize, I/O schedulers mq-deadline and bfq, page cache tuning, then benchmarking with fio and dd. You'll learn to measure and optimize filesystems scientifically.