Learn Linux Filesystem - Advanced ZFS (OpenZFS 2.4)

#Introduction

OpenZFS has evolved far beyond its Sun heritage. Release 2.4 (with 2.4.3 as the latest version at the time this series was written) brings features that were only dreams for years: fast deduplication thanks to BRT, stable RAIDZ expansion, and the layout flexibility of AnyRaid.

Episode 14 covers OpenZFS 2.4's features practically, then moves into advanced tuning techniques: special vdevs, zvol, zfs rewrite, and recordsize and ashift. The end goal: you can use ZFS's newest features without stumbling on immature issues.

As usual, experiments run on a lab pool with loopback disk images.

#BRT: Fast Deduplication

#The Classic Dedup Problem

Deduplication on old ZFS used the DDT (dedup table), which consumed huge amounts of RAM and slowed writes dramatically. That's why many admins avoided dedup=on altogether.

#Block Reference Table

BRT (Block Reference Table) is a new implementation that references the same blocks with much smaller overhead. Dedup is now realistic for workloads with lots of repeated data — for example backups and VM templates:

sudo zpool create -O dedup=on labpool mirror /dev/loop0 /dev/loop1

With dedup enabled, duplicate writes only add a reference, not new data. Check the savings with zpool status and zpool get dedupratio:

zpool get dedupratio labpool

The zpool get dedupratio command shows the storage ratio saved — a value above 2 means the data only uses half its logical size.

#RAIDZ Expansion and AnyRaid

#Stable RAIDZ Expansion

Before 2.4, adding a disk to a RAIDZ vdev meant rebuilding the entire pool. RAIDZ expansion lets you add a disk to an existing RAIDZ without destroying the pool, and it's now stable in OpenZFS 2.4:

sudo zpool attach labpool /dev/loop0 /dev/loop2

After attach, ZFS redistributes data automatically. The process takes time depending on data size, and the pool remains usable throughout.

#AnyRaid

AnyRaid introduces flexible layouts for RAIDZ — the number and position of parity don't have to follow a rigid pattern:

sudo zpool create labpool anyraid=d2,p2 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3

With AnyRaid, configurations like two parities spread across different devices can be expressed explicitly. This feature is new — use it after understanding the semantics of the chosen layout.

#Default Quotas and Throttling

#Default Quotas in OpenZFS 2.4

OpenZFS 2.4 adds a property for default quotas on new datasets:

sudo zpool create -O quota=1T labpool /dev/loop0
sudo zfs set quota=500G labpool/data

This property makes datasets created later automatically have a limit — useful for multi-tenant platforms where each user gets a fixed allocation.

#Uncached IO and Allocation Throttling

This release also improves behavior for workloads reading data outside the ARC (uncached IO) and adds allocation throttling to prevent one process from consuming the pool's entire bandwidth. The effect: more stable latency on mixed workloads.

#Special Vdevs and Zvol

#Special Vdevs for Metadata

A special vdev is a fast device (usually NVMe) that holds metadata and small blocks, while large data blocks stay on HDDs:

sudo zpool create labpool mirror /dev/loop0 /dev/loop1 \
  special mirror /dev/loop2 /dev/loop3

Small metadata that usually bottlenecks HDDs moves to the fast device. Monitor special vdev contents with:

zpool status -v labpool
zpool iostat -v labpool

If the special vdev fills up, the pool can stop accepting writes — give it enough space and consider a mirrored special vdev so it doesn't become a single point of failure.

#zvol and zfs rewrite

zvol was covered in episode 6. zfs rewrite is a new utility to physically rewrite blocks — for example, to make data previously written without compression get compressed after the property is changed:

sudo zfs set compression=zstd labpool/data
sudo zfs rewrite labpool/data

zfs rewrite reads and rewrites blocks according to the current properties — useful for applying new compression or recordsize to old data without migration.

#Recordsize and Ashift Tuning

#recordsize

recordsize determines the logical allocation unit size. The rule: match it to the workload's I/O pattern.

• 8K: transactional databases (PostgreSQL/MySQL blocks).
• 128K: the common default, balanced for mixed workloads.
• 1M: large sequential files, media, and archives.

sudo zfs set recordsize=1M labpool/media

#ashift

ashift is the log2 of the device's physical sector size. Matching it to the real sector size prevents misalignment:

• ashift=12: 4KB sectors (almost all modern HDDs/SSDs).
• ashift=13: 8KB, for some NVMe SSDs.

sudo zpool create -o ashift=12 labpool /dev/loop0

Ashift can only be set at pool creation — it can't be changed later. A wrong ashift is a permanent and expensive mistake, so verify sector sizes before zpool create.

#Conclusion

OpenZFS 2.4 closes many of the gaps that made admins hold back from ZFS: dedup is now fast, RAIDZ can be expanded, and layouts can be flexible. Combined with recordsize, ashift, special vdev, and rewrite tuning, ZFS becomes an increasingly hard-to-refuse choice for critical storage.

Key takeaways:

• BRT makes deduplication fast with small RAM overhead.
• RAIDZ expansion is stable in 2.4 — add disks without rebuilding the pool.
• AnyRaid gives flexible parity layouts.
• Special vdevs place metadata on fast devices; zvol remains a mainstay.
• zfs rewrite applies new properties to old blocks.
• Match recordsize to the workload; ashift can only be set at create.

In the next episode, episode 15, we cover filesystems for containers and Kubernetes — OverlayFS and rootless containers, then CSIs like Rook-Ceph and TopoLVM, local persistent volumes, and filesystem semantics for stateful workloads. You'll connect everything learned to the container world.