Learn RAID - ZFS RAID Advanced (BRT, Tiering)

#Introduction

Episode 8 introduced RAIDZ and dRAID. Episode 19 explores the advanced layers of OpenZFS 2.4: BRT for fast deduplication, RAIDZ expansion now stable, dRAID mature for large scale, and tiering to combine flash speed with HDD capacity.

These features mark ZFS's direction: not just safe storage, but smart storage — detecting duplicate blocks, adding disks without teardown, and placing hot data on fast media. Let's start with the dedup that makes ZFS unique.

#BRT: Fast Deduplication

#The Traditional Dedup Problem

Deduplication discards repeated data — if many files contain the same blocks, ZFS stores them once. Traditional dedup uses a large hash table that must live in RAM; if it doesn't fit, performance tanks and RAM runs out quickly. This is why old ZFS dedup was rarely used in production.

#The Block Reference Table

BRT (Block Reference Table) changes how ZFS dedup works. With BRT, dedup becomes much faster and more resource-efficient — block reference counting no longer demands a runaway hash table:

Check support and the OpenZFS version

zpool version
cat /sys/module/zfs/parameters/zfs_dedup_table_quota 2>/dev/null

The zpool version output shows the pool version. Features like BRT, RAIDZ expansion, and AnyRaid require OpenZFS 2.4 or later — make sure your distro provides the latest packages.

#Enabling Dedup

For a specific dataset, dedup can be enabled per dataset:

Enable dedup on a dataset

sudo zfs set dedup=on tank/data
zpool status tank

The zfs set dedup=on command enables dedup. Use it wisely: dedup is most effective on data with lots of duplicates, like VM images, backup archives, or container repos. For random data, dedup just wastes resources.

#RAIDZ Expansion and dRAID

#Stable RAIDZ Expansion

Since episode 8 we've known RAIDZ expansion allows adding disks to an existing RAIDZ vdev. In OpenZFS 2.4 this feature is considered stable for production use:

Add a disk to a RAIDZ vdev

sudo zpool attach tank raidz1-0 /dev/sde
zpool status tank

The zpool status tank output shows the data redistribution progress. The process runs gradually without downtime — data remains accessible throughout the expansion.

#Mature dRAID

dRAID, introduced in episode 8, is now mature in OpenZFS 2.4. Its advantage for large arrays remains the same: distributed spares make rebuilds fast and even, and it resists multiple failures:

Check the dRAID pool status

zpool status draid-pool

The zpool status draid-pool output shows config: with a draid2:... vdev and distributed spares. For servers with 16+ disks, dRAID is the primary choice.

Info

RAIDZ expansion and dRAID serve different needs: expansion for adding capacity to an existing vdev without teardown, dRAID for designing a large array from the start with fast rebuilds. Know when to use each.

#Tiering with Special Vdevs

#What Is a Special Vdev

A special vdev is an SSD device that stores metadata and (optionally) frequently accessed small blocks. By placing metadata on flash, metadata-heavy operations — directory listings, inodes, small databases — become much faster without adding many SSDs:

Add a special vdev to the pool

sudo zpool add tank special /dev/nvme0n1
zpool status tank

The zpool status tank output shows special as a special section in the config. Use a reliable SSD — losing a special vdev can make the pool unmountable.

#Configuring Special Small Blocks

ZFS can move small blocks to the special vdev using special_small_blocks:

Direct small blocks to the special vdev

sudo zfs set special_small_blocks=64K tank/data

The zfs set special_small_blocks=64K command directs blocks up to 64K to the special vdev. This reduces HDD load for small I/O and speeds up workloads like mail spools or databases.

#Hybrid Flash and HDD

#Tiering Strategies in the Real World

The flash-plus-HDD combination is the heart of modern tiering:

HDD for large capacity and cold data.
SSD for metadata, logs, and hot small blocks.
L2ARC for caching frequently read data.

The ideal configuration for many NAS and homelab servers:

Pool with special and cache

sudo zpool create tank raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde \
    special /dev/nvme0n1 cache /dev/nvme1n1
zpool status tank

The zpool create tank raidz2 command with added special and cache combines HDD capacity with flash speed — the approach used by products like TrueNAS Enterprise tiering.

#Monitoring Effectiveness

Measure whether tiering works with ZFS statistics:

Per-vdev IO statistics

zpool iostat -v tank

The zpool iostat -v tank output shows per-vdev statistics including L2ARC and special. Watch the cache hit ratio and per-vdev load to judge whether the added flash is paying off.

#Conclusion

Episode 19 completed the advanced ZFS discussion: BRT for fast dedup, stable RAIDZ expansion, mature dRAID, and tiering with special vdevs and hybrid flash plus HDD. You can now design smart, layered ZFS storage.

Key takeaways:

BRT makes ZFS dedup fast and RAM-efficient in OpenZFS 2.4.
RAIDZ expansion is stable for adding disks without tearing down a vdev.
dRAID is mature for large arrays with fast, even rebuilds.
A special vdev speeds up metadata access with just a little SSD.
special_small_blocks directs small blocks to fast media.
The HDD plus special and cache combination is modern practical tiering.

In the next episode, episode 20, we'll discuss RAID-as-a-Service & Cloud — RAID 0 and 10 on EC2 EBS, managed RAID and cloud NVMe, object storage as a redundancy layer, and local persistent volumes and storage abstractions in Kubernetes with Ceph. You'll bring RAID concepts to the cloud.