Learn NAS - Advanced ZFS (Tiering, sVDEV, Tuning)

#Introduction

ZFS has many features untouched by basic usage. Episode 17 covers advanced ZFS: how to use SSDs for metadata and caching, adjust block sizes, set up flash and HDD tiering, and manage ARC, compression, and dedup wisely.

The end goal isn't just making the system look fast — it's understanding the trade-offs of every option. An SSD cache doesn't help if the workload doesn't fit. Dedup can eat RAM if applied carelessly. This knowledge is what separates ZFS users from ZFS administrators.

By the end of this episode you'll be able to install a special vdev, configure L2ARC and SLOG, tune recordsize, understand dataset tiering, manage ARC, use zstd, and decide when dedup is worth it.

#Special VDEVs for Metadata

#What Is a Special VDEV

A special vdev (sVDEV) stores metadata and small blocks separately, typically on SSDs. Because operations like file creation and directory listing depend on metadata, moving it to fast media dramatically improves responsiveness.

zpool add tank special /dev/nvme0n1

The zpool add tank special command adds an SSD as a special vdev. Once installed, pool metadata moves there; the main HDDs focus on large data.

#sVDEV Considerations

A special vdev isn't a cache — if it fails, metadata is lost and the pool can become inaccessible. For that reason, use a mirror for the special vdev and make sure the SSD is enterprise-grade. This isn't a place for cheap SSDs.

#L2ARC and SLOG

#L2ARC for Read Caching

L2ARC is a second-level read cache on SSD or NVMe that speeds up access to frequently read data that doesn't fit in RAM. Unlike ARC in RAM, L2ARC can be much larger.

zpool add tank cache /dev/nvme1n1

The zpool add tank cache command adds a device as L2ARC. Remember: L2ARC still consumes RAM for its metadata index, so the benefit must outweigh the RAM cost.

#SLOG for Sync Writes

SLOG captures the intent log to speed up synchronous writes, as discussed in episode 3. SLOG isn't a data cache; if lost, the data remains safe because the intent can be rebuilt. Use an SSD with power-loss protection.

zpool add tank log /dev/nvme2n1

The zpool add tank log command adds a device as SLOG. Its focus is fsync latency, not large capacity.

#Tuning Recordsize

#The Right Recordsize

recordsize determines a dataset's logical block size. The 128k default is good for general files, but specific workloads benefit from adjustment.

zfs create -o recordsize=16k tank/data/db
zfs create -o recordsize=1m tank/data/media

The commands above create a db dataset with 16k recordsize for a database, and a media dataset with 1m for large files. Rule of thumb: small for databases, large for media, 128k for ordinary files.

#Changing Recordsize

Recordsize only affects new files; existing files keep their old block size. Set recordsize when the dataset is created, or recopy the data if you want to apply a new value.

#Dataset Tiering

#The Flash and HDD Tiering Concept

Dataset tiering is the concept of placing data on media with different characteristics: hot data on high-speed SSDs, cold data on high-capacity HDDs. TrueNAS Enterprise brings tiering features, and in a homelab you can simulate it with separate datasets per medium.

zfs create ssd-pool/hot
zfs create hdd-pool/archive

The commands above create datasets in two different pools. This manual policy gives you full control over data placement without relying on automated features.

#Practical Tiering Strategies

• Frequently accessed data: SSD pool.
• Archive data: HDD RAIDZ pool.
• Move data between tiers based on age and access frequency.

TrueNAS SCALE Enterprise's automatic tiering automates this movement. For a homelab, scheduled rsync scripts can do something similar.

#ARC Size and Compression

#Managing ARC

ARC (Adaptive Replacement Cache) uses RAM for read caching. By default ZFS uses a large portion of RAM, which can interfere with other applications. Limit ARC if needed, but give enough room to storage workloads.

arc_summary

The arc_summary command shows ARC statistics: hit rate, size, and amount of memory used. This data informs decisions to raise or lower RAM.

#Zstd Compression

zstd is a modern, fast, and effective compression. Enable it for almost all data; except already-compressed data like video and archives, which just wastes CPU with no meaningful savings.

zfs set compression=zstd tank/data

The zfs set compression=zstd command enables compression on the dataset. Verify its effectiveness through the per-dataset compression ratio.

#Dedup and When to Use It

#How Dedup Works

Dedup removes identical data blocks, very useful for data with many duplicates like VM snapshots and backups. However, dedup needs a DDT (dedup table) in RAM; the rule of thumb is about 5 GB of RAM per terabyte of deduplicated data.

zfs set dedup=on tank/data/vm

The zfs set dedup=on command enables dedup. Use it with extreme care: a miscalculation of RAM can drastically degrade performance.

#When Dedup Is Worth It

Dedup is worthwhile if the data is truly highly duplicated and there's enough RAM. Before enabling it, estimate the space savings from the actual duplication ratio. For most homelabs, dedup harms more often than it helps.

#Performance Monitoring

#zpool iostat

Monitor pool I/O to validate tuning:

zpool iostat -v 5

The zpool iostat -v 5 command shows throughput and I/O per second per vdev, refreshed every 5 seconds. It's the main tool for seeing whether sVDEV, L2ARC, or SLOG really help.

#Reading the Monitoring Results

• Compare write latency before and after SLOG.
• Watch ARC and L2ARC hit rates to assess cache adequacy.
• Detect vdevs that become bottlenecks compared to others.

#Closing

In this episode 17 you went deep into ZFS: special vdevs for metadata, L2ARC and SLOG, recordsize tuning, dataset tiering, ARC management, zstd compression, dedup, and performance monitoring with zpool iostat.

Key takeaways:

• A special vdev speeds up metadata but must be mirrored and high quality.
• L2ARC and SLOG serve different workloads; measure their impact.
• Recordsize must match the workload type when the dataset is created.
• Tiering separates hot and cold data onto the right media.
• Dedup saves space only if RAM is sufficient and the data is truly duplicated.

In the next episode, episode 18, we'll cover applications and containerization — from TrueNAS Apps, Docker and Portainer, Helm, to Plex, Nextcloud, Vaultwarden, and Gitea containers with resource limits, plus the Docker and KVM plugins on OpenMediaVault. ZFS is optimal; now it's time to fill the NAS with applications.