Learn Linux Filesystem - Monitoring & Integrity

#Introduction

Most storage problems are detected early — the signs are in df, smartctl, and zpool status long before data is actually lost. Episode 17 covers how to monitor filesystems comprehensively: from one-liner tools to full monitoring integration with Prometheus.

Monitoring isn't just looking at numbers; it's building an alerting system: when to act, when to wait, and when to replace a disk. We also cover scrub and resilver — the two processes that actively maintain data integrity on COW filesystems.

By the end of this episode, you'll have a complete list of metrics that must be monitored along with their thresholds.

#Basic Monitoring Tools

#df and du

df reports filesystem capacity and usage; du computes directory usage:

df -hT
du -sh /var/lib/docker

Note the difference between them: df reads filesystem metadata, while du computes from directory contents. A large difference between the two indicates deleted files still held open by a process (space not released), or a COW filesystem retaining old blocks.

To find the biggest space consumers:

du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head -10

#iostat and vmstat

iostat shows per-device performance; vmstat reads system activity:

iostat -x 5
vmstat 5

The most meaningful columns: %util, await, r/s, w/s in iostat, and wa in vmstat. %util near 100 percent with high await is a sign of a full queue.

#Monitoring btrfs and ZFS

#btrfs device stats

btrfs records error counts per device:

sudo btrfs device stats /mnt/lab
sudo btrfs filesystem usage /mnt/lab

The output shows counters like write_io_errs, read_io_errs, flush_io_errs, corruption_errs, and generation_errs. A growing non-zero number is a sign of damage — check with btrfs scrub.

#zpool status and zpool iostat

zpool status is the pool health dashboard:

zpool status
zpool iostat -v 5

Watch the CKSUM and REPAIR columns — both should be zero. Pool status must be ONLINE; a DEGRADED status means a vdev has failed or is resilvering.

#SMART Monitoring

#Reading SMART with smartctl

SMART (Self-Monitoring, Analysis and Reporting Technology) is the disk's built-in telemetry. smartctl from the smartmontools package reads it:

sudo smartctl -a /dev/sda
sudo smartctl -H /dev/sda

smartctl -H shows overall health; -a shows full attributes. The most important attributes to monitor:

• Reallocated_Sector_Ct: sectors remapped because of bad sectors — should be zero or constant.
• Current_Pending_Sector: sectors waiting to be remapped — increasing means a dying disk.
• CRC_Error_Count: cable/contact errors — check the SATA cable if it grows.

Run an offline test for early detection:

sudo smartctl -t offline /dev/sda
sudo smartctl -l selftest /dev/sda

#Background SMART Monitoring

Some attributes only update while the disk is in use. Set up scheduled polling so fresh values are always available:

sudo smartctl -o on /dev/sda
sudo smartctl -S on /dev/sda

-o on enables automatic offline self-testing, -S on enables attribute autosave.

#Scrub and Resilver

#Regular Scrub Schedule

Scrub reads all data and verifies checksums. Schedule it regularly outside busy hours, because a scrub running alongside production load degrades performance:

30 2 * * 0 sudo btrfs scrub start /mnt/lab
30 3 * * 0 sudo zpool scrub labpool

#Resilver: Rebuild from Redundancy

Resilver is the process of recovering data when a disk is replaced in a mirror/RAIDZ vdev. Run it after replacing a disk, and don't add heavy load while it runs — resilver reads all data from the remaining disks:

sudo zpool replace labpool /dev/sdb /dev/sdc
zpool status

A resilvering status appears while the process runs.

#Prometheus Integration and Alerting

#node_exporter and Filesystem Metrics

node_exporter exposes filesystem and disk metrics to Prometheus:

node_exporter --collector.filesystem --collector.diskstats --collector.smartmonitor

Important metrics at the /metrics endpoint:

• node_filesystem_avail_bytes and node_filesystem_size_bytes: capacity.
• node_disk_read_bytes_total and write_bytes_total: throughput.
• node_disk_io_time_seconds_total: device active time.
• node_smartmon_reallocated_sector_s: reallocated sectors from smartctl.

#Alerting Rules

Example Prometheus rules for thresholds:

groups:
  - name: storage.rules
    rules:
      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1
        for: 30m
        labels:
          severity: warning
      - alert: ZpoolDegraded
        expr: node_zfs_pool_health != 1
        labels:
          severity: critical

node_zfs_pool_health exposes pool health status as a number. With the rules above, you're notified early — long before data is lost.

#Conclusion

Monitoring turns a filesystem from a black box into a predictable system. df, du, and iostat give a daily view; btrfs device stats and zpool status maintain integrity; SMART warns about physical damage; and Prometheus automates all of that supervision.

Key takeaways:

• Monitor capacity with df and usage with du regularly.
• btrfs device stats and zpool status show COW filesystem health.
• SMART: monitor Reallocated_Sector_Ct, Pending_Sector, and CRC_Error_Count.
• Schedule btrfs/zfs scrubs outside busy hours; watch resilver when replacing disks.
• node_exporter exposes filesystem, diskstats, and SMART metrics to Prometheus.
• Alerting must be selective to avoid alarm fatigue.

In the next episode, episode 18, we cover the filesystem selection guide — a guide to choosing ext4, XFS, btrfs, ZFS, and tmpfs based on needs, complete with feature trade-offs, RAM overhead, and vendor support. You'll get a decision framework you can use in real projects.