Learn Linux Filesystem - Production Hardening

#Introduction

So far we've been building capability. Episode 21 shifts focus to defense: concrete steps to ensure production filesystems withstand failures, attacks, and human error. Hardening isn't a single feature — it's layers of safeguards that cover each other.

We start with a foundation that's often overlooked: making sure journal and checksums are actually active. Then strict mount options, quotas and limits, the RAID + snapshot + offsite backup combination, UPS for COW filesystems, and correct fsck ordering at boot.

By the end of this episode, you'll have a checklist you can apply directly to production servers.

#A Foundation Often Overlooked

#Active Journal and Checksums

Before building something complex, make sure the foundation is active:

sudo dumpe2fs -h /dev/sda1 | grep -E "Filesystem features|Journal"
sudo xfs_info /dev/sda1

For ext4, make sure has_journal is in the features and there's no ^has_journal. For XFS, check log=internal and crc=1. For btrfs/ZFS, make sure checksums are active and not nodatacow:

sudo btrfs subvolume show /mnt/lab | grep flags
zfs get checksum labpool/data

Disabling journal or checksums for performance without fully understanding the trade-off is the most dangerous decision in filesystems — crash protection and corruption detection are both lost.

#Strict Mount Options

Partitions that don't need execution — especially /tmp — must be mounted with restrictions:

tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev,mode=1777,size=2G 0 0

The noexec,nodev,nosuid options prevent executing binaries from a publicly writable area, block devices, and disable setuid. For /var, /home, and other data partitions, apply similar policies if the workload allows.

Verify the active options:

findmnt -no OPTIONS /tmp

#Quotas and Limits

#Limit Usage

Without quotas, a single rogue process can fill the entire disk and stop other services. Enable limits from the start:

sudo tune2fs -O quota /dev/sda1
sudo mount -o usrquota /dev/sda1 /home
sudo setquota -u www-data 0 20G 0 0 /home

For btrfs, use qgroups (episode 8):

sudo btrfs quota enable /mnt/data
sudo btrfs qgroup limit 50G 1/0 /mnt/data

Don't forget to monitor global capacity and warn before the disk is truly full — quota is a safety net, not the only defense.

#RAID, Snapshots, and Offsite Backup

#Three Layers of Protection

Three different mechanisms handle three kinds of failure:

• RAID (mirror/RAIDZ/parity): handles disk hardware failure.
• Snapshots: handle human error and logical corruption.
• Offsite backup: handles building disasters and total failure.

They complement each other; none replaces another. RAID doesn't protect against deleted files; snapshots don't protect against a dead disk; backups don't protect against slow restores.

#Example Implementation

sudo zpool create data mirror sda sdb          # RAID
sudo zfs snapshot data@harian-$(date +%F)      # snapshot harian
sudo zfs send data@harian-$(date +%F) | ssh offsite "zfs receive backup"  # offsite

The same pattern applies to btrfs: multi-device mirror + snapshot + send/receive. What matters isn't the tool, but that all three layers are present and scheduled.

#UPS for COW Filesystems

#Why COW Is Sensitive to Power Loss

btrfs and ZFS rely on the atomicity of pointer operations — if power dies in the middle of an operation sequence, transactions can end up in an inconsistent state. Although both are designed to recover from crashes, repeated events increase the risk.

This is why a UPS (Uninterruptible Power Supply) matters on servers running btrfs or ZFS:

• It gives time for a clean shutdown (not a forced power-off).
• It avoids transactions cut off mid-stream.
• It protects logs and transactions currently being written.

#The Correct Shutdown Procedure

Make sure the system unmounts and exports pools cleanly when the UPS signals:

sudo zpool export data

Synchronize power management with the UPS through apcupsd or nut, and test the shutdown procedure several times. A healthy COW filesystem is one that rarely experiences forced power-off.

#Boot: fsck Timeouts and Fallbacks

#fsck That Doesn't Hang Boot

An fsck that runs automatically at boot can hang the system if it requests interactive input. Configure a timeout:

fsck.mode=force fsck.repair=yes

The kernel parameter fsck.repair=yes makes fsck repair automatically without waiting for input. Combining fsck.mode=force forces a full check for maintenance windows.

#Read-Only Fallback

A filesystem that fails at boot should enter read-only mode rather than allow writes to a damaged structure. For ext4, this option is set at mkfs or mount:

/dev/sda1 / ext4 defaults,errors=remount-ro 0 1

With errors=remount-ro, the filesystem becomes read-only on error — protecting the remaining data from further damage until an admin steps in.

#systemd fsck Ordering

systemd handles fsck ordering through the systemd-fsck unit. Make sure the ordering is correct:

systemctl status systemd-fsck-root.service
systemd-analyze verify /etc/fstab

systemd-analyze verify /etc/fstab validates fstab entries and the generated unit dependencies. Fix all warnings before declaring the system production-ready.

#Conclusion

Production hardening is a discipline, not a one-time checklist. Ensuring journal and checksums are active, strict mount options, quotas, three layers of data protection, UPS for COW, and correct fsck ordering — all of this forms layered defense. Each layer can fail, but not all at once.

Key takeaways:

• Make sure the journal (ext4/XFS) and checksums (btrfs/ZFS) are truly active.
• Mount /tmp and public areas with noexec,nodev,nosuid.
• Enable quotas and monitor capacity before the disk fills up.
• Combine RAID + snapshot + offsite backup — three layers, three functions.
• Install a UPS and test clean shutdowns for btrfs/ZFS.
• Configure fsck.repair=yes, errors=remount-ro, and validate fstab with systemd.

In episode 22 — the final episode — we cover the ecosystem, alternatives, and a final reflection. We compare all the filesystems, get to know alternatives like F2FS and NILFS2, underline that RAID isn't a filesystem and a filesystem isn't a backup, then recap the entire series with a complete production checklist. Your journey is almost complete.