Learn RAID - Maintenance & Scrub

#Introduction

Building an array is a one-time event; maintaining it is ongoing. Episode 15 covers the routine care that keeps storage healthy for years: scrubbing to check data integrity and physical maintenance to protect the storage environment.

A scrub is a thorough sweep that looks for damaged blocks before they're needed — analogous to a medical check-up done while healthy, not while sick. Combined with care like UPS and firmware updates, you can extend an array's lifespan and avoid needless failures.

#Scrub in mdadm: --check and --repair

#Checking Array Integrity

mdadm provides two scrub modes: --check which only verifies consistency, and --repair which fixes discrepancies found:

echo check > /sys/block/md0/md/sync_action
cat /proc/mdstat

The echo check > /sys/block/md0/md/sync_action command starts the check. The /proc/mdstat output shows check = 42.3% with progress. When the check finds discrepancies, it records them in the mdadm and kernel logs.

#The --repair Mode

If --check finds problems and you have the redundancy to fix them:

echo repair > /sys/block/md0/md/sync_action

The echo repair command fixes discrepancies using consistent data from other disks. Remember: a scrub uses full I/O resources while running — for large disks, make sure it's scheduled outside peak hours.

#ZFS and btrfs Scrub

#zpool scrub

ZFS offers the most powerful scrub: it checks the checksums of all blocks and repairs damaged ones automatically:

sudo zpool scrub tank
zpool status tank

The zpool status tank output shows a scan: scrub in progress since ... line with X.X% done. When finished, the status becomes scrub repaired 0B in ... with 0 errors on .... A non-zero error count means corruption was found — investigate further.

#Checking and Canceling a Scrub

zpool status tank
sudo zpool scrub -s tank

The zpool scrub -s tank command stops a running scrub. This is useful if the scrub is interfering with production workloads — move the schedule to a quieter time.

#btrfs scrub

For the btrfs filesystem, scrub runs per mounted filesystem:

sudo btrfs scrub start /mnt/data
sudo btrfs scrub status /mnt/data

The btrfs scrub start /mnt/data command checks all blocks and repairs errors using redundancy. The btrfs scrub status output shows the status and error counts.

#Scheduling Routine Scrubs

#Automation with cron

Unscheduleed scrubs often get forgotten. Schedule them with cron, for example every Sunday morning:

0 3 * * 0 /usr/sbin/zpool scrub tank
0 4 * * 0 echo check > /sys/block/md0/md/sync_action

The cron lines above run ZFS and mdadm scrubs every Sunday. For mdadm, it's better to use mdadm --check via a systemd timer or a script that waits until completion.

#A Reasonable Frequency

A practical rule: scrub once a month for ZFS and mdadm on normal data, or once a week for arrays with critical data. Scrubbing too often wastes I/O; too rarely lets corruption pile up undetected.

#Physical and Environmental Maintenance

#Power and UPS

Unstable power is the most common storage killer. Install a UPS adequate for the server, make sure the system shuts down automatically when the battery runs low, and check the UPS battery status periodically:

apcaccess status

The apcaccess status output shows LINEV, LOADPCT, BCHARGE, and STATUS. A worn-out UPS battery provides deceptive protection — replace the battery before it actually dies.

#Disk Aging and Firmware

Disks have a lifecycle. Pay attention to:

• Temperature: keep the room cool; temperatures above 45 degrees shorten HDD lifespan.
• Vibration: use stable mounting; vibration accelerates bearing wear.
• Firmware updates: vendors release updates that fix bugs; check periodically on the vendor's site.

#Notes on Firmware Updates

Updating disk firmware is a risky operation that can sometimes even change how a disk behaves. The safe rules: read the vendor changelog, do it on a single spare disk first, and never update the firmware of all array disks at once — you don't want every disk offline at the same time.

#Conclusion

Episode 15 equipped you with maintenance habits: scrubbing with mdadm --check and --repair, zpool scrub, and btrfs scrub, scheduled routinely, plus physical care like UPS, temperature control, and careful firmware updates.

Key takeaways:

• mdadm --check verifies, --repair fixes discrepancies on the array.
• zpool scrub checks the checksums of all ZFS blocks and repairs automatically.
• btrfs has btrfs scrub to check and repair the filesystem.
• Schedule monthly scrubs for normal data, weekly for critical data.
• Don't run a scrub at the same time as a rebuild.
• Maintained UPS, temperature, vibration, and firmware extend disk lifespan.

In the next episode, episode 16, we'll discuss troubleshooting & recovery scenarios — facing a degraded array, a failed rebuild, corrupted metadata, even OOM and kernel panic, complete with --force assemble and ddrescue for damaged disks. You'll learn to stay calm when storage misbehaves.