Learn RAID - Backup & Disaster Recovery

#Introduction

Back in episode 1 we mentioned a saying we'll keep repeating: RAID is not a backup. Episode 12 proves it and gives you a complete recipe for surviving disasters. RAID protects against one kind of data loss — dead disks — but it doesn't protect against accidental deletion, ransomware, or a burning building.

The solution is layered defense: RAID as the foundation of availability, snapshots for human error, replication for local disasters, and offsite backups for full-scale disasters. Let's break down each layer and how to assemble them.

#Why RAID Isn't a Backup

#Three Scenarios RAID Doesn't Handle

Think about these three scenarios:

• You accidentally type rm -rf /mnt/raid/project — RAID does nothing, the files are permanently gone.
• Ransomware encrypts the entire contents of the array — every mirror copy gets encrypted too.
• The server room catches fire — all the array's disks burn together.

All three make the point: RAID keeps hardware running, not data safe from mistakes. A backup is the layer that handles that domain.

#A Working Definition of Backup

A proper backup meets the 3-2-1 rule: three copies of the data, on two different media, with one copy in a different location. RAID can contribute the first copy at the first location, but the other two copies must come from snapshots, replication, and offsite backup.

#Defense Layers

#ZFS and btrfs Snapshots

A snapshot is an instant copy of the data state at one point in time. In ZFS, snapshots are nearly free because they only record block differences:

sudo zfs snapshot tank/data@2026-08-10
sudo zfs list -t snapshot

The zfs snapshot tank/data@2026-08-10 command marks the dataset's state. The zfs list -t snapshot output shows the stored snapshots. btrfs has a similar mechanism with btrfs subvolume snapshot. Snapshots protect against human error — just roll back to the latest snapshot.

#Replication to Another Machine

Snapshots also fuel replication: sending the data delta to another server. In ZFS, replication uses zfs send and zfs receive:

sudo zfs send tank/data@2026-08-10 | ssh backup-host sudo zfs receive backup/data

The zfs send command sends the snapshot over ssh, and zfs receive stores it on the other host. This process only sends the blocks that changed since the last snapshot — efficient for data that grows little each day.

#Offsite Backup

The final layer is a copy in a different location. Options include:

• Encrypted tape or hard disks stored somewhere else.
• Cloud object storage (S3, R2) using tools like restic or rclone.
• Cross-region replication for truly critical data.

Choose according to your budget and needs. What matters: there's a copy that survives even if one physical location is lost.

#Reassemble Procedure and Rebuild Ordering

#Reassemble After a Partial Failure

When a machine reboots after several disks had issues, the correct recovery order is:

sudo mdadm --detail --scan
sudo mdadm --assemble /dev/md0 /dev/sdb /dev/sdc /dev/sdd

The mdadm --assemble command explicitly assembles the array from the available disks. Never jump straight to --force without checking first — forced assemble must only be used when you're confident about which disks are correct.

#The Correct Rebuild Ordering

Once the array is active, follow this order:

1. Check the state: make sure no disk was assembled wrongly.
2. Replace the failed disk: --fail, --remove, then --add the replacement.
3. Wait for the rebuild to finish: monitor /proc/mdstat until 100%.
4. Verify: mdadm --detail must show all devices active.

while true; do cat /proc/mdstat; sleep 60; done

The cat /proc/mdstat command in a loop monitors progress until done. After a full [UUU] and a clean state, only then may you add normal workload back.

#Testing the Recovery Plan

#Restore Drills

A backup that's never been tested isn't a backup. Schedule periodic restore drills:

sudo zfs rollback -r tank/data@2026-08-10

The zfs rollback -r command returns the dataset to the snapshot point. Practice restores on a spare machine, record the time it takes, and measure your RTO (Recovery Time Objective) and RPO (Recovery Point Objective). These numbers become your service contract in a crisis.

#Conclusion

Episode 12 made clear that RAID is a foundation, not a substitute for backup. You now know how to build layered defense — ZFS and btrfs snapshots for human error, remote replication for local disasters, offsite backups for full-scale disasters — plus the reassemble procedure and correct rebuild ordering after partial failures.

Key takeaways:

• RAID doesn't protect against accidental deletion, ransomware, or physical disasters.
• Use the 3-2-1 rule: three copies, two media, one in another location.
• ZFS or btrfs snapshots protect against human error instantly.
• Replication uses zfs send and zfs receive for remote backups.
• Don't --force assemble without analysis; follow the correct rebuild order.
• Test restores regularly and measure your RTO and RPO.

In the next episode, episode 13, we'll discuss failure detection & monitoring — monitoring disks with smartctl, predicting failures through SMART attributes, and building a monitoring stack with mdadm monitor, node_exporter, and Prometheus plus dashboards. You'll build an early warning radar for storage.