Learn RAID - Spare, Replace & Hot-swap

#Introduction

Episode 5 taught you how to spot the signs of a failing disk. Episode 6 teaches what to do when that sign actually arrives: replacing the failed disk without shutting down the server. The concept is called hot-swap, and its fuel is the hot spare.

This ability is the real test of redundancy. A RAID array feels pointless if you have to endure hours of downtime every time a disk dies. With a hot spare and a proper replace procedure, a dead disk can be handled within minutes — and the data stays safe. Let's start with the spare concept.

#Hot Spare: A Reserve Ready to Step In

#What Is a Hot Spare

A hot spare is a disk installed in the machine but not used as an active array member. When an active device fails, the kernel automatically pulls the spare into the array and starts a rebuild without manual intervention. This is the first line of defense against disk failure.

sudo mdadm --add-spare /dev/md0 /dev/sdd
sudo mdadm --detail /dev/md0

The mdadm --add-spare /dev/md0 /dev/sdd command marks /dev/sdd as a spare. The --detail output then shows Spare Devices : 1. After an active disk fails, this spare automatically gets promoted to an active device and the rebuild begins.

#Re-adding a Disk That Was Temporarily Disconnected

Not every incident means a dead disk. A loose cable or a power hiccup can make a disk "disappear" temporarily and then come back. If the same disk returns, don't add it as a new device — use --re-add so mdadm recognizes its old position:

sudo mdadm --re-add /dev/md0 /dev/sdb

The mdadm --re-add command only works if the data on the disk still matches its previous position in the array. If it doesn't, mdadm will refuse and you'll have to add it as a spare first.

#The Disk Replace Process

#Step by Step

When a disk really fails, the sequence looks like this:

cat /proc/mdstat
sudo mdadm --detail /dev/md0

The output shows [2/3] [_UU] and State : degraded. From here, mark the failed device, remove it from the array logically, then add the replacement:

sudo mdadm --fail /dev/md0 /dev/sdb
sudo mdadm --remove /dev/md0 /dev/sdb
sudo mdadm --add /dev/md0 /dev/sdd

The command sequence above separates the broken disk (--fail and --remove), then adds the new disk (--add), which automatically starts the rebuild. If there's no spare, --add directly promotes the new disk to an active member.

#Verifying the Rebuild

A rebuild takes time depending on disk size. Monitor it until completion:

watch -n 5 cat /proc/mdstat
sudo mdadm --detail /dev/md0

The /proc/mdstat output shows recovery = 45.3% with an estimated completion time. Once finished, --detail should show State : clean, Working Devices : 3, and Failed Devices : 0.

#Physical Hot-swap

#Making Sure Hot-swap Is Supported

Physical hot-swap depends on the hardware. Make sure of the following:

• The chassis and backplane support hot-swap bays.
• The disks use SATA/SAS connectors with removable brackets.
• The SATA ports have hot-plug enabled; SAS generally supports it from the start.
• The HBA/controller driver allows new devices to be recognized without a reboot.

On enterprise machines with a SAS backplane, replacing a physical disk is a safe operation. On desktops or entry-level servers with SATA connected directly to the board, the result depends on the chipset — test carefully in the lab first.

#Safe Hot-swap Sequence

The correct sequence when physically replacing a disk:

sudo mdadm --fail /dev/md0 /dev/sdb
sudo mdadm --remove /dev/md0 /dev/sdb
echo "disk sdb ready to be removed" | sudo tee /dev/console

Once the device is failed and removed, the kernel no longer writes to that disk. The bay's LED indicator (if any) lights up, and you can safely pull the disk. Install the new disk, then wait for the kernel to detect it:

lsblk
dmesg | tail -20

The dmesg output shows new device recognition messages such as sdg: sda 4GB. Once you know the device name, add it to the array with --add and let the rebuild run.

#Setting Up a Hot Spare in an Array

If your machine has a free slot, make one of the disks a permanent spare:

echo "MAILADDR admin@example.com" | sudo tee -a /etc/mdadm.conf
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf
sudo update-initramfs -u

After this, the spare status is preserved even across reboots. The combination of MAILADDR and the ARRAY line in mdadm.conf keeps monitoring and the array structure exactly as you set them.

#Conclusion

Episode 6 gave you a complete disk failure handling procedure: preparing a hot spare, re-adding a disk that was temporarily disconnected, replacing a failed disk with --fail, --remove, and --add, safely performing physical hot-swaps, and verifying the rebuild until it's clean.

Key takeaways:

• A hot spare automatically joins the array and rebuilds when an active disk fails.
• --add-spare marks a reserve disk; --re-add reattaches an old disk.
• Replace procedure: --fail, --remove, then --add the replacement disk.
• During a rebuild the array is at risk — reduce I/O load and don't panic.
• Physical hot-swap needs backplane and chipset support; test in the lab first.
• Final verification: State : clean and Failed Devices : 0.

In the next episode, episode 7, we'll discuss resize & reshape — growing array capacity with --grow, expanding the filesystem with resize2fs, and changing the RAID level or chunk size. You'll learn to grow storage without tearing the array down from scratch.