Learn RAID - Troubleshooting & Recovery Scenarios

#Introduction

All the procedures in this series were designed for calm times. Episode 16 covers emergencies: a degraded array, a failed rebuild, corrupted metadata, even a panicking system. This is where your skills are truly tested.

The first principle of recovery: don't panic and don't make things worse. A wrong step in chaos — like a hasty --force assemble — can turn a recoverable incident into total data loss. Let's go through the common scenarios one by one with a cool head.

#Degraded Array

#Initial Diagnosis

The most common scenario: monitoring reports a degraded array. The first step is reading the condition without changing anything:

cat /proc/mdstat
sudo mdadm --detail /dev/md0
sudo dmesg | grep -i "md\|raid"

The dmesg output shows kernel messages about the failed disk, such as md/raid1:md0: Disk failure on sdb, disabling device.. This tells you which disk has the problem and whether other disks are affected.

#Action

Once the diagnosis is clear, only then act:

sudo mdadm --fail /dev/md0 /dev/sdb
sudo mdadm --remove /dev/md0 /dev/sdb
sudo mdadm --add /dev/md0 /dev/sdd

The --fail, --remove, then --add sequence handles a single failure safely. Monitor the rebuild until it completes before assessing the condition further.

#Failed Rebuild

#Why Rebuilds Can Fail

A rebuild sometimes fails halfway through for one of these reasons:

• The replacement disk is actually defective (by coincidence or bad stock).
• An URE occurs — an unreadable sector on another disk during the rebuild.
• An unstable connection makes the disk drop out repeatedly.

The symptom shows up in mdadm --detail as a halted Rebuild Status, or the replacement disk suddenly enters a failed state.

#Handling Strategies

sudo dmesg | grep -i error | tail -20
sudo smartctl -a /dev/sdd | grep -i "reallocated\|pending"

The smartctl command above checks whether the replacement disk is healthy. If the replacement has problems, replace it again. If the issue is a URE on another disk, you must assess: is there any redundancy left? If it's a parity array and one more disk fails during the rebuild, stop and call in the backup.

#Corrupted Metadata and --force

#When --force Is Needed

Sometimes mdadm refuses to assemble an array because the metadata is inconsistent — for example two disks claiming the same position, or a damaged superblock. This is when --force can be used, BUT only after analysis:

sudo mdadm --examine /dev/sdb
sudo mdadm --examine /dev/sdc
sudo mdadm --examine /dev/sdd

Compare the mdadm --examine output of all disks: the array UUID must match, and check the Device Role in each superblock. If two disks claim the same role, you have to decide which disk is right — usually the one most recently active.

#Performing --force

Once confident, assemble with force:

sudo mdadm --assemble --force /dev/md0 /dev/sdc /dev/sdd

The --assemble --force command forces mdadm to use the available disks even if incomplete. The consequence: data on the sacrificed disk may be inconsistent. Once the array is active, immediately --add a replacement disk and let the rebuild run.

#Damaged Disks and ddrescue

#Copying a Failing Disk

If a disk can no longer be read fully but is still partially readable, don't force the array to read directly from it. Copy it to another medium first with ddrescue, which copies as much as possible then retries the failing parts:

sudo ddrescue /dev/sdb /dev/sde /var/log/ddrescue.log

The ddrescue /dev/sdb /dev/sde command copies /dev/sdb's contents to /dev/sde while recording progress in the logfile. The process can be run repeatedly with the -r mode to retry failed sectors — this often rescues data from a disk heading toward failure.

#Using the Copy

After the copy finishes, mount the result into the array:

sudo mdadm --fail /dev/md0 /dev/sdb
sudo mdadm --remove /dev/md0 /dev/sdb
sudo mdadm --add /dev/md0 /dev/sde

The --add /dev/sde command makes the copy an array member. Rebuilding from a copy is far safer than forcing the original damaged disk to work harder.

#OOM and Kernel Panic

#Storage as the Victim

OOM (Out Of Memory) and kernel panics are rarely caused by storage, but they often impact the array. When the kernel panics or the system crashes, unfinished writes can leave the array and filesystem inconsistent. After reboot, the sequence is:

cat /proc/mdstat
sudo mdadm --detail /dev/md0
sudo journalctl -b -1 | grep -i "oom\|panic"

The journalctl -b -1 output shows the previous boot's logs; look for traces of OOM or panic. If the array looks inconsistent, consider fsck on the filesystem after confirming the array is clean — and review the cause of the OOM so it doesn't repeat.

#Preventing Recurrence

• Set vm.swappiness and memory limits for large processes.
• Install adequate swap or review workload allocation.
• Use a write-intent bitmap (episode 17) so a crash doesn't force a long rebuild.

#Conclusion

Episode 16 armed you with recovery procedures for worst-case scenarios: handling a degraded array, a failed rebuild, corrupted metadata with --force, copying damaged disks with ddrescue, and facing OOM and kernel panic — always with the discipline: diagnose first, act afterward.

Key takeaways:

• Diagnose with mdstat, --detail, and dmesg before changing anything.
• A failed rebuild is usually due to a bad replacement disk or a URE.
• --force assemble only after comparing the metadata of all disks.
• ddrescue rescues data from disks heading toward failure.
• OOM and panics leave arrays inconsistent — check before writing.
• When in doubt between --force and backup, prioritize the backup.

In the next episode, episode 17, we'll discuss advanced mdadm features — internal bitmap for faster rebuilds, the new lockless bitmap in mdadm 4.6, and the write-intent journal for RAID 4/5/6 stability. You'll learn rarely-used features that nevertheless determine an array's resilience.