Learn RAID - mdadm: Detail, Status & Monitoring

#Introduction

In episode 4 you learned how to create an array. Now the question is: how do you know the array is healthy? Episode 5 teaches reading and monitoring the array — two skills that distinguish a reactive admin from a proactive one.

The bad news is that disks fail without announcement. The good news is that the Linux kernel provides a complete information window through /proc/mdstat and the mdadm --detail command. By understanding both, you can detect problems before users report them. Let's start with the first window.

#Reading /proc/mdstat

#Output Anatomy

The /proc/mdstat file is the fastest source of array status in Linux. Read it with:

cat /proc/mdstat

A typical output shows:

Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sdb[0] sdc[1]
      4193280 blocks super 1.2 [2/2] [UU]
      [==========>...........] resync = 55.3% (2323488/4193280) finish=2.1min speed=15000K/sec
unused devices: <none>

The [2/2] [UU] figure is the core thing you must understand: the numerator states the number of active devices, the denominator the number there should be. U means the device is up, _ means the device has failed. If you see [1/2] [_U], one disk is already dead.

#States During Operations

Besides clean or active, /proc/mdstat shows ongoing operations:

• resync: the first full synchronization after the array is created.
• recovery: refilling data onto a new disk during a rebuild.
• reshape: changing the array structure, for example switching levels or adding disks.
• check and repair: scrub operations that we'll discuss in episode 15.

The speed and finish lines give an estimate of when the operation will complete. If a rebuild runs too long, you can adjust its speed — we'll cover how in episode 16.

#mdadm --detail and State Interpretation

#Details of a Single Array

For complete information on one array, use:

sudo mdadm --detail /dev/md0

The output shows State : clean, Active Devices : 2, Working Devices : 2, Failed Devices : 0, and Spare Devices : 0. There's also Rebuild Status : 55% complete during an ongoing process.

#Array States to Watch For

Here's the list of states you'll encounter:

• clean: the array is healthy, no operation in progress.
• active: the array is active and serving I/O.
• degraded: one or more devices have failed; the array still runs but has lost redundancy.
• rebuilding: the array is refilling a replacement disk.
• resyncing: a full synchronization is in progress.
• recovering: the array is restoring data to a spare or new disk.

#Scanning All Arrays

You can see a summary of all arrays with a single command:

sudo mdadm --detail --scan

The output of mdadm --detail --scan prints a line like ARRAY /dev/md0 metadata=1.2 name=... UUID=... for each array. This is the form copied into mdadm.conf in episode 4.

#Monitoring with mdadm --monitor

#Monitor Mode

mdadm --monitor is a daemon process that watches for array status changes and takes action according to configuration. The simplest example:

sudo mdadm --monitor --scan --test --mail=admin@example.com

The --test option sends a test email to make sure the configuration works before real use. Without --test, the daemon keeps running in the foreground — for production, let systemd run it as a service.

#Monitored Events

mdadm monitor notifies you of events such as:

• DeviceDisappeared: a member disk is no longer visible.
• Fail: a disk has entered a failed state.
• DegradedArray: the number of active devices has decreased.
• RebuildStarted and RebuildFinished: the start and end of the rebuild process.
• RebuildFailed: a rebuild failed — one of the most critical moments.

Each event can be sent by email or run as an external program using the --program option.

#systemd Integration with mdmonitor

#Enabling the Service

On almost all modern distros, the mdadm package provides the mdmonitor.service systemd unit that runs the monitor automatically:

sudo systemctl enable mdmonitor.service
sudo systemctl start mdmonitor.service
sudo systemctl status mdmonitor.service

With the service active, you don't need to run mdadm --monitor manually at every boot. Email and program configuration is set in /etc/mdadm.conf via the MAILADDR and PROGRAM lines.

#Testing the Notification Flow

Test the entire chain — from failure detection to the email arriving:

sudo mdadm --monitor --scan --test --mail=admin@example.com --syslog

The mdadm --monitor --scan --test command above sends a test email and writes to syslog. If the test email arrives, it means the daemon, MAILADDR, and the local mail service (for example mailutils) are wired up correctly.

#Routine Inspection Summary

Make it a habit to read the array's condition periodically:

• Daily: cat /proc/mdstat to make sure no array is degraded.
• Weekly: sudo mdadm --detail /dev/mdX to check failed and spare devices.
• Automated: mdmonitor.service for real-time failure notifications.

With this combination, you'll know about problems within minutes, not days.

#Conclusion

Episode 5 equipped you with observation skills: reading /proc/mdstat, interpreting mdadm --detail and array states, running mdadm --monitor for alerts, and making use of systemd's mdmonitor.service. You can now keep your array healthy proactively.

Key takeaways:

• /proc/mdstat is the fastest array status window; read [U] and [_].
• mdadm --detail gives details on state, device count, and rebuild progress.
• A degraded state means lost redundancy — replace the disk as soon as possible.
• mdadm --monitor sends email alerts on failure events.
• Enable mdmonitor.service so monitoring runs automatically at boot.
• Test the notification flow with --test before you actually need it.

In the next episode, episode 6, we'll discuss spare, replace & hot-swap — adding a hot spare with --add-spare, replacing failed disks, and making sure the rebuild runs to completion. You'll practice handling disk failures without panic.