Learn RAID - Failure Detection & Monitoring

#Introduction

Episode 5 taught you how to read an array's status; episode 13 goes further — detecting failures before they happen. Disks rarely die suddenly without warning. Most disks give early warning signals readable from SMART (Self-Monitoring, Analysis and Reporting Technology) data.

With proper monitoring, you can replace an aging disk before it damages the array. This episode builds that early warning radar: from smartctl at the disk level, to mdadm --monitor at the array level, up to a Prometheus and Grafana dashboard for a complete single view.

#Reading SMART with smartctl

#Getting SMART Data

The main tool is smartctl from the smartmontools package:

sudo smartctl -a /dev/sdb

The smartctl -a /dev/sdb output shows a SMART Attributes section with a list of attributes like Reallocated_Sector_Ct, Current_Pending_Sector, and Temperature_Celsius. The SMART overall-health section gives a brief healthy-or-not summary.

#Attributes to Watch

Not all attributes are equally important. Focus your attention on:

• Reallocated_Sector_Ct: spare sectors already used. A continuously rising number indicates the media is starting to fail.
• Current_Pending_Sector: sectors awaiting verification — if they keep growing, replace the disk soon.
• Temperature_Celsius: consistently high temperatures shorten disk lifespan.
• Raw_Read_Error_Rate and Seek_Error_Rate: read-write health indicators.

sudo smartctl -H /dev/sdb

The smartctl -H /dev/sdb output shows SMART overall-health self-assessment test result: PASSED or FAILED. But don't rely on this value alone — many disks already having problems still report PASSED.

#Predictive Failure

The term predictive failure refers to attributes crossing their thresholds. A combination of rising reallocated sectors, accumulating pending sectors, and a worsening Raw_Read_Error_Rate is the typical pattern of a disk heading toward failure. When this pattern appears, replace the disk on a schedule — don't wait for an incident.

#Scheduling SMART Tests

#Periodic Self-tests

smartctl can run self-tests that write to and read from the disk surface:

sudo smartctl -t long /dev/sdb
sudo smartctl -l selftest /dev/sdb

The smartctl -t long command starts a full self-test; -l selftest shows the history and results. Schedule it via cron to run outside peak hours. A failed long self-test is a signal to replace the disk.

#Monitoring All Disks at Once

On a server with many disks, run the check for all of them at once:

for d in /dev/sd[b-e]; do sudo smartctl -H "$d"; done

The smartctl -H command inside the for loop shows the health of all disks in one pass. Combine it with a cron schedule and notifications so the results reach you without logging in.

#Monitoring Stack with Prometheus

#node_exporter and the mdadm Collector

node_exporter is a Prometheus exporter for system metrics, including the mdadm and smartmon collectors:

node_exporter --collector.mdadm --collector.smartmon

The node_exporter --collector.mdadm command exposes metrics like node_md_disks, node_md_state, and node_md_disks_state. With the smartmon collector, SMART attributes are also published as metrics.

#Querying in Prometheus

With Prometheus scraping node_exporter, a query for degraded arrays:

node_md_state{state="degraded"}

Prometheus alerting rules can fire when this metric appears, then forward to Alertmanager for email, Slack, or webhooks. The combination of mdadm monitor and Prometheus gives you two complementary notification channels.

#Dashboards

Grafana visualizes all metrics in one view:

• Charts of node_md_disk per array and per disk.
• Temperature and reallocated sectors trends per disk.
• Exporter up/down status and storage latency.

With a dashboard, storage health is visible at a glance without digging through logs — and temperature or sector anomalies become visible before reaching a critical point.

#Conclusion

Episode 13 built an early warning radar for storage: reading and interpreting SMART, recognizing predictive failure patterns, scheduling self-tests, and assembling a monitoring stack with node_exporter, Prometheus, and dashboards. Disk failures no longer come without warning.

Key takeaways:

• smartctl -a shows SMART attributes; focus on reallocated and pending sectors.
• Predictive failure is an attribute pattern heading toward failure — don't wait for an incident.
• Schedule smartctl -t long periodically outside peak hours.
• node_exporter with the mdadm and smartmon collectors exposes array and disk metrics.
• Prometheus and Alertmanager turn metrics into notifications.
• Grafana brings everything together in one dashboard.

In the next episode, episode 14, we'll discuss security & encryption — LUKS and dm-crypt encryption above or below RAID, native ZFS encryption, key management, and checksums for detecting silent corruption. You'll learn to protect data from theft and silent corruption.