RAID has failed: read this before you rebuild

RAID exists to survive a disk failure. A disk drops, the array keeps running, you replace the disk. That is the theory. In practice the largest data losses happen at exactly this moment.

Why a rebuild is risky

During a rebuild every disk in the array is read end to end. That is the heaviest load those disks will ever see.

Your array was probably bought at once, is the same age, and has run in the same conditions. If one is tired, the others are too. If a second disk fails under the rebuild load — in RAID 5 the array is gone.

On high-capacity disks a rebuild can take days. That is how long the exposure lasts.

Before starting a rebuild

• Read the state. How many disks dropped? Is the array degraded or failed? If failed, a rebuild is not the answer.
• Is your backup current? Know the answer before you begin. If not, back up first.
• Check the health of the other disks. Are any reporting SMART warnings?
• Image the disks first if you can. With critical data, taking a sector-level copy before rebuilding is the safest route.
• Make sure you pull the right disk. Removing a healthy disk kills a running array instantly.

Never do this

• Do not reinitialise or recreate the array. Options like initialize or create new array wipe the metadata; the data may still be on the disks but the information about which piece belongs where is gone.
• Do not mix the disks up. Order matters. Note which disk was in which bay before removing anything.
• Do not swap disks at random to see what happens. Every attempt complicates the picture.
• Do not change the controller and carry on. A different controller may write metadata differently.

If the array is already gone

That does not mean the data is unrecoverable. If the disks are physically sound, the array geometry can be worked out and the data extracted — stripe size, disk order and parity layout can all be determined.

But this is only possible if nothing further has been written. Every extra attempt reduces the chance. At this point the right move is to stop and describe the situation.

RAID is not backup

Often repeated, often forgotten. RAID protects against disk failure. It does not protect against:

• Accidental deletion
• Ransomware — encryption is written across every disk at once
• File corruption
• Controller failure
• Fire, water, theft

RAID protects uptime, not data. Backup protects data.