How to Replace a Failed Disk in Linux Software RAID1 (mdadm)

In a RAID1 mirror, two disks hold the same data, so the server keeps running when one of them fails. But until you replace the failed disk, there is no second copy: if the remaining disk fails too, the data is gone. This guide walks through a real replacement on a server using Linux software RAID (mdadm), from the first alert to a fully rebuilt mirror.

Quick answer

1. Run cat /proc/mdstat to find the failed disk. A [U_] means one disk is missing.
2. Note the serial number of the healthy disk with lsblk -o NAME,SERIAL, so the data centre removes the right one.
3. After the new disk is fitted, copy the partition table from the healthy disk to the new disk with sfdisk.
4. Add each new partition back into its array with mdadm /dev/mdX --add, and wait for the rebuild.
5. Install the bootloader on the new disk with grub-install (or grub2-install), so the server can still boot if the other disk fails.

What you need: root access, a recent backup, and a replacement disk at least as large as the old one. The example uses /dev/sda for the healthy disk and /dev/sdb for the failed one. Check the names on your own server before running anything: a command aimed at the wrong disk can destroy the good copy of your data.

1. Check the array status

The first sign is usually an email from mdadm's monitoring: "DegradedArray event detected on md device". Confirm it:

cat /proc/mdstat

Personalities : [raid1]
md2 : active raid1 sda3[0]
      1918777408 blocks super 1.2 [2/1] [U_]
      bitmap: 8/15 pages [32KB], 65536KB chunk

md1 : active raid1 sda2[0]
      1046528 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[0]
      33520640 blocks super 1.2 [2/1] [U_]

This server has three mirrored arrays: md0, md1 and md2, each made of one partition from each disk. Every array shows [2/1] [U_]: two members expected, one working. Only sda partitions are listed, so sdb has failed. Linux md documentation

For more detail on one array:

mdadm --detail /dev/md2

Look at State (degraded) and the device list at the bottom, which shows which member is missing or marked faulty. mdadm manual

2. Remove the failed disk from the arrays

If the disk died completely, as here, the kernel has already dropped it and you can skip to step 3.

If the failing disk is still listed in /proc/mdstat (shown with (F) for faulty, or still active but throwing errors), mark it failed and remove it from every array it belongs to before it is pulled:

mdadm --manage /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
mdadm --manage /dev/md1 --fail /dev/sdb2 --remove /dev/sdb2
mdadm --manage /dev/md2 --fail /dev/sdb3 --remove /dev/sdb3

3. Identify the disk by serial number

The data centre needs to know which physical disk to replace. Serial numbers are printed on the disk label, so they are the safest way to identify it.

lsblk -o NAME,SIZE,SERIAL

NAME      SIZE SERIAL
sda       1.8T K5HRNX7A
├─sda1     32G
│ └─md0    32G
├─sda2      1G
│ └─md1  1022M
└─sda3    1.8T
  └─md2   1.8T

A dead disk often does not appear at all, as here. In that case, give the data centre the serial of the healthy disk (K5HRNX7A) and ask them to replace the other one. If the failing disk is still visible, give its serial directly; smartctl -i /dev/sdb also shows it.

Book the replacement with your data centre. Most need the server switched off for a few minutes. Some support hot-swap, which needs no downtime.

4. Copy the partition table to the new disk

After the swap, check that the new disk is present and empty:

lsblk -o NAME,SIZE,SERIAL

The new disk should appear (here as sdb) with a new serial number and no partitions. The new disk must have exactly the same partitions as the healthy one. First save a copy of the healthy disk's partition table, just in case:

sfdisk --dump /dev/sda > /root/sda-partition-table.bak

Then copy the table from the healthy disk to the new disk. Read this command twice before you run it: the first disk is the source (healthy), the second is the destination (new).

sfdisk --dump /dev/sda | sfdisk /dev/sdb

sfdisk handles both older MBR and newer GPT partition tables. sfdisk manual

GPT disks only: a copied GPT table also copies the disk's unique IDs, and two disks with the same IDs confuse Linux. Give the new disk fresh ones (the sgdisk command is in the gdisk package):

sgdisk -G /dev/sdb

To check which type you have, run fdisk -l /dev/sda and look at Disklabel type: gpt or dos (MBR). sgdisk manual

5. Add the new disk to the arrays

Add each new partition to the array that matches it, using the layout from step 1:

mdadm --manage /dev/md0 --add /dev/sdb1
mdadm --manage /dev/md1 --add /dev/sdb2
mdadm --manage /dev/md2 --add /dev/sdb3

The arrays start copying data to the new disk straight away. Follow the progress:

watch cat /proc/mdstat

md2 : active raid1 sdb3[2] sda3[0]
      1918777408 blocks super 1.2 [2/1] [U_]
      [=>...................]  recovery =  5.8% (112027840/1918777408) finish=179.6min speed=167571K/sec

md1 : active raid1 sdb2[2] sda2[0]
      1046528 blocks super 1.2 [2/1] [U_]
        resync=DELAYED

md0 : active raid1 sdb1[2] sda1[0]
      33520640 blocks super 1.2 [2/2] [UU]

Arrays rebuild one at a time, so resync=DELAYED just means waiting its turn. A large disk can take several hours. The server works normally meanwhile, but disk-heavy tasks are slower. Press Ctrl+C to leave watch.

Do not reboot or remove anything until every array shows [UU].

6. Install the bootloader on the new disk

The new disk has your data but cannot boot the server yet. If the old disk fails later, the server would not start. Install GRUB on the new disk now; it can run while the rebuild continues.

First check how the server boots:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS

BIOS (legacy) boot. On Debian or Ubuntu:

grub-install /dev/sdb

On AlmaLinux, Rocky Linux or CloudLinux:

grub2-install /dev/sdb

It should finish with Installation finished. No error reported. GRUB manual

UEFI boot. The boot files live on a separate EFI partition rather than in the RAID. How to copy it depends on how your server was set up. On many servers the EFI partition is itself a RAID1 array, which the rebuild in step 5 has already covered. Otherwise, follow your distribution's documentation for setting up a second EFI partition.

7. Check the result

When the rebuild is finished:

cat /proc/mdstat
mdadm --detail /dev/md2

Every array should show [2/2] [UU], and mdadm --detail should show State : clean with Failed Devices : 0.

Finally, make sure mdadm monitoring will email you next time. Check that /etc/mdadm/mdadm.conf (Debian/Ubuntu) or /etc/mdadm.conf (AlmaLinux/Rocky) has a MAILADDR line with an address you read.

To spot the next failing disk before it fails, see How to Check SSD Health in Linux.