Recovering a failed raid1/raidintegrity member under IO load

[Lalufu]

I'm playing around with a VM to get a feeling for operational issues on LVM RAID setups.

I have two small drives (vdc and vdd) made into a LVM RAID1 with raidintegrity, with an XFS file system on top

# vgcreate vg_raid /dev/vdc1 /dev/vdd1

# lvcreate \
    --type raid1 \
    -m 1 \
    -L 9G \
    -n lv_raid \
    --raidintegrity y \
    --raidintegrityblocksize 4096 \
    --raidintegritymode journal \
    vg_raid

# mkfs.xfs -f /dev/vg_raid/lv_raid

The file system gets mounted on /mnt, I create a file on there, and start an fio process that's basically randomly reading from the file

# dd_rescue /dev/zero /mnt/outfile -m 6G
# fio --name=randread --ioengine libaio --rw=randread --bs=4k --direct=1 --filename=/mnt/outfile --direct=1 --numjobs=1 --runtime=1h --time_based=1

This all works fine, the fio process happily reads from the file.

I then deliberately destroy some data on one RAID member

# dd_rescue -R /dev/urandom -S 1G -m 16M /dev/vdc1

As expected the system recognizes this and very quickly kicks out the member

Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3207640 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: redirecting sector 3207640 to other mirror: dm-3
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: dm-3: rescheduling sector 2626672
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 2626672 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: redirecting sector 2626672 to other mirror: dm-3
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3067160 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3206504 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3987864 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3353064 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3740936 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3950200 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 2524880 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 2408672 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3442760 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: read error corrected (8 sectors at 3896792 on dm-3)
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: dm-3: Raid device exceeded read_error threshold [cur 21:max 20]
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: dm-3: Failing raid device
Mar 08 13:44:22 almalinux-9-test kernel: md/raid1:mdX: Disk failure on dm-3, disabling device.
                                         md/raid1:mdX: Operation continuing on 1 devices.

dmsetup agrees that one member is dead

# dmsetup status vg_raid-lv_raid
0 18874368 raid raid1 2 DA 18874368/18874368 idle 0 0 -

My question is how one is supposed to recover from this. As far as I can make out, the recovery procedure is to call lvchange --refresh (which should get the broken member back into the RAID), followed by lvchange --syncaction check which will scan the RAID for broken sectors and repair them.

My issue is that this doesn't seem to work as long as there is any sort of read IO going on. I can execute

# lvchange --refresh vg_raid/lv_raid

and that causes the below kernel messages

Mar 08 13:53:57 almalinux-9-test kernel: device-mapper: raid: Faulty raid1 device #0 has readable super block.  Attempting to revive it.
Mar 08 13:53:57 almalinux-9-test kernel: md: recover of RAID array mdX
Mar 08 13:53:57 almalinux-9-test kernel: md: mdX: recover done.
Mar 08 13:53:58 almalinux-9-test kernel: device-mapper: integrity: dm-3: Checksum failed at sector 0x1ff4b8
Mar 08 13:53:58 almalinux-9-test kernel: md/raid1:mdX: dm-3: rescheduling sector 2094264
Mar 08 13:53:58 almalinux-9-test kernel: md/raid1:mdX: dm-3: Raid device exceeded read_error threshold [cur 23:max 20]
Mar 08 13:53:58 almalinux-9-test kernel: md/raid1:mdX: dm-3: Failing raid device
Mar 08 13:53:58 almalinux-9-test kernel: md/raid1:mdX: Disk failure on dm-3, disabling device.
                                         md/raid1:mdX: Operation continuing on 1 devices.
Mar 08 13:53:58 almalinux-9-test kernel: md/raid1:mdX: redirecting sector 2094264 to other mirror: dm-7

The failed member gets re-added, but immediately gets kicked out again because there's still IO, and it still encounters errors. And as long as the failed member isn't added back, syncaction will not do anything with it. If there is a way to increase the read_error threshold I haven't found it yet.