lvmcache spends 5 hours flushing clean cache

[xanni]

I have a 64GB SSD writeback cachevol on top of an SMR HDD. When I run lvconvert --splitcache even when the cache is clean with no dirty blocks, it takes around 2 hours to flush the (clean) cache before completing the splitcache. During this time the VG is locked and system performance is very bad - the syslog also reports that:

"kernel: workqueue: do_work hogged CPU for >10000us 131 times, consider switching to WQ_UNBOUND"

And "kernel: INFO: task btrfs-transacti:1899 blocked for more than 122 seconds."

This seems very suboptimal.

Thanks,
Andrew

[xanni]

In the end it actually took 5 hours to flush the cache. A second equally sized cache on a different SSD and SMR HDD took only 30 minutes, but both should have been able to complete in seconds.

[zkabelac]

This looks like plain 'shoot in the dark'.
Please provide more info how your device tree looks like?

Are you running your LV devices directly on top of real physical block devices - or there are some 'loops' and other things involved ?
I'm seeing combination of 'btrfs' with lvm2 - which is also somewhat weird.

It looks very suspicious a clean writeback cache would be removed in 2 hours - however SMR drives are very strange beast with unusual requirements on write patterns - cleanup of writeback cache is not doing 'any' optimizations of write patterns - it strictly follows write barriers.

But for all these things we would need to know way more details for proper diagnostic.

[xanni]

$ lsblk -fMe7
    NAME                     FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
    sda                      LVM2_member LVM2 001       QqVsdt-aP86-rl9e-rfFX-iL8W-NTxB-QEkzVt                
    sdb                      LVM2_member LVM2 001       NBJ2WR-iDx8-Dw8z-9oFG-uMvq-glzs-y5khPI                
    └─vg0-lvol2              btrfs                home  85069ce9-be06-4c92-b8c1-8a0f685e43c6                  
    sdc                      LVM2_member LVM2 001       OeRdpd-Uu8J-4bJf-hu0l-ebRf-2Qnv-xtXCqB                
    ├─vg0-lvol0              btrfs                home  85069ce9-be06-4c92-b8c1-8a0f685e43c6      2.9T    47% /home
┌┈▶ └─vg0-lvol1_corig                                                                                         
┆   sdd                      LVM2_member LVM2 001       jWce9k-Ofpc-tqIt-4d2P-tpzP-0qIt-6ypGsE                
┆   └─vg0-lvol5_cvol                                                                                          
├┈▶   ├─vg0-lvol5_cvol-cdata                                                                                  
└┬▶   └─vg0-lvol5_cvol-cmeta                                                                                  
 └┈┈vg0-lvol1                ext4        1.0      local c03c916c-1b55-487a-bddd-ccc417016f29      1.1T    35% /local

Real physical block devices, no loops. sda and sdd are SSDs, sdb and sdc are SMR HDDs.

[xanni]

This is with the caches split. Once my btrfs scrub completes I will run:

lvconvert -H -c 128k --cachemode writeback -y --cachevol vg0/lvol4 vg0/lvol0
lvconvert -H -c 128k --cachemode writeback -y --cachevol vg0/lvol3 vg0/lvol2

to reattach the caches.

[zkabelac]

Please also add 'lvs -a' and likely system log - at least from device mapper messages.

Wasn't there some unexpected power-failure causing inconsistency in metadata ?

[xanni]

  LV            VG  Attr       LSize  Pool         Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  lvol0         vg0 -wi-ao---- <5.46t                                                                   
  lvol1         vg0 Cwi-aoC--- <1.82t [lvol5_cvol] [lvol1_corig] 100.00 14.77           0.00            
  [lvol1_corig] vg0 owi-aoC--- <1.82t                                                                   
  lvol2         vg0 -wi-ao---- <5.46t                                                                   
  lvol3         vg0 -wi------- 64.00g                                                                   
  lvol4         vg0 -wi------- 64.00g                                                                   
  [lvol5_cvol]  vg0 Cwi-aoC--- 64.00g

[xanni]

Device mapper messages:

2026-02-01T00:06:08.069619+11:00 rainbowdash fstrim[606167]: /local: 1.3 TiB (1432505810944 bytes) trimmed on /dev/mapper/vg0-lvol1
2026-02-01T00:06:08.075679+11:00 rainbowdash fstrim[606167]: /home: 32.2 GiB (34573672448 bytes) trimmed on /dev/mapper/vg0-lvol2
2026-02-01T03:30:42.394847+11:00 rainbowdash mount[624575]: mount: /home: /dev/mapper/vg0-lvol2 already mounted or mount point busy.
2026-02-01T04:30:01.226780+11:00 rainbowdash kernel: device-mapper: cache: Origin device (dm-8) discard unsupported: Disabling discard passdown.
2026-02-01T04:30:03.034045+11:00 rainbowdash systemd[1]: Started dm-event.service - Device-mapper event daemon.
2026-02-01T04:30:28.235778+11:00 rainbowdash kernel: device-mapper: cache: Origin device (dm-8) discard unsupported: Disabling discard passdown.
2026-02-01T05:30:28.378483+11:00 rainbowdash systemd[1]: dm-event.service: Deactivated successfully.
2026-02-01T11:05:04.930691+11:00 rainbowdash kernel: device-mapper: cache: Origin device (dm-3) discard unsupported: Disabling discard passdown.

Note that the splitcache was run just after 11:00 local time, so all but the last syslog entry predate it. There have been no other mapper log entries apart from those issued by the btrfs scrub operation.

[xanni]

If the metadata was inconsistent, shouldn't that be logged somewhere? And is there a way to clean it?

[xanni]

The scrub has finished and it's back to the normal configuration, as it was before the splitcache:

$ lsblk -fMe7
       NAME                     FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
       sda                      LVM2_member LVM2 001       QqVsdt-aP86-rl9e-rfFX-iL8W-NTxB-QEkzVt                
       └─vg0-lvol3_cvol                                                                                          
   ┌┈▶   ├─vg0-lvol3_cvol-cdata                                                                                  
   ├┈▶   └─vg0-lvol3_cvol-cmeta                                                                                  
   ┆   sdb                      LVM2_member LVM2 001       NBJ2WR-iDx8-Dw8z-9oFG-uMvq-glzs-y5khPI                
   └┬▶ └─vg0-lvol2_corig                                                                                         
    └┈┈vg0-lvol2                btrfs                home  85069ce9-be06-4c92-b8c1-8a0f685e43c6                  
       sdc                      LVM2_member LVM2 001       OeRdpd-Uu8J-4bJf-hu0l-ebRf-2Qnv-xtXCqB                
   ┌┈▶ ├─vg0-lvol0_corig                                                                                         
┌┈▶┆   └─vg0-lvol1_corig                                                                                         
┆  ┆   sdd                      LVM2_member LVM2 001       jWce9k-Ofpc-tqIt-4d2P-tpzP-0qIt-6ypGsE                
┆  ┆   ├─vg0-lvol4_cvol                                                                                          
┆  ├┈▶ │ ├─vg0-lvol4_cvol-cdata                                                                                  
┆  └┬▶ │ └─vg0-lvol4_cvol-cmeta                                                                                  
┆   ┆  └─vg0-lvol5_cvol                                                                                          
├┈▶ ┆    ├─vg0-lvol5_cvol-cdata                                                                                  
└┬▶ ┆    └─vg0-lvol5_cvol-cmeta                                                                                  
 ┆  └┈┈vg0-lvol0                btrfs                home  85069ce9-be06-4c92-b8c1-8a0f685e43c6      2.9T    47% /home
 └┈┈┈┈┈vg0-lvol1                ext4        1.0      local c03c916c-1b55-487a-bddd-ccc417016f29      1.1T    35% /local

$ sudo lvs -a
  LV            VG  Attr       LSize  Pool         Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  lvol0         vg0 Cwi-aoC--- <5.46t [lvol4_cvol] [lvol0_corig] 0.16   14.83           0.00            
  [lvol0_corig] vg0 owi-aoC--- <5.46t                                                                   
  lvol1         vg0 Cwi-aoC--- <1.82t [lvol5_cvol] [lvol1_corig] 99.99  14.77           0.00            
  [lvol1_corig] vg0 owi-aoC--- <1.82t                                                                   
  lvol2         vg0 Cwi-aoC--- <5.46t [lvol3_cvol] [lvol2_corig] 0.17   14.83           0.00            
  [lvol2_corig] vg0 owi-aoC--- <5.46t                                                                   
  [lvol3_cvol]  vg0 Cwi-aoC--- 64.00g                                                                   
  [lvol4_cvol]  vg0 Cwi-aoC--- 64.00g                                                                   
  [lvol5_cvol]  vg0 Cwi-aoC--- 64.00g

[zkabelac]

hmm can you grab log of 'dmsetup status' output taken i.e. every 10 seconds for couple minutes.

Also 'iotop' result might be useful.

I'm not expert myself on SMR drives - might be when there is high load spreading updates of writeback cached sectors across whole drive it might take huge amount of time.

Good SMR drive seems to be around 40MB/s - this would mean ~26minutes just to streamline 64G
But when there is high 'seek' load it can go probably much lower when internal SMR buffers are exhausted.

Note - 'online' uncache needs to strictly follow write pattern order.

Possibly some 'offline optimized' uncache that could combine 'areas' together and ignore any write ordering could be really useful here - but that's like for some SMR expert.

[xanni]

I have been unable to replicate this issue - splitcache now flushes 0 blocks and completes immediately. I suspect you may be right that inconsistent metadata from a previous unclean shutdown may have been the cause, but in that case why did lvdisplay and lvs show that there were zero dirty blocks? And why does it then spend 5 hours flushing these invisible dirty blocks?

Surely on booting with an unclean cache, the system should clean it rather than leave it for the next cache deactivation.

[zkabelac]

So I believe things are likely working as expected - SMR drive once they run out of free writeable areas are going to be really slow - those drives are suitable for 'backup' or large drive - but definitely not for 'regular' workflow - you would likely need to use much bigger cache device to fit-in your daily IO workflow.

There can be a difference if the drive would be 'host-managed' - as there can be some control over logic handled by linux driver - but with 'drive-managed' device you are typically left with logic implemented in drive itself. Typically those drives can accommodate couple GiB of 'random' data - but then there starts the game for hunting of re-writable zones and flushing of 64GB might be a big time problem - you could probably look at block tracing how fast IO are served.

ATM it doesn't look like there is any bug on DM side - as it simply waits till kernel confirms, data are safely demoted to origin device - assuming you machine was mostly stuck and idle waiting...

Personally for regular filesystem usage I'd recommend to skip use of SMR drives.

[xanni]

Sure, I understand that SMR drives have very poor performance for large amounts of random write traffic. (Without the LVM cache, it would occasionally block for 20-30 seconds when under heavy write load; with the 64GB cache, it performs very well and the cache spends most of the time clean.)

But I still wonder: if this was caused by lvmcache metadata corruption, where is this reported? And why report the cache as 0% dirty, then wait until cache deactivation to address the issue with a complete flush?

So the bug report is now: given a cachevol reporting 0% dirty, why does it need to demote anything on splitcache, let alone hundreds of thousands of blocks?

[zkabelac]

Which version of kernel and lvm2 is in use? There used to be bad logic many years back in time where plain power-off caused dirtying whole cache - this was later solved with journaling - but this is over 6 or more years (time flies).

If the driver finds any error - it would be logged in your journal - unsure at which date this error might have happened - so you might seek over boot. Messages from DM should always have "device-mapper:" string.