Bug with btrfs: ./quickbench: line 38: /sys/block/?/stat: No such file or directory & division by zero

[Krysztophe]

Thanks for this nice tool.

Config: Debian12, kernel 6.1.0-32-amd64, single NVMe drive, LVM partitions
Reproduced on Ubuntu 24.04 without LVM

[ Storage info ]
Name      Bus     Size(GB)  QDepth  Model
nvme0n1   nvme       1863        ?  WD_BLACK SN850X 2000GB                  

[ Storage bandwidth - 4k blocks (?) ]
./quickbench: ligne 38: /sys/block/?/stat: Aucun fichier ou dossier de ce type
./quickbench: ligne 38: /sys/block/?/stat: Aucun fichier ou dossier de ce type
./quickbench: ligne 69: _q / _io : division par 0 (le symbole erroné est « _io »)

This happens only on my btrfs partitions (with LVM or not), $fspart cannot be found. (cifs fails too but I'm not surprised.)

The script works on ext4 partitions.

With a set -x before [Storage bandwith]:

[ Storage info ]
Name      Bus     Size(GB)  QDepth  Model
nvme0n1   nvme       1863        ?  WD_BLACK SN850X 2000GB                  
++ stat -c %Hd:%Ld .
+ fspart=0:39
++ head -n1
++ grep -l '^0:39$' /sys/block/dm-0/dev /sys/block/dm-1/dev /sys/block/dm-10/dev /sys/block/dm-11/dev /sys/block/dm-12/dev /sys/block/dm-2/dev /sys/block/dm-3/dev /sys/block/dm-4/dev /sys/block/dm-5/dev /sys/block/dm-6/dev /sys/block/dm-7/dev /sys/block/dm-8/dev /sys/block/dm-9/dev /sys/block/loop0/dev /sys/block/loop1/dev /sys/block/loop2/dev /sys/block/loop3/dev /sys/block/loop4/dev /sys/block/loop5/dev /sys/block/loop6/dev /sys/block/loop7/dev /sys/block/nvme0n1/dev /sys/block/nvme0n1/device/dev /sys/block/nvme0n1/nvme0n1p1/dev /sys/block/nvme0n1/nvme0n1p2/dev /sys/block/nvme0n1/nvme0n1p3/dev
+ fsdev=
+ fsdev=
+ fsdev=
+ fsdev='?'
+ trap 'rm -f dummy' EXIT
+ for bsize in 4k 1M
+ echo

+ echo '[ Storage bandwidth - 4k blocks (?) ]'
[ Storage bandwidth - 4k blocks (?) ]
+ for j in 1 4 16
+ io_bench Writes 1 dd bs=4k status=none if=/dev/zero of=dummy oflag=direct conv=notrunc,fdatasync
+ title=Writes
+ shift
+ jobs=1
+ shift
+ io_stat '?'
+ dev='?'
+ _time=1742293989468164
+ read -r _rio _rmrg _rsct _rtck _wio _wmrg _wsct _wtck _cur _run _qud _dio _dmrg _dsct _dtck
./quickbench: line 38: /sys/block/?/stat: No such file or directory
+ _t0=1742293989468164
+ _io0=0
+ _sct0=0
+ _q0=
+ (( i=0 ))
+ (( i<1 ))
+ sleep 1
+ trap : SIGINT
+ sleep 5
+ dd bs=4k status=none if=/dev/zero of=dummy oflag=direct conv=notrunc,fdatasync
+ (( i++ ))
+ (( i<1 ))
+ wait
+ kill 3010905
+ io_stat '?'
+ dev='?'
+ _time=1742293994469345
+ read -r _rio _rmrg _rsct _rtck _wio _wmrg _wsct _wtck _cur _run _qud _dio _dmrg _dsct _dtck
./quickbench: line 38: /sys/block/?/stat: No such file or directory
+ _t=5001181
+ _io=0
+ _sct=0
+ _q=0
+ io=0
./quickbench: line 69: _q / _io: division by 0 (error token is "_io")
+ rm -f dummy

[mhercelin]

Same problem on Ubuntu 22.04

Error is on variable fspart=$(stat -c %Hd:%Ld .)
On my Ubuntu, result is
?d:?d

Stat format doesn't seem to be supported.

[zerodeux]

Thanks for your report !

@Krysztophe Could I have the output of ls /sys/block and lsblk from your system ?

@mhercelin: indeed, I checked and it only works on Debian >= 12 / coreutils >= 9.1. I'll see if I can find a workaround.

[Krysztophe]

NB: I forgot to say that I have encryption of the whole disk, but it should be a problem, as my other computer is not encypted and has the same problem.

sudo ls -l /sys/block
total 0
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-10 -> ../devices/virtual/block/dm-10
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-11 -> ../devices/virtual/block/dm-11
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-12 -> ../devices/virtual/block/dm-12
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-5 -> ../devices/virtual/block/dm-5
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-6 -> ../devices/virtual/block/dm-6
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-7 -> ../devices/virtual/block/dm-7
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-8 -> ../devices/virtual/block/dm-8
lrwxrwxrwx 1 root root 0 18 mars  13:10 dm-9 -> ../devices/virtual/block/dm-9
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 18 mars  13:10 loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 18 mars  13:09 nvme0n1 -> ../devices/pci0000:00/0000:00:06.0/0000:01:00.0/nvme/nvme0/nvme0n1

lsblk
NAME                              MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
nvme0n1                           259:0    0  1,8T  0 disk  
├─nvme0n1p1                       259:1    0  512M  0 part  /boot/efi
├─nvme0n1p2                       259:2    0  488M  0 part  /boot
└─nvme0n1p3                       259:3    0  1,8T  0 part  
  └─nvme0n1p3_crypt               253:0    0  1,8T  0 crypt 
    ├─somevgname-root               253:1    0 29,3G  0 lvm   /
    ├─somevgname-var                253:2    0 19,3G  0 lvm   /var
    ├─somevgname-swap_1             253:3    0   69G  0 lvm   [SWAP]
    ├─somevgname-tmp                253:4    0 11,9G  0 lvm   /tmp
    ├─somevgname-home               253:5    0  390G  0 lvm   /home
    ├─somevgname-PGDATA1            253:6    0  333G  0 lvm   /var/lib/postgresql
    ├─somevgname-PGDATA2            253:7    0   30G  0 lvm   /var/lib/postgresql2
    ├─somevgname-docker             253:8    0   50G  0 lvm   /var/lib/docker
    ├─somevgname-machinesvirtuelles 253:9    0  195G  0 lvm   /var/lib/images
    ├─somevgname-pglog              253:10   0   10G  0 lvm   /var/log/postgresql
    ├─somevgname-lvol0              253:11   0   10G  0 lvm   /var/lib/postgresql/vol10
    └─somevgname-lvol1              253:12   0    2G  0 lvm   /var/lib/postgresql/vol2

And I do not find the 0:39 major/minor of the btrfs partition here:

ll /dev/block/
total 0
drwxr-xr-x  2 root root  540 18 mars  13:10 ./
drwxr-xr-x 21 root root 4,3K 18 mars  16:53 ../
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:0 -> ../dm-0
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:1 -> ../dm-1
lrwxrwxrwx  1 root root    8 18 mars  13:10 253:10 -> ../dm-10
lrwxrwxrwx  1 root root    8 18 mars  13:10 253:11 -> ../dm-11
lrwxrwxrwx  1 root root    8 18 mars  13:10 253:12 -> ../dm-12
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:2 -> ../dm-2
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:3 -> ../dm-3
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:4 -> ../dm-4
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:5 -> ../dm-5
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:6 -> ../dm-6
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:7 -> ../dm-7
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:8 -> ../dm-8
lrwxrwxrwx  1 root root    7 18 mars  13:10 253:9 -> ../dm-9
lrwxrwxrwx  1 root root   10 18 mars  13:10 259:0 -> ../nvme0n1
lrwxrwxrwx  1 root root   12 18 mars  13:10 259:1 -> ../nvme0n1p1
lrwxrwxrwx  1 root root   12 18 mars  13:10 259:2 -> ../nvme0n1p2
lrwxrwxrwx  1 root root   12 18 mars  13:10 259:3 -> ../nvme0n1p3
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:0 -> ../loop0
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:1 -> ../loop1
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:2 -> ../loop2
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:3 -> ../loop3
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:4 -> ../loop4
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:5 -> ../loop5
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:6 -> ../loop6
lrwxrwxrwx  1 root root    8 18 mars  13:10 7:7 -> ../loop7

[firecat53]

Same error on ZFS/NixOS 24.11:

[ Storage info ]
Name      Bus     Size(GB)  QDepth  Model
nvme0n1   nvme       1863        ?  SHPP41-2000GM
nvme1n1   nvme       1863        ?  SHPP41-2000GM
sda       scsi        931       32  CT1000MX500SSD1
sdb       scsi        931       32  CT1000MX500SSD1
sdc       scsi       3726       32  WDC WD4003FFBX-6
sdd       scsi       3726       32  WDC WD4003FFBX-6
sde       scsi       3726       30  CT4000X9SSD9

[ Storage bandwidth - 4k blocks (?) ]
./quickbench: line 38: /sys/block/?/stat: No such file or directory
./quickbench: line 38: /sys/block/?/stat: No such file or directory
./quickbench: line 69: _q / _io: division by 0 (error token is "_io")

$ ls /sys/block/
dm-0  dm-1  loop0  loop1  loop2  loop3  loop4  loop5  loop6  loop7  nvme0n1  nvme1n1  sda  sdb  sdc  sdd  sde
$ lsblk
NAME                                               MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                                  8:0    0 931.5G  0 disk
├─sda1                                               8:1    0 931.5G  0 part
└─sda9                                               8:9    0     8M  0 part
sdb                                                  8:16   0 931.5G  0 disk
├─sdb1                                               8:17   0 931.5G  0 part
└─sdb9                                               8:25   0     8M  0 part
sdc                                                  8:32   0   3.6T  0 disk
├─sdc1                                               8:33   0   3.6T  0 part
└─sdc9                                               8:41   0     8M  0 part
sdd                                                  8:48   0   3.6T  0 disk
├─sdd1                                               8:49   0   3.6T  0 part
└─sdd9                                               8:57   0     8M  0 part
sde                                                  8:64   0   3.6T  0 disk
└─sde1                                               8:65   0   3.6T  0 part  /mnt/restic
nvme0n1                                            259:0    0   1.8T  0 disk
├─nvme0n1p1                                        259:2    0     1G  0 part  /boot/efis/nvme-eui.ace42e0026eed9c82ee4ac0000000001-part1
├─nvme0n1p2                                        259:3    0     4G  0 part
├─nvme0n1p3                                        259:4    0   1.8T  0 part
├─nvme0n1p4                                        259:5    0     8G  0 part
│ └─dev-disk-byx2did-nvmex2deui.ace42e0026eed9c82ee4ac0000000001x2dpart4
│                                                  254:0    0     8G  0 crypt [SWAP]
└─nvme0n1p5                                        259:6    0  1000K  0 part
nvme1n1                                            259:1    0   1.8T  0 disk
├─nvme1n1p1                                        259:7    0     1G  0 part  /boot/efis/nvme-eui.ace42e0035b8defe2ee4ac0000000001-part1
├─nvme1n1p2                                        259:8    0     4G  0 part
├─nvme1n1p3                                        259:9    0   1.8T  0 part
├─nvme1n1p4                                        259:10   0     8G  0 part
│ └─dev-disk-byx2did-nvmex2deui.ace42e0035b8defe2ee4ac0000000001x2dpart4
│                                                  254:1    0     8G  0 crypt [SWAP]
└─nvme1n1p5                                        259:11   0  1000K  0 part

$ mount|rg /root
rpool/nixos/root on / type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/nixos/root on /nix/store type zfs (ro,relatime,xattr,posixacl,casesensitive)
bpool/nixos/root on /boot type zfs (rw,relatime,xattr,posixacl,casesensitive)

[BlackLotus]

Copy pasted from here https://lobste.rs/s/ghkg71/quickbench_zero_dependency_linux

Waaaay too many assumptions and bad handling when it comes to devices/fs. The dev likes to use ? as a placeholder when the script doesn’t work as expected, but never checks when it is necessary. For instance

fspart=$(stat -c %Hd:%Ld .)  # ex: 254:1
fsdev=$(grep -l "^$fspart\$" /sys/block/*/dev /sys/block/*/*/dev |head -n1)  # ex: /sys/block/vda/vda1/dev
fsdev="${fsdev##/sys/block/}"
fsdev="${fsdev%%/*}"
fsdev="${fsdev:-?}"

doesn’t work on btrfs (afaik) and using df -T or findmnt -T would be easier and more reliable.

[zerodeux]

Thanks for all the info.

The idea for the IO bench is to have a simple and intuitive interface : you benchmark the block device backing the filesystem you're sitting on (in "current working directory" terms). This way you simply cd to the place which is meaningful for you to benchmark, and quickbench adapts. Also most of the time you don't have direct access to a block device, thus we benchmark through the filesystem.

Turns out that resolving a filesystem path as a blockdevs not easy :

• stat -c %Hd:%Ld needs a recent coreutils (works in 9.1)
• the same command does not work for btrfs, because btrfs exposes pseudo device major/minors (they're not linked to the real backing device)

I'll try to figure out another way.

NB: I forgot to say that I have encryption of the whole disk, but it should be a problem, as my other computer is not encypted and has the same problem.

I have an encrypted disk (luks2 + LVM + ext4) and it works for me. I concur that encryption does not seem to be a problem.

[zerodeux]

The dev likes to use ? as a placeholder when the script doesn’t work as expected, but never checks when it is necessary.

That needs some error checking, ineed. Adding that right now.

doesn’t work on btrfs (afaik) and using df -T or findmnt -T would be easier and more reliable.

I thought about this, but 1/ it does not return a canonical blockdev name (eg. /dev/mapper/bla instead of dm-X), 2/ it does not work with all filesystems (just checked: also returns emptyness for btrfs, sigh...), 3/ I always favor the command-less approach if possible. The /sys exploration is not unreasonable in this context.

[zerodeux]

Oh my bad, findmnt -T does work with btrfs (I historically tried findmnt alone, which did not).

It works for the stats collection (having the proper blockdevice to monitor), but does not make for a clear display since we benchmark a 'dm-0' instead of a 'nvme0n1' on LVM/luks/md stacks.

Fortunately lsblk which is in the same util-linux package as findmnt can clearly show what you're benchmarking through your fs, eg. for a luks-over-md-raid5 :

# lsblk -s /dev/dm-0
NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
data    253:0    0 21.8T  0 crypt /srv
└─md1     9:1    0 21.8T  0 raid5 
  ├─sda   8:0    0  5.5T  0 disk  
  ├─sdb   8:16   0  5.5T  0 disk  
  ├─sdc   8:32   0  5.5T  0 disk  
  ├─sdd   8:48   0  5.5T  0 disk  
  ├─sde   8:64   0  5.5T  0 disk  
  └─sdf   8:80   0  5.5T  0 disk  

[zerodeux]

@Krysztophe + @mhercelin : would you care to test the new version and tell me if it works for you ?

[firecat53]

On ZFS still an error:

[ Storage info ]
Name      Bus     Size(GB)  QDepth  Model
nvme0n1   nvme       1863        ?  SHPP41-2000GM                           
nvme1n1   nvme       1863        ?  SHPP41-2000GM                           
sda       scsi        931       32  CT1000MX500SSD1 
sdb       scsi        931       32  CT1000MX500SSD1 
sdc       scsi       3726       32  WDC WD4003FFBX-6
sdd       scsi       3726       32  WDC WD4003FFBX-6
sde       scsi       3726       30  CT4000X9SSD9    
Could not figure out the backing block device for /home/firecat53/.local/tmp, sorry

[Krysztophe]

With commit 7640b83:

• On a btrfs LVM partition:

[ Storage bandwidth - 4k blocks ]
Writes ( 1 thread[s]):  29444 MB/s  7143640 IO/s       0 µs avg. lat.
Writes ( 4 thread[s]):  73039 MB/s  14259171 IO/s       0 µs avg. lat.
Writes (16 thread[s]):  73294 MB/s  17824899 IO/s       0 µs avg. lat.
Reads  ( 1 thread[s]):  37862 MB/s  9243301 IO/s       0 µs avg. lat.
Reads  ( 4 thread[s]):  43961 MB/s  10729655 IO/s       0 µs avg. lat.
Reads  (16 thread[s]):  33807 MB/s  7422212 IO/s       0 µs avg. lat.

• On a CIFS mount:

Could not figure out the backing block device for /home/chr/@mono/tmp, sorry

• On a encfs fuse mount:

Could not figure out the backing block device for /home/taf/demo, sorry

• On tmpfs (/run):

Could not figure out the backing block device for /run, sorry

For such a simple tool, I won't ask for more. 👍

[zerodeux]

Ok frankly I wouldn't benchmark a CIFS mount with this stool (it's not part of the device itself) and FUSE and tmpfs are also pretty out of scope, most of the time they don't have a backing block device.

[zerodeux]

@firecat53 : what is the outpout of findmnt -T /home/firecat53/.local/tmp for you ? And if it has a non-empty output, what do you get with readlink <SOURCE value> ?