"vdo_use_sparse_index" and "vdo_index_memory_size_mb" incompatible values ? Gives "VDO Status: Out of space" error

[tigerblue77]

Hello there,

I'm encountering a problem with the following LVM VDO metadata profile:

# Custom configuration for VDO using compression/deduplication and more CPUs (based on local vCPU count)
# Custom parameters based on: https://is.muni.cz/th/rq7e2/petrovic_diploma_thesis.pdf

allocation {
  vdo_use_compression=1
  vdo_use_deduplication=1
  vdo_use_metadata_hints=1
  vdo_minimum_io_size=4096
  vdo_block_map_cache_size_mb=16384
  vdo_block_map_period=16380
  vdo_check_point_frequency=0
  vdo_use_sparse_index=1
  vdo_index_memory_size_mb=3072
  vdo_slab_size_mb=8192
  vdo_ack_threads=$((VCPU_COUNT/4))
  vdo_bio_threads=$VCPU_COUNT
  vdo_bio_rotation=64
  vdo_cpu_threads=$((VCPU_COUNT/2))
  vdo_hash_zone_threads=$((VCPU_COUNT/4))
  vdo_logical_threads=$((VCPU_COUNT/4))
  vdo_physical_threads=1
  vdo_write_policy=\"sync\"
  vdo_max_discard=1
}

FYI, on my system:

VCPU_COUNT=8

When I run lvcreate using the previous metadata profile:

DESTINATION_LVM_VG="/dev/Pepper-Potts-vg"
VDOLV_NAME="COMPRESSED-DEDUPLICATED-VDOLV-1"

lvcreate -y --vdopool "COMPRESSED-DEDUPLICATED-VDOPOOL-1" --name $VDOLV_NAME --size 12G --virtualsize 24G --metadataprofile $LVM_METADATA_PROFILE_NAME $DESTINATION_LVM_VG

I get the following error:

vdoformat: formatVDO failed on '/dev/Pepper-Potts-vg/COMPRESSED-DEDUPLICATED-VDOPOOL-1': VDO Status: Out of space
  Command /usr/bin/vdoformat failed.
  Cannot format VDO pool volume Pepper-Potts-vg/COMPRESSED-DEDUPLICATED-VDOPOOL-1.

But I do have sufficient free space on volume group, here is the output of vgs command:

VG              #PV #LV #SN Attr   VSize   VFree
Pepper-Potts-vg   1   5   0 wz--n- <99,52g <84,63g

I did some tests and found that the problem comes from these lines :

vdo_use_sparse_index=1

and

vdo_index_memory_size_mb=3084

Here are the results of my different tests:

		vdo_index_memory_size_mb value
		not set	256 (default value)	512	1024	2048	3072
vdo_use_sparse_index value	0	✅	✅	❌❌	❌	❌	❌
	1 (default value)	❌	❌	❌	❌	❌	❌
	not set	✅	❌	❌	❌	❌	❌

In "❌❌" case, I get a different error:

vdoformat: parseIndexConfig failed: Invalid argument
  Command /usr/bin/vdoformat failed.
  Cannot format VDO pool volume Pepper-Potts-vg/COMPRESSED-DEDUPLICATED-VDOPOOL-1.

I feel that I'm doing something wrong but don't really know what 😅 thanks per advance !

[lorelei-sakai]

This is due to the way that the index size requirements are specified. The vdo_index_memory_size_mb parameter represent the amount of memory (RAM) that the index requires. Using 3072 means that the index should require around 3GB of memory. The index also stores its contents of disk, though. As a general rule of thumb, the amount of storage space required for a standard (dense) index is around ten times the amount of RAM. So using a size of 3072 would require around 30GB of storage space for just the index. On top of that, a sparse index uses the specified amount of RAM, but requires ten times more storage space than a similar dense index. So in this case, the sparse index requires around 300GB of storage for the index. Since your device only has around 84GB free, vdoformat cannot even create the layout for the index. (Out of space here means that your device does not have enough space for the layout that VDO tries to create.)

As a side note, experimentally it seems that most volumes don't gain much, if any, deduplication improvement from using an index larger than the default of 256MB. And the default vdo_use_sparse_index should be 0; I would recommend using that. Using a 3GB index would almost certainly eat up memory and storage space without providing much benefit to the device. Keep in mind that any space used for the index is over and above space actually used to store user data in the volume.

I am a little surprised that vdo_use_sparse_index=0 and vdo_index_memory_size_mb=1024 does not work. It appears to me that you should have enough storage space for that configuration.

With regard to the XX error you mention at the end, 512 should be a valid size for an index, but there was a bug recently that we didn't parse it properly. I'd have to check whether we published the fix for that issue.

[tigerblue77]

Out of space here means that your device does not have enough space for the layout that VDO tries to create.

Oh okay... Didn't have any idea about those calculations. Maybe error message should be more detailed/explicit ?

the default vdo_use_sparse_index should be 0; I would recommend using that

I was thinking to use sparse index because it's the official RedHat recommendation... ? "The UDS Sparse Indexing feature is the recommended mode for VDO"

I am a little surprised that vdo_use_sparse_index=0 and vdo_index_memory_size_mb=1024 does not work. It appears to me that you should have enough storage space for that configuration.

I just tested again both 512/1024 cases, to be sure, as it was late when I wrote this yesterday, and yes, it doesn't seem to work, exactly as I explained first.

there was a bug recently that we didn't parse it properly. I'd have to check whether we published the fix for that issue.

Okay thanks for checking. Can I ask you how are you working on VDO/KVDO projects ? Because I don't see any recent commit on those Github repositories so... Do you have another private git repository or... Are these projects paused/stopped ?

It's a bit hard to find good documentation about VDO except:

• Redhat's, that doesn't explain much, for example everything about LVM integration
• LVMVDO man page which is good but miss details

Do you have any interesting other documentation source to share with me ?

[lorelei-sakai]

Out of space here means that your device does not have enough space for the layout that VDO tries to create.

Oh okay... Didn't have any idea about those calculations. Maybe error message should be more detailed/explicit ?

The vdoformat utility should report the amount of space required for a given configuration, if it does not have enough space. However, I am not certain if this is visible to you or if lvm hides it. (There might be information in the system logs; search for "vdo" or "uds".)

the default vdo_use_sparse_index should be 0; I would recommend using that

I was thinking to use sparse index because it's the official RedHat recommendation... ? "The UDS Sparse Indexing feature is the recommended mode for VDO"

I see that, but I think in this case the documentation is wrong. For a smaller volume the sparse index is unlikely to help and will only consume space.

there was a bug recently that we didn't parse it properly. I'd have to check whether we published the fix for that issue.

Okay thanks for checking. Can I ask you how are you working on VDO/KVDO projects ? Because I don't see any recent commit on those Github repositories so... Do you have another private git repository or... Are these projects paused/stopped ?

There is active work but it is in a private repository currently. We are in the process of transferring the project to a more public repository, and that process means we have not prioritized updating this repository. And this particular issue has not been fixed as of the most recent update here.

You can also try an index size of 768, which might be enough smaller that 1024 that it might work for you.

It's a bit hard to find good documentation about VDO except:

* [Redhat's](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_storage/index), that doesn't explain much, for example everything about LVM integration

* [LVMVDO man page](https://manpages.debian.org/testing/lvm2/lvmvdo.7.en.html) which is good but miss details

Do you have any interesting other documentation source to share with me ?

I was in fact just looking to see what documentation is available. Unfortunately I couldn't find anything aside from those two sources you mentioned. I think these are somewhat out-of-date, and I acknowledge that they do not do a good job of explaining the UDS storage requirements.