[Question] Can DiskANN support dataset larger than uint32?

[hhy3]

I see that diskann mainly uses uint32_t to manipulate data (like in .bin file), so the maximum dataset diskann can supoprt is 4 billions. I wonder if there's a way to support dataset larger than 4 billions?

[jinwei14]

depends on your machine's SSD and ram budget. you can leverage the disk version to test it (build_disk_index). For 400M data size it took me 2-3 days to build with 60core and 256G memory, I think it will take a lot more time to build with your data size and require a very powerful machine.

in reality I suggest you build the index into different partitions. (eg: 300/500M per part) and route the traffic for search.

[hhy3]

Thanks for you reply. The real problem I'm facing is that the data format diskann uses is .bin file, which use first 4 bytes for num_points, second 4 bytes for dimensions, and other bytes for vectors. So the maximum dataset size for this file format (and diskann can handle) is std::numeric<uint32_t>::max(), which is about 4 billions right? So what I want to know is can diskann handle dataset larger than this limit?