More out of memory [OOM] errors compared to C++ compilation of `rocksdb`

[gamesguru]

The compilation overhead in the Rust wrapper seems worse as compared with a baseline compilation with cc and make. The subsequent compilation of the main Rust app also goes smoother once RockDB is separately pre-compiled.

I can largely avoid the issue by precompiling RocksDB and linking, i.e., bypassing this wrapper.

I would assume with a dependency of this size, about 500k lines of C++, which generates 5-10 GB of build intermediates on AMD64... I would think there would be some sort of advanced caching mechanism system-wide. At least presented to the user, if not enabled-by-default.

[zaidoon1]

do you see this issue with release builds (cargo build --release) or dev builds only (cargo build) or both?

[gamesguru]

I see it with both, however my team does not use feature flags properly, I think (I have a wip PR to fix it with them). I see tracing-flame and many heavy packages still compiling. What would release mode generally change in the way rocksdb is handled, I'm curious?

I'm happy to re-run some builds with diagnostic or debug output if it might help.

[gamesguru]

I guess the separate question is if it should be tied to the target/ folder at all, since it's such a huge library to build and we sometimes delete that folder just to cleanup or fix caching issues.

And when we do this, I notice especially IDEs, like RustRover, have an incredibly hard time rebuilding it.

If I just install the assembly/shared object and point ld/mold there, RustRover re-initializes much quicker after rm -rf target/.

Also, doing so results in much smaller target/ folders. Probably only 3-4 GB compared to 10 or 15 for the first incremental build with rocksdb in native rust.

[zaidoon1]

What would release mode generally change in the way rocksdb is handled, I'm curious?

you can check this in progress commit as I believe this will explain what you are seeing:

e1dfe59