Building FairSoft requires too much storage

[YanzhaoW]

Problem

Hi,

I am having a problem while building a docker image of FairSoft in the public github runners. Since last latest version of ROOT (3.36.04), the building process fails because it uses up too much storage, which exceeds the Github storage quota. The main storage usage is from ROOT and Geant4.

Solution

The storage usage can be reduced if the build tree files (in the build folder) are immediately deleted after a library is installed. At the current version, in the end of the build process, both the built tree and install tree exist, which doubles the amount of storage required by all the libraries.

Another solution is to separate the build process of ROOT and Geant4, such that users could just delete the build tree after the installation of each library.

[dennisklein]

Each pkg (lower-case name as listed in the readme) maps to a make target, perhaps you could try this:

cmake --build $BUILDDIR --target root
# delete some big files
cmake --build $BUILDDIR # continue build

Not sure, if this will work. One needs to care not to delete files that may trigger a rebuild.

What do you think?

Can you elaborate how you would like to split the build in terms of the user interface?

[YanzhaoW]

Hi,

Hmm, I think it definitely triggers the CMake to build ROOT again.

Maybe we could have a flag to exclude ROOT from the build?

But are they any other packages that are relying on ROOT to build? If so, we also need to let those packages to find ROOT from other locations.

[fuhlig1]

@YanzhaoW,

I did a quick test and currently it looks like you are correct. At least if I remove the complete build directory of ROOT the package is compiled again. I will have a look into the issue.

Any package which relies on another one should use the information from the install directory not the build directory, so in principle removing the build directory after installation should work.

I will have a look into the issue but probably not this week.

[dennisklein]

I drafted a CI_BUILD_MODE based on your initial comment to delete build dirs after the per pkg install step. You can find it in https://github.com/dennisklein/FairSoft/tree/ci-build-mode, specifically dennisklein@6675de0. Let me know what you think.

[YanzhaoW]

Thanks, I will try the change and see if it works.

[YanzhaoW]

@dennisklein

Does this still work if I build it using multiple threads, like

cmake -DCMAKE_INSTALL_PREFIX=${SIMPATH} -DUSE_ROOT_VERSION=${ROOT_VERSION} -DCI_BUILD_MODE=ON -GNinja ..
ninja -j4

[dennisklein]

yes, only that ninja will not work very well, unless you use the Kitware patched version that adds jobserver support. For FairSoft I recommend using the default Makefile generator.

[YanzhaoW]

Ok, thanks for the info. But could you explain why ninja would fail in this case?

[dennisklein]

The CMake-based "legacy" build system produces a top-level makefile which calls the makefile per pkg built. If you naively pass -j <n> to all those make instances, it may overload your system as each make will fan out <n> workers. The GNU Make jobserver protocol fixes this, because a nested make will discover it is nested beneath another make and lets the top-level make instance schedule all the workers, limited by <n> for the whole make process tree.

Unfortunately, upstream ninja developers never merged this feature (IIRC they think the jobserver protocol is fundamentally racy and flawed). There is a fork maintained by Kitware (the company who built CMake) that implements the missing jobserver protocol: https://github.com/Kitware/ninja

[YanzhaoW]

Ok, if I understand correctly, this is only about the thread scheduler for each package.

What I'm worrying about is when I use make -j4, ROOT package and Geant4 package could be built at the same time. If so, the storage issue is still not solved.

Do your changes also make sure packages are built sequentially (though each package can use multiple cores for the building) if I use make -jN?

[dennisklein]

Do your changes also make sure packages are built sequentially (though each package can use multiple cores for the building) if I use make -jN?

root and geant4 are not serialized in my branch, you would e.g. have to add geant4 to roots dependency list here:
https://github.com/dennisklein/FairSoft/blob/6675de0154fa84b4a51f32ec61420470c04d791e/cmake/legacy.cmake#L494

[YanzhaoW]

I see. Thanks. Maybe you should also add the root dependency to Geant4 in your branch if CI_BUILD_MODE is enabled?

[dennisklein]

Sure, let's first wait if it helps for your use case, I will then make a proper feature PR to FairSoft and we can finalize it. Perhaps you find additional issues.

[YanzhaoW]

Thanks, the docker container is now built successfully in the Github runner.

Closing the issue here.