llvmPackages should be more easily overridable

[LunNova]

llvmPackages doesn't support overrideScope so it's a pain to override its parts. If you override libllvm you need to manually override everything else that references it including patches. It's easy to accidentally allow a non-overridden copy of libllvm to sneak in.

Example of not-fun overriding lots of separate packages in https://github.com/NixOS/nixpkgs/blob/0f5740f6d9c3d0feb3da06e1da2383204fd6f92e/pkgs/development/rocm-modules/6/llvm/default.nix

There seems to be some support for overriding with .override but it's not clear how to use it. You can't use

llvmPackages.overrideScope (final: prev: {
libllvm = prev.libllvm.overrideAttrs { ... };
})

https://github.com/NixOS/nixpkgs/blob/0f5740f6d9c3d0feb3da06e1da2383204fd6f92e/pkgs/development/compilers/llvm/common/default.nix#L30-L34

cc @dtzWill @Ericson2314 @lovek323 @alyssais @RossComputerGuy @rrbutani @sternenseemann

[RossComputerGuy]

This is a future project of mine that I plan on doing, I haven't had the time to focus on it yet and there's more important things I've been doing.

[LunNova]

@RossComputerGuy do you have a vague idea when you expect to get to this? I could give it a try if you don't anticipate getting to it for a long time.

[RossComputerGuy]

do you have a vague idea when you expect to get to this?

Probably after the lib.systems PR, bolt setup hook, and CPU model improvements PR are merged so probably during the summer.

[reckenrode]

This would be of interest to the Darwin bootstrap. It would allow the bootstrap-specific hacks in the LLVM derivations to be removed. It might even make it possible to rely on the wrapper definitions in LLVM instead of inlining them into the bootstrap.

[RossComputerGuy]

Yeah, also to add what I mentioned on Matrix. I'm thinking of staging LLVM similarly to how we do it for the stdenv bootstrap. This will likely be the bulk of the work but I would like to work on after some of the other PR's I have are done. The reason being is merge conflicts. However, I am able to start some of the pre-work. The first thing being, moving patches from pkgs/development/compilers/llvm/common/default.nix to each package's default.nix. This is to make the packages more "standalone" from the LLVM common default.nix.

[RossComputerGuy]

Decided to nixfmt everything for LLVM in #376179 since I'm getting tired of looking at the old formatting. This will make the next set of work easier to perform.

[pwaller]

There seems to be some support for overriding with .override but it's not clear how to use it. You can't use:

That's correct. It does allow you to get an overridden llvmPackages, but only for things that are fed through the inputs. So you can't straightforwardly override libllvm's attrs in the way you want. It is possible, but it not straightforward, especially in various cross scenarios. It would be good to make it so.

[pwaller]

It seems to me there is a bit of a rats nest of problems here. Incidentally I bumped into a problem caused by this, and excacerbated by the fact that .override does not compose. I think (foo.override { lost = 1; }).override { wins = 1; } has last-override-wins semantics, which further complicates writing composable overlays.

I wonder if this really needs thinking about in broader terms, for example:

• Bring all packaging layers into a single fixpoint #273815
• Unify pkg.override and drv.overrideAttrs #295039
• overridding arguments to cudnn is difficult #116691

I have a specific use case for this right now which is dropping patches from llvmPackages_git which have landed upstream. If you do something like drop a patch from compiler-rt, and drop a patch from clang, it is difficult to get a pkgsLLVM which is correct, without bumping into issues like infinite recursion.

For this usecase I was wondering about introducing an llvmPackages argument which enables patch filtering as a band aid. Perhaps it would be useful to collect other specific use cases here.

[greg-hellings]

It seems to me there is a bit of a rats nest of problems here. Incidentally I bumped into a problem caused by this, and excacerbated by the fact that .override does not compose. I think (foo.override { lost = 1; }).override { wins = 1; } has last-override-wins semantics, which further complicates writing composable overlays.

I wonder if this really needs thinking about in broader terms, for example:

* [Bring all packaging layers into a single fixpoint #273815](https://github.com/NixOS/nixpkgs/issues/273815)

* [Unify pkg.override and drv.overrideAttrs #295039](https://github.com/NixOS/nixpkgs/issues/295039)

* [overridding arguments to cudnn is difficult #116691](https://github.com/NixOS/nixpkgs/issues/116691)

I have a specific use case for this right now which is dropping patches from llvmPackages_git which have landed upstream. If you do something like drop a patch from compiler-rt, and drop a patch from clang, it is difficult to get a pkgsLLVM which is correct, without bumping into issues like infinite recursion.

For this usecase I was wondering about introducing an llvmPackages argument which enables patch filtering as a band aid. Perhaps it would be useful to collect other specific use cases here.

For me, I want to fix a compile error in the pkgsCross.ucrtAarch64 target. The PR for the build fix has fallen completely silent for the past 4 weeks, but I don't want that to stall my progress in actually using the packages to build my downstream software.

I have been headbutting against the version of llvmPackages.libunwind and trying to override it with the version from my branch, but to no avail. It would be great if it was relatively easy to grab my patched version of the derivation and use it downstream until the PR unsticks.