lib.customisation.makeOverridable: lose `override` after `<scope>.overrideScope` overriding

[ShamrockLee]

Description

lib.customisation.makeOverridable not only adds a new overrider override to the result set, but also takes care of existing overriders overrideAttrs and overrideDerivation, ensuring that they works together with the added override.

However, it doesn't take into account some other potentially existing overriders, such as overrideScope and extend, causing the override result of overrideScope and extend to lose the override attribute.

Reproduction steps

Reproduction with nix repl at the nixpkgs project root:

nix-repl> lib = import ./lib                                                    
nix-repl> pkgs = import ./. { }                                                 
nix-repl> scope = lib.makeOverridable ({ lib }: lib.makeScope lib.callPackageWith (final: { a = 1; b = 2; })) { inherit lib; }

nix-repl> scope
{
  a = 1;
  b = 2;
  callPackage = «lambda callPackageWith @ /path/to/nixpkgs/lib/customisation.nix:386:15»;
  newScope = «lambda newScope @ /path/to/nixpkgs/lib/customisation.nix:745:20»;
  override = { ... };
  overrideDerivation = «lambda overrideDerivation @ /path/to/nixpkgs/lib/customisation.nix:313:32»;
  overrideScope = «lambda overrideScope @ /path/to/nixpkgs/lib/customisation.nix:747:25»;
  packages = «lambda @ «string»:1:67»;
}

nix-repl> scope.override
{
  __functionArgs = { ... };
  __functor = «lambda __functor @ /path/to/nixpkgs/lib/trivial.nix:1009:17»;
}

nix-repl> scope.overrideScope
«lambda overrideScope @ /path/to/nixpkgs/lib/customisation.nix:747:25»

nix-repl> scope' = scope.overrideScope (final: previous: { c = final.a + final.b; })                                           

nix-repl> scope'
{
  a = 1;
  b = 2;
  c = 3;
  callPackage = «lambda callPackageWith @ /path/to/nixpkgs/lib/customisation.nix:386:15»;
  newScope = «lambda newScope @ /path/to/nixpkgs/lib/customisation.nix:745:20»;
  overrideScope = «lambda overrideScope @ /path/to/nixpkgs/lib/customisation.nix:747:25»;
  packages = «lambda extends @ /path/to/nixpkgs/lib/fixed-points.nix:323:7»;
}

nix-repl> scope'.overrideScope
«lambda overrideScope @ /path/to/nixpkgs/lib/customisation.nix:747:25»

nix-repl> scope'.override     
error: attribute 'override' missing
       at «string»:1:1:
            1| scope'.override
             | ^

Additional context

@LunNova discovered this bug when working on PR #445668. See the comment for detail: #445668 (comment)

[emilazy]

I have been thinking about something along the lines of the “single fixpoint” idea from #273815 as applied more generally to .override/.overrideAttrs/.overrideScope (whereas that issue only considers stdenv.mkDerivation), to give s a single consistent override/overlay interface that handles all these layers in a unified manner, is clearer about what counts as public vs. private API (.override currently mixes intended‐as‐public flags and dependency injection interfaces with what is essentially monkey‐patching), and is more flexible (keeping .override interfaces stable is awkward right now and its lack of clear namespacing is unfortunate, and callPackage clobbers .override in ways that can be somewhat unfortunate).

This is also relevant to moving overrides into pkgs/by-name and for #442066.

Unfortunately these thoughts are somewhat half‐formed right now, although I have hacked on a bunch of test files trying to work out what things should look like :) I hope you don’t mind me picking your brain about this, and let me know if there’s a more appropriate issue I should be doing it on.

AIUI, you would prefer things like .overridePythonAttrs to not exist, in favour of every attribute being passed down to stdenv.mkDerivation so that .overrideAttrs can be used directly, right? And presumably similarly about the Haskell package set’s mkDerivation, which rather unfortunately exposes something .overrideAttrs‐shaped as .override right now.

I’m wondering how this extends to .overrideScope in your view, though; scopes aren’t derivations, but they are fixed points, so I suppose they could have an .overrideAttrs all the same? (With structured aliases, it is quite desirable to have a distinction between the “scope member definitions” and the “realized scope definitions”, as the final form of an alias may be a bare throw, so the equivalent of finalAttrs.finalPackage would be quite nice to have.)

(Other interesting cases are things like fetchFromGitHub and various generic builders for multiple variants of a single package – which have .overrides that are somewhat .overrideAttrs-y, and yet also have quite high‐level interfaces that feel distinct from ones that allow you to put whatever you want into any phase. Deciding what counts as public vs. private override API outside of the context of a concrete package seems difficult.)