Document how to use crane inside of an overlay

[samestep]

Is your feature request related to a problem? Please describe.
I have a project written in Rust for which I provide an overlay to make it easy to use via, e.g., Home Manager. I'm not confident that I wrote the overlay correctly; currently I use crane.mkLib final, but I can't tell if I should actually be doing crane.mkLib prev instead, or if actually neither are correct and I should do something more subtle.

Describe the solution you'd like
It would be great if the crane docs could describe whether, when defining a Nixpkgs overlay that provides a package built with crane, one should use crane.mkLib final, or crane.mkLib prev, or do something else entirely.

Describe alternatives you've considered
I can't think of any alternatives.

[dpc]

craneLib is a library, so it is not something you'd add to nixpks overlay. AFAICT you are adding a software built with crane to an overlay which is done exactly like adding any other software to an overlay.

[samestep]

@dpc OK, so... sorry, I'm still somewhat new to Nix. Which should I use, crane.mkLib final or crane.mkLib prev? Or do you mean I should be doing neither of those, in which case, what should I be doing instead?

[dpc]

@dpc OK, so... sorry, I'm still somewhat new to Nix. Which should I use, crane.mkLib final or crane.mkLib prev? Or do you mean I should be doing neither of those, in which case, what should I be doing instead?

Yeah, we've all been there, don't worry. :D

Probably final, unless you have some fundamental issues causing circular dependencies. final are the "final" packages (typically named pkgs) after all overlays are composed. So you want to use the final state as much as you can.

[samestep]

@dpc gotcha, thanks; I think I understand that in general. Could you help me understand this specific case in the context of the general advice given in the Nixpkgs manual? (emphasis mine)

The first argument (final, self) corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of rr in the example above come from final, as well as the overridden dependencies used in the boost override.

The second argument (prev, super) corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs. For example, the original recipe of boost in the above example, comes from prev, as well as the callPackage function.

Given that crane.mkLib uses lib from Nixpkgs, this made me think that prev should actually be used instead of final in this case:

crane/lib/default.nix

Line 2 in d9e7531

lib,

Is there some particular reason why that advice doesn't apply here?

[dpc]

Honestly, I'm not 100% sure which one to use here. The problem with final/prev in overlays is that you might accidentally trigger a circular recursion, even between different overlays, as they could be overriding stuff in final that the other side uses for evaluating the override itself. If you're bringing in an extral crate like crane to build some package, I think the chances of that are minimal if not possible at all, so I think final will be fine. If you use prev it will work just as well, just crane will use set of packages that are pre this and other overlays itself.

[ipetkov]

I second everything that has been said earlier. There's nothing crane specific about using overlays, any general documentation around them (and any advice when to use final vs prev) would similarly apply here.

Though happy to review any PRs with additional documentation improvements around this!

[samestep]

@ipetkov thanks, makes sense; so just to clarify, are you saying that similar to @dpc, your stance is also that it's not 100% clear which to use between final and prev? Do you ever write an overlay like the one I linked above, and if so, do you use final or do you use prev? Or if you don't ever write such an overlay, is there a reason why not? Is there a different thing that I should be doing instead?

[ipetkov]

your stance is also that it's not 100% clear which to use between final and prev?

No. My stance is that is is clear which one to use, but you have to take into account:

1. what your package depends on
2. where you expect your package to be consumed downstream (i.e. by callers importing your overlay)
3. whether there are any (evaluation) cycles between the above

If there are no cycles, use final for maximum flexibility/least surprise for further downsteam overlays. Otherwise if there are cycles (you get infinite recursion errors) or you aren't sure if there may be, use prev.

IME for the majority of overlays which define their own packages (i.e. not in nixpkgs, and not replacing anything in nixpkgs) the distinction doesn't really matter so pick whichever works for your use case!

[eureka-cpu]

I've been doing things like this:

      overlays = {
        # Command line tool and nix library for building REAPER extension plugins.
        cargo-reaper = final: prev: {
          inherit (self.inputs.cargo-reaper.packages.${prev.stdenv.hostPlatform.system}) cargo-reaper;
          cargoReaperLib = self.inputs.cargo-reaper.mkLib {
            inherit (prev) lib;
            inherit (final) cargo-reaper;
          };
        };
        # Handles settings rust toolchain policies for building workspace packages.
        crane = final: prev:
          let
            inherit (self.inputs) crane;
            inherit (prev) fenix cargoReaperLib;

            components = [
              "cargo"
              "rustfmt"
              "clippy"
              "llvm-tools"
            ];
          in
          {
            craneStable =
              let
                craneLib = (crane.mkLib prev).overrideToolchain (fenix.stable.withComponents components);
              in
              craneLib // (cargoReaperLib.crane {
                inherit craneLib;
              });
            craneNightly = final.craneStable.overrideToolchain (fenix.complete.withComponents components);
          };
        # Adds workspace packages and metadata for checks to nixpkgs.
        default = import ./nixos/overlays;
      };

Keep in mind, I also am using the fenix overlay for my toolchain. cargo-reaper is a cargo plugin I wrote that extends crane to build REAPER extension plugins, so you can ignore that bit. Then you can use it pretty much like you would in any other overlay. The one I'm currently using this for looks like this:

final: prev:
let
  inherit (prev)
    stdenv
    craneStable
    ;
  inherit (prev.lib) optional optionals;
  inherit (prev.lib.fileset)
    fileFilter
    toSource
    unions
    ;
  inherit (craneStable.fileset) commonCargoSources;
  inherit (craneStable)
    crateNameFromCargoToml
    buildDepsOnly
    buildReaperExtension
    buildPackage
    ;

  # Workspace metadata from the root `Cargo.toml`.
  workspace =
    let
      root = ../../.;
      src = toManifestSource { crate = root; };
    in
    {
      inherit root src;
      inherit (crateNameFromCargoToml { inherit src; }) pname version;
    };

  # Source filtering by crate or workspace.
  toManifestSource =
    let
      docSources = from: fileFilter (file: file.hasExt "md") from;
      testDataSources = from: fileFilter (file: file.hasExt "json" || file.hasExt "RPP") from;
      sqlxSources = from: fileFilter (file: file.hasExt "json" || file.hasExt "sql") from;
      cssSources = from: fileFilter (file: file.hasExt "css") from;
      assetSources = from: fileFilter (file: file.hasExt "ttf" || file.hasExt "svg") from;
    in
    { crate, deps ? [ ] }:
    let
      inherit (workspace) root;
    in
    toSource {
      inherit root;
      fileset = unions ([
        (root + "/.cargo")
        (root + "/Cargo.toml")
        (root + "/Cargo.lock")
        (root + "/.reaper.toml")
        (commonCargoSources crate)
        (commonCargoSources (root + /src/.cargo_hakari))
        (docSources crate)
        (sqlxSources crate)
        (testDataSources crate)
        (cssSources (root + "/site/static/palette"))
        (assetSources (root + "/.assets"))
      ] ++ (map
        (dep: unions [
          (commonCargoSources (root + dep))
          (docSources (root + dep))
          (sqlxSources (root + dep))
        ])
        deps));
    };

  # Defines the necessary dependencies and package information for the build
  buildArgs = {
    inherit (workspace) pname version src;
    strictDeps = true;

    nativeBuildInputs = with prev; [
      llvmPackages.bintools
      libclang.lib
    ] ++ optional stdenv.isLinux prev.autoPatchelfHook;
    buildInputs = optional stdenv.isDarwin prev.libiconv ++ optionals stdenv.isLinux (with prev; [
      gcc.cc
      gcc.cc.lib
    ]);
  };

  # Build *just* the cargo dependencies (of the entire workspace),
  # so we can reuse all of that work (e.g. via cachix) when running in CI
  cargoArtifacts = buildDepsOnly buildArgs;

  individualCrateArgs = buildArgs // {
    inherit cargoArtifacts;
    inherit (workspace) version;
    doCheck = false;
  };
in
{
  inherit
    workspace
    cargoArtifacts
    buildArgs
    ;

  reaper_lib_cloud_reaper = buildReaperExtension (individualCrateArgs // {
    package = "reaper_lib_cloud_reaper";
    src = toManifestSource {
      crate = workspace.root + "/src/reaper_lib_cloud_reaper";
      deps = [
        "/src/libreacloud"
        "/src/libreacloudd"
        "/src/file_system"
        "/src/stitches_client"
      ];
    };
  });

  reacloudd = buildPackage (individualCrateArgs // {
    pname = "reacloudd";
    src = toManifestSource {
      crate = workspace.root + "/src/reacloudd";
      deps = [
        "/src/reacloudctl"
        "/src/libreacloud"
        "/src/libreacloudd"
        "/src/file_system"
        "/src/stitches_client"
      ];
    };
    cargoExtraArgs = "--bin reacloudd";
  });

  reacloudctl = buildPackage (individualCrateArgs // {
    pname = "reacloudctl";
    src = toManifestSource {
      crate = workspace.root + "/src/reacloudctl";
      deps = [
        "/src/libreacloud"
        "/src/libreacloudd"
        "/src/file_system"
        "/src/stitches_client"
      ];
    };
    cargoExtraArgs = "--bin reacloudctl";
  });

  stitches = buildPackage (individualCrateArgs // {
    pname = "stitches";
    src = toManifestSource {
      crate = workspace.root + "/src/stitches";
      deps = [
        "/src/libreacloud"
        "/src/file_system"
        "/src/stitches_client"
      ];
    };
    cargoExtraArgs = "--bin stitches";
  });
}

[eureka-cpu]

For what it's worth, I've wished there was an overlay for some time now, and have re-written variations of this many times.