feat: init nix flake

[ConcurAc]

Proposal of initial module and package structure for nix flake. Nix modules are contained in the nix/ folder.

Addresses #437

Currently includes support for NixOS (client + server modules) and home-manager (client module), through separate modules inside nix/nixos/ and nix/home-manager/ respectively.

Retrom packages are stored in nix/pkgs/<name>/package.nix

Flake currently implementing the following...

pkgs:

• retrom-service
• retrom-unwrapped
• retrom: wrapper currently only allowing WEBKIT_DISABLE_DMABUF_RENDERER to be set

modules:

• server (services.retrom): configure the retrom service and enable postgresql integration declaratively
• client (programs.retrom): add the client to the system environment, wrapping the application to set additional env

I set this as draft because I figured once the structure is established documentation should also be done before the PR.

[ConcurAc]

I am able to build the NixOS and home-manager modules successfully. I have also tested that all packages build for the x86_64-linux target.

[JMBeresford]

This is awesome! Thanks so much, @ConcurAc !

My big remaining questions are about how to automate updating this file on release. Retrom releases are fully-automated and I would like to be certain these new files are appropriately updated as needed when a new version is published. What are your thoughts on this?

Additionally, let's go ahead and mark this PR as ready (non-draft) so the user that opened #437 can try this out and provide feedback as well. Docs can be initialized in /docs/Installation.md as a new section at your leisure -- I'm good with the current folder structure for now.

Thanks again!

[JMBeresford]

I am able to build the NixOS and home-manager modules successfully. I have also tested that all packages build for the x86_64-linux target.

@ConcurAc Do we need to test on ARM? I can spin up a new github action that runs a nix build on both arm64 and x86_64 ubuntu runners -- would this suffice? What about macOS (also a runner target, if needed)?

[ConcurAc]

Do we need to test on ARM?

@JMBeresford I can do cross-compiled builds of all the targets on my computer so you shouldn't need to. I haven't yet because I forgot how easy it was to with nix.

edit: I forgot that tauri doesn't use webkitGTK on macOS so the client won't run on macOS right now without adding the necessary runtime dependencies. I also can't (or haven't figured out how to) cross compile for macOS.

I can spin up a new github action that runs a nix build on both arm64 and x86_64 ubuntu runners

I was thinking for the update script to prefetch the hashes, It might be a bit easier to use a NixOS runner, I don't know the process of installing nix on ubuntu but it is also possible.

[JMBeresford]

I was thinking for the update script to prefetch the hashes, It might be a bit easier to use a NixOS runner, I don't know the process of installing nix on ubuntu but it is also possible.

@ConcurAc Hm, I don't fully understand. Does this action remove the need for static SHA values in the configuration?

[ConcurAc]

Does this action remove the need for static SHA values in the configuration?

@JMBeresford Sorry I resolved the thread I was referring to. I mentioned a custom script which could be used to patch the SHA values by running the download segments of the build process to prefetch the hashes. The SHA values could then be updated/replaced.

I suggested a nix runner because the script relies on using nix for the downloads.

[typedrat]

I do this a bunch at work and we're only using standard Ubuntu-based runners. DeterminateSystems/nix-installer works seamlessly.

[JMBeresford]

Would it be best to have these files templated on release, and provided via a tarball in the release assets rather than statically defined in the git tree? I can help with setting that up, so long as you can provide me with the required templated values.

See #436 for an example of how this was recently done for the new flatpak manifest file.

[JMBeresford]

Why is this statically defined here, but the tag is templated from finalAttrs? What is finalAttrs exactly, how is it parameterized?

[ConcurAc]

Apologies for not properly explaining the distinction between version and finalAttrs.version.

The reason to distinguish the two is overlay support. They work like similar to patches to allow modifications to the underlying derivation and take args final and prev.

Referencing https://wiki.nixos.org/wiki/Overlays this is the data flow visualised

To highlight a practical use of finalAttrs, in the following example...

cargoLock.lockFile = "${finalAttrs.src}/Cargo.lock";

Simply

• finalAttrs.src = new src
• src = original src

So this will allow the derivation to use the new Cargo.lock file instead of the original one.

[JMBeresford]

Ah! Ok interesting, this makes a bit more sense. Thanks for the detailed clarification!

[typedrat]

Would it be best to have these files templated on release, and provided via a tarball in the release assets rather than statically defined in the git tree? I can help with setting that up, so long as you can provide me with the required templated values.

See #436 for an example of how this was recently done for the new flatpak manifest file.

Nix actively fetches things from Git repositories as part of its reproducibility, and while it can download tarballs, that would break automatic updates.

[JMBeresford]

@typedrat I see, I think I may be misunderstanding the purpose of this change then. I was under the assumption that nix users would be expecting a ready-made flake that could be cloned/downloaded from this repo and then used to build+install Retrom on their machine.

I have never used nix/nixOS myself, and need to know the following to ensure I make informed decisions on how to distribute a flake:

1. How does auto-updating come into play when Retrom is distributing a flake w/ statically defined versions/tags/dependency SHAs? Unless some external tool is used to facilitate grabbing the flake from the repo?
2. How does providing this flake as a build artifact from the release pipeline differ from providing it directly in the source if the files are identical?
3. What are the standard method(s) for installing nix flakes
   1. Clone repo -> build -> install ?
      1. In this case, why would nix ever need to fetch resources from GH again?
   2. nix install jmberesford/retrom or similar?
   3. Publishing to some upstream flake repository? I see there is a FlakeHub, is this preferred over using the repo as a source?

Regardless of method, it seems we will need to template these files to automate updating the version/tag and dependency SHA values. I can look into doing this once everything else is aligned on here. Thanks to both of you for your help and guidance on this! @ConcurAc @typedrat

[ConcurAc]

I was under the assumption that nix users would be expecting a ready-made flake that could be cloned/downloaded from this repo

In this case, why would nix ever need to fetch resources from GH again?

@JMBeresford You are correct with this and in retrospect you can 100% remove the predefined SHA hashes for the local source dependency. An example of this is here https://github.com/noctalia-dev/noctalia-shell/blob/main/nix%2Fpackage.nix

Most of my experience for writing package derivations come from referencing nixpkgs as opposed to flakes, which only deals with external git repos so it didn't naturally occur to me that this would be an option.

That being said as the pnpm dependencies are referenced outside the repo they still need to be prefetched.
The Cargo.lock can be referenced to avoid specifying the hash, and while there are SHA-512 values defined in pnpm_lock.yaml I haven't found any way to reference it in the documentation.

How does auto-updating come into play when Retrom is distributing a flake w/ statically defined versions/tags/dependency SHAs? Unless some external tool is used to facilitate grabbing the flake from the repo?

The binary will be completely managed by the nix store, which is read only. You wouldn't be able to modify it in with auto update but this isn't necessarily a limitation, its just a property of nix/NixOS as a immutable package manager/distribution. If you were to use external tools such as the nix cli for updating it would only make sense if you assume a non declarative install, as otherwise it would be updated by the sys admin with nix flake update followed by nixos-rebuild switch or home-manager switch

How does providing this flake as a build artifact from the release pipeline differ from providing it directly in the source if the files are identical?

From my understanding Functionally they are the same but typically flakes are defined by referencing the remote URI, which means unless you download and reference the source archive as a path or something all references would point to the remote. To work around this you would use FlakeHub to host the build artifacts instead.

nix install jmberesford/retrom or similar?

Currently the a user has the option of

• declarative install (add the flake to their config, and the package to their environment or enable the nixos/home-manager module)
• non declarative install of any packages from flake with nix profile add github:JMBeresford/retrom#retrom
  • branch can also be specified with identifier format github:JMBeresford/retrom/<branch>
  • alternatively it can be run without permanent install using nix run instead of nix profile add

Publishing to some upstream flake repository? I see there is a FlakeHub, is this preferred over using the repo as a source?

I would consider an upstream external repository like FlakeHub worth it if search availability is important. Alternatively NixOS has a 1st party search util for flakes, NixOS Search however the process of being included in the search means creating a PR update to https://github.com/NixOS/nixos-search/blob/main/flakes/manual.toml. In addition this does not seem very mature compared to FlakeHub.

[ConcurAc]

@JMBeresford What is also worth integrating into the flake release is a cache like cachix which would allow users to install without them building from scratch, its just built once on release and added to the cache. Although I don't have experience setting this up.