Switch to pixi

[fjclark]

Description

Switches to pixi for package management. This fixes issues such as the one described here avoiding duplication of shared dependencies in conda environment yaml files and also makes the Makefile redundant as we can specific commands for pixi in pyproject.toml. I've also updated the CI and documentation. All tests pass for pydantic > and < 1 environments once merged with #129 (but will fail as-is).

Happy to merge this and #129 into a single PR with passing CI if that's preferable. Thanks!

Status

• Ready to go

[fjclark]

Happy to merge this and #129 into a single PR with passing CI if that's preferable.

I've gone ahead and done this, just to check the CI runs.

[Yoshanuikabundi]

Looks good! I have a few nits on how the environments are set up - would be good to improve the experience for developers who don't have a Linux/CUDA machine, which is I think most of the people who are likely to work on this. Unfortunately Pixi makes it a bit difficult to work with PyTorch from custom PyPi indexes, so we're limited in how accessible that is for now.

Comments marked "(non-blocking)" are improvements that I think will help most other devs working on this project, but if you judge that they will impair your work more than they will help everyone else that's your call. Comments marked "(blocking)" are the bare minimum for things to work on certain other machines, so I think they're important.

Hope you're enjoying Pixi!

[Yoshanuikabundi]

(non-blocking) Using solve groups makes sure that you're using the same versions of shared packages across different environments. This makes debugging upstream issues easier to collaborate on. The default is for each environment to be solved independently:

Suggested change

	default = { features = ["cu12", "mm", "fe", "examples", "dev", "docs", "pydantic-2"] }
	cpu = { features = ["cpu", "mm", "fe", "examples", "dev", "docs", "pydantic-2"] }
	# The fe feature is excluded when pydantic < 2 as femto (used by absolv) needs pydantic >= 2
	pydantic-1-cu12 = { features = ["cu12", "mm", "examples", "dev", "docs", "pydantic-1"] }
	pydantic-1-cpu = { features = ["cpu", "mm", "examples", "dev", "docs", "pydantic-1"] }
	default = { features = ["cu12", "mm", "fe", "examples", "dev", "docs", "pydantic-2"], solve-group = "default" }
	cpu = { features = ["cpu", "mm", "fe", "examples", "dev", "docs", "pydantic-2"], solve-group = "default" }
	# The fe feature is excluded when pydantic < 2 as femto (used by absolv) needs pydantic >= 2
	pydantic-1-cu12 = { features = ["cu12", "mm", "examples", "dev", "docs", "pydantic-1"], solve-group = "pydantic-1" }
	pydantic-1-cpu = { features = ["cpu", "mm", "examples", "dev", "docs", "pydantic-1"], solve-group = "pydantic-1" }

[fjclark]

Thanks, I tried this, but ended up going back to separate solve groups to allow the pytorch version to be different between the cpu and cuda environments.

[Yoshanuikabundi]

Ahh yeah that makes sense, good call!

[fjclark]

Thanks for the very helpful review! And yes, I love pixi.

I think I've addressed your points other than adding osx-64, which seemed a bit of a pain to get working from a quick initial shot (running into issues with antechamber failing, maybe related to conda-forge/ambertools-feedstock#140 though I haven't looked in detail). How important is it to include this? I'm happy to put more time into adding it if it's important, but notice that e.g. Interchange doesn't provide osx-64 environments.

[mattwthompson]

For me it's +1 on osx-arm64 and -1 on osx-64 - I don't think any of the team currently uses Intel-based Macs and, as you've found, upstreams are quietly becoming more and more of a headache with them

We quietly dropped support for this elsewhere and, at least for a first pass, I don't think it's worth worrying about here

[Yoshanuikabundi]

Matt's take makes complete sense to me!

[lilyminium]

Thanks @fjclark, please merge when you're ready -- just added you to write access :-)

[fjclark]

Thanks @lilyminium!