[strawman] Add UvPipCompile

[bollwyvl]

References

• RFC 3: using a dependency solver #10

Changes

• add pyodide-lock[uv]

  • the dependency on uv 0.9.27 is fairly bleeding-edge, but this version fixes a fairly serious inconsistency around --python-version

• add UvPipCompile and helper classes
• add tests

  • well, technically just one, but with a fair number of parameterized inputs encountered in JupyterLite situations, but representative of a number of classes of issues

• add minimal example to README

Future work

potentially out-of-scope for this PR

• add a CLI
• add docs
• fixing PEP-420 import name discovery

[ryanking13]

Thanks for the PoC @bollwyvl. Overall, I am +1 for the direction to provide an easy way to create a lockflie with custom dependencies, and it is great that we can leverage uv pip compile for that.

In the long term, I would prefer to create a lockfile from scratch instead of passing pyodide-lock.json as an input. This should be available when PEP 783 is accepted, or it is even possible now by using alternative package indices.

We are also considering adopting pylock.toml format and replace pyodide-lock.json with it for better compatibility (pyodide/pyodide#5617), thogh it would require some more work.

[bollwyvl]

In terms (ha!) of schedule: sure, it certainly has taken a long time for all of the upstream features and specs to line up for:

• an platform-correct solve
• using only pip-installable tools
• including local wheels
• in under a second
• aware of the constraints of a host pyodide site

But great as the enemy of the good and all: I'm very motivated to do whatever it takes to get this landed here. pyodide-lock.json is what we have to work with right now, and I don't know of a better place to land this feature set which is sorely lacking for folks that want to offer their users a better, faster Pyodide experience.

PEP 783

Yep, that and many other great PEPs (e.g. PEP-794) are coming soon to an ecosystem near you... in the long term. Perhaps pessimistically, I'm assuming "use anything from PyPI" will need to be tweaked for a while yet to work properly in the harsh climate of the browser.

Further, I don't know if PyPI really wants to be signing up for it to be easier for potentially hundreds of megabytes of downloads per-page-view. Developers are already pretty cavalier about caching in e.g. CI; browser users might not even know they are using Python.

As a side effect of having to backfill PEP-764 by inspection, the proposed API encourages self-hosted wheels. In previous attempts at these features, I actually wired up a local caching proxy to intercept all wheel and API calls to avoid the second hit to download the file, as uv doesn't provide a nice way to get/reconstruct the actual wheels it downloads. But more subprocess, more problems, and 2 * whl < uncached_page_views_forever * whl.

require some more work.

Right: TOML not being part of the web "standard library", I'm confident a JS-based parser can be found, workarounds for bespoke, not-wheel files engineered, etc... in the long term.

lockfile from scratch

Assuming a hard switch that breaks downstream workflows is being considered, the proposed API could be adapted to make that easier. Pep751Toml could become a more full read/write, validating model, maybe in a separate source file unless a good, lightweight (ideally no pydantic), preferrably PyPA-supported one appears.

Indeed, there's probably value in this repository providing a path (ha!) from an existing pyodide-lock.json to a pylock.toml if the output file ends in .toml, and anything in-between.

[hoodmane]

Seems like pypi is interested in serving our packages. The main blocker on pep 783 is the Donald Stufft problem -- he's supposed to decide on whether to approve the pep or not but he does not read it. I have a meeting with the steering council in a few weeks and hopefully they'll come up with an alternative for me.

[bollwyvl]

pypi is interested in serving our packages

Great! But this doesn't change the fact that there are currently zero wheels on PyPI or any Pyodide distribution that would be marked properly. The use case of:

add a locally-built, unpublished wheel to the existing auto import mechanism provided by pyodide-lock.json with (potentially modified) PyPI dependencies

... is still an enormous pain for downstream users, and therefore downstream maintainers, such as myself.

Perhaps #10, or a brand-new issue, would be a better place to discuss future PEPs and future breaking changes. In the meantime, what would take for the capability proposed here to land in pyodide-lock 0.1.2? Again, happy to make whatever changes are needed, or pursue another approach.

[ryanking13]

Sure, even after the PEP 783 is landed, it will take a lot of time for the package maintainers to adopt it and publish wheels. So having a feature to generate a lockfile from local wheels, wheels hosted from other places would still be useful.

Let me take a look at this PR soon and see what kind of changes we want before landing, and sorry for the delay. I've been busy these days and didn't have time to think about the lockfile.

[ryanking13]

How about preprocessing the base_url_for_missing at the constructor, so that we don't need to remove the suffix here.

[bollwyvl]

I guess my goal is to change the input lock as little as possible, and isolate all rewriting in the post-processing step.

[bollwyvl]

Had a quick look at the uv 0.10 upstream, and don't think anything would have changed much, but will kick CI to check...

[bollwyvl]

I see pyodide-build 0.33.0 dropped... does that impact any changes here (e.g. new ABI schemes, etc?)

[ryanking13]

I see pyodide-build 0.33.0 dropped... does that impact any changes here (e.g. new ABI schemes, etc?)

No, it should not affect anything for pyodide-lock.

[ryanking13]

Thanks! Let me merge this to unblock your needs. I am planning some re implementation of pyodide-lock after PEP 783 is accepted, so that people can easily generate lockfile from packages in PyPI from scratch. Meanwhile people can use this as a workaround.

[bollwyvl]

thanks so much! ❤️

between this and github actions supporting non-archived artifacts, many interesting things might become much easier.