Pyodide updates: add 314.0 alpha, switch to pyemscripten platform tag (PEP 783), and remove hardcoded Emscripten location + update dependencies

[agriyakhetarpal]

as stated in the PR title. Through this PR, we now use pyodide config get emscripten_dir to get the Emscripten installation location, and since PEP 783 is close to being accepted, we have updated the wheel platform tag to have pyemscripten_year_number instead of pyodide. Lastly, new dependency updates, via the constraints updater that I split out from #2810.

[henryiii]

It has been accepted. :D

[agriyakhetarpal]

One question: should we rename Pyodide to Pyemscripten in most other places as well?

[hoodmane]

We should keep calling it Pyodide, nobody knows what pyemscripten is.

[henryiii]

If we wanted to match everything else, then the platform would be emscripten (like linux), the identifier would be pyemscripten (like manylinux) or maybe even pyodide. I don't think it's important enough to change now, though now would be the time to do it when this is about to make a fairly big change to the wheel name.

[ryanking13]

One question: should we rename Pyodide to Pyemscripten in most other places as well?

+1 for keeping it as Pyodide.

[ryanking13]

Why do we need force?

[agriyakhetarpal]

We need it to ensure that we always have a fresh copy of the Emscripten toolchain installed and patched. One might argue that cibuildwheel runs in CI on fresh runners and therefore this wouldn't be necessary, but I would suggest there are two reasons:

1. Sequential builds: if, for whatever reason, someone builds multiple Pyodide wheels in CI using separate cibuildwheel CLI invocations or GitHub Actions within the same job in a workflow, then we will always have a self-contained build that does not rely on a stale Emscripten toolchain.
2. Security: although this may be somewhat unlikely, cibuildwheel does execute user code, which could potentially tamper with the Emscripten toolchain with the right payload when building the wheel. I am aware that this risk could apply to any file on a system too, but even if we confine ourselves to this scenario, it would be beneficial for the user to have confidence that their Emscripten installation is always the one we intended them to use. This also supports the use case for cibuildwheel outside of CI and ties into the staleness aspect in point 1. I tend to use cibuildwheel to build Pyodide wheels locally quite often.

[ryanking13]

I wonder if we should put this path before or after the original PATH.

In pyodide-build, if a system emscripten is installed, it dominates. This is to ensure we don't shadow the user-installed emscripten. However, regarding the cibuildwheel would normally run in CI (I guess)? It is rare that people would want to install emscripten themselves, so this might make sense.

[agriyakhetarpal]

Good question. We have been doing env["PATH"] = os.pathsep.join([str(emcc_path.parent), env["PATH"]]) even before #2800, i.e., in all our previous releases with Pyodide support. So, yes, I would expect a user of pyodide-build standalone to want a higher level of control over the build process and be able to their own Emscripten installation, whereas a user of cibuildwheel would trust what it does and not mess with the moving parts required to build Pyodide wheels. This means cibuildwheel should dominate with its Emscripten installation indeed, like it does right now.

That said, we can always change this behaviour and append to PATH if someone requests it at a later point in time, of course.

[agriyakhetarpal]

Alright, I think Pyodide stays as the name, then. We can always create an alias and add both Pyodide and pyemscripten at a time when the latter gains popularity.

[hoodmane]

The existing tags should stay the same right? There is no version of Pyodide that accepts a wheel tagged pyemscripten_2024_0 so if we're going to make them we might as well just delete support for versions older than 314. But it'd be better not to change the existing behavior.

[agriyakhetarpal]

I believe this is a regression, then. @ryanking13, were we too early in changing to the pyemscripten platform tag in pyodide-build? Ideally the previous xbuildenvs (0.27.7, 0.29.3, etc.) should have set USE_LEGACY_PLATFORM, but then that's not possible to fix retroactively as we didn't know that at the time. So maybe we should yank pyodide-build==0.34.1, set USE_LEGACY_PLATFORM to 1 and create a new release with it (0.35.0 because semver), and then ensure that the upcoming 314.0 xbuildenv(s) will set USE_LEGACY_PLATFORM to 0 in Makefile.envs. I would like to support both 0.29.3 and 314.0 releases at the same time, to allow users extra time to switch ABIs.

[ryanking13]

Well, I am not sure to be honest. If we keep the pyodide tag, then the package will not be uploadable to PyPI, if we change it to pyemscripten, old versions of micropip will deny the package. So both choices have downsides.

I am personally +1 for using the pyemscripten name for even older versions for consistency. Older Pyodide versions can still either 1) install newer micropip from PyPI, or 2) install it directly through pyodide.loadPackage. I think it is less confusing than outputting different platform names for differenet Python versions.

[agriyakhetarpal]

Hmm, given that users do have an option to update micropip, and now that the first alpha of 314.0 is out (thanks to you!), I will add it in this PR and drop Pyodide 0.27.7.

Will that resolve your concerns, @hoodmane? Or, do you think that cibuildwheel should work around it now, and rename 2024_0 and 2025_0 ABI wheels that contain pyemscripten in the platform tag string back to pyodide for them?. We do have some time until this PR is merged and released, and it brings pyodide-build 0.34.1, so it's better that we do things right in the first go.

[henryiii]

If a patch release of pyodide 3.13 with updated micropip was possible, then I'd go:

• drop 3.12
• switch 3.13 to pyemscripten_2025
• keep 3.14 as "pre-release" until pyemscripten_2026 is declared stable.

If a patch release is hard to do... Well, that's uglier. Are these "normal" enough that wheel tag can re-tag?

[agriyakhetarpal]

I've added the new alpha release (prerelease) and corresponding code, and dropped Pyodide 0.27.7. Thanks!

[agriyakhetarpal]

Git/GitHub shows this file as renamed and can't identify this – more accurately, the 3.12 constraints were deleted and followed by the creation of a new file for the 3.14 constraints.

[henryiii]

You can split them into two commits, and then we can rebase and merge instead of squash.

[agriyakhetarpal]

Having noticed several moving parts in the Pyodide test suite that I didn't fully address while running it locally, I took the liberty of writing a short maintainer-specific guide for Pyodide maintainers in 6092434. The guide outlines what to do on upcoming occasions when we have Pyodide releases in relation to the cibuildwheel code. It is not intended to be overly detailed, and I intentionally omitted test-related information because tests drift over time. cc: @ryanking13 and @hoodmane, if you'd like to read it.

This guide could be expanded to include documentation for additional nitty-gritties or decisions about our code that currently have no designated place – such as those buried in TODO comments (as is the case with the PyodidePrerelease identifier group comment that I modified in this PR).

@joerick and @henryiii, I'd love to know what you think! Do you approve of this initiative? :) If so, I’d be happy to ping our maintainers from other platforms to see if they would be willing to contribute similar documentation for the code they interact with, too!

[hoodmane]

The node_version situation is a bit wishy-washy. We should improve our docs on node versions. No action needed here.

[agriyakhetarpal]

Agreed

[hoodmane]

Can we simplify this to say just whatever the most likely thing is? We're talking to ourselves here, if we later disagree with the advice we can deviate from it or change it.

[agriyakhetarpal]

I think we need to decide on this before I change this sentence, because we want to be aligned on a policy for how many Pyodide versions/ABIs we should support at a time in cibuildwheel. So far, it's always been two at a time (or even one, some time back). Please read my other comment: #2812 (comment)

[hoodmane]

Is there a reason to remove 3.12 here?

[agriyakhetarpal]

Yes, since I don't think it's useful to have three ABIs enabled at once. We don't support anything but the latest version. Pyodide 0.29.3 and 314.0.0a1 have two different ABIs, which should be enough? Since we don't have a stable 314.0.0 release and IMO cibuildwheel shouldn't be as disruptive to users' workflows, I think it's okay to keep 0.29.3 – but three different ABIs becomes a bit excessive. People can still build for arbitrary (read: older) versions, such as Pyodide 0.27.7, using CIBW_PYODIDE_VERSION. Or would you prefer I keep three ABIs?

[henryiii]

What's the usage of the old versions? IMO, people can upgrade, and probably should, but it's also easy to forget. I'm sure I have webpages that are using old versions.

I think the pyemscripten tag is much more important than the one you can't upload to PyPI, so I think non-pyemscripten are really low priority, I don't mind dropping any of them.

[hoodmane]

Per jsdelivr, the most-used version is Python 3.12, followed by 3.13 and 3.10 which are almost tied.

Python 3.13: 2,033,897
Python 3.12: 6,602,877
Python 3.11:   697,276
Python 3.10: 1,929,739

[ryanking13]

I think it is slightly uncomfortable that we need to turn this on and off every time we make an alpha release. Would there be a way to remove this?

As we standardized pyemscripten platform now, I think there is no reason to differentiate Pyodide prerelease and stable. Both versions should satisfy the pyemscripten platform constraints and users doesn't have to know whether they are using alpha xbuildenv or not.

[hoodmane]

We still need a notion of alpha versus stable I'd say. We can change the platform after alpha releases so people should wait for a stable release to upload wheels.

[ryanking13]

We can change the platform after alpha releases

Hmm, my idea was to freeze the ABI as of alpha release, and that is what I was going to say in the blog post (pyodide/pyodide-blog#66).

I think we all agreed freezing the ABI earlier before making a stable release (pyodide/pyodide#5580), do you think we need another, say "beta" release for that then?

[agriyakhetarpal]

Yes. I agree with @hoodmane here. I think we should still make the difference between alpha and stable. This is because even for PyPI and wheels for conventional (desktop) platforms, it is advised to only test building wheels with an upcoming ABI – not upload them. Uploading to PyPI does not happen until there is a stable CPython release.

Doing this also protects us in the event where may need to change the ABI after our proclamation of stabilising it, due to any bugs or regressions that get caught. We only have to do this process a few times a year, at most, so I would personally be fine with it flipping the switches on and off in this manner.

[ryanking13]

I see. I thought PyPI allows uploading Python wheels when CPython alpha or beta release is out. If it doesn't, then it would make sense for us to do the same approach as well.

[agriyakhetarpal]

Yes, I think CPython does so starting with b1, but it is still discouraged, IIRC. @henryiii, I assume you'd be the right person to know about this topic!

[agriyakhetarpal]

Even then, freezing the ABI and making the announcement is still fine I think – the more eyes we receive on this, the better our adoption will be for PEP 783, as downstream package authors will be better prepared to make the jump.

[henryiii]

CPython is not guaranteed to be ABI stable till the last beta release (historically, everyone basically uses RC1 as the first "stable" release, and that's what cibuildwheel does, but officially it's last beta, beta 4, at least in the past).

So you should not upload a wheel build with an early beta, but I don't think anyone stops it. In cibuildwheel you acknowledge it being a bad idea to upload with the enable opt-in.

IMO this should be removed whenever the ABI is stable. Doesn't have to match some special "final" release, but it has to be ensured to be stable.

[ryanking13]

As I mentioned in the previous comment, I think we should remove differentiating Pyodide alpha from stable at least in cibuildwheel-side of view.

[henryiii]

You can split them into two commits, and then we can rebase and merge instead of squash.

[henryiii]

CPython is not guaranteed to be ABI stable till the last beta release (historically, everyone basically uses RC1 as the first "stable" release, and that's what cibuildwheel does, but officially it's last beta, beta 4, at least in the past).

So you should not upload a wheel build with an early beta, but I don't think anyone stops it. In cibuildwheel you acknowledge it being a bad idea to upload with the enable opt-in.

IMO this should be removed whenever the ABI is stable. Doesn't have to match some special "final" release, but it has to be ensured to be stable.

[henryiii]

What's the usage of the old versions? IMO, people can upgrade, and probably should, but it's also easy to forget. I'm sure I have webpages that are using old versions.

I think the pyemscripten tag is much more important than the one you can't upload to PyPI, so I think non-pyemscripten are really low priority, I don't mind dropping any of them.