Expose emscripten_dir variable via config

[agriyakhetarpal]

Make the Emscripten directory available via config so that we don't have to compute the xbuildenv_path mess and go back and forth to find out where the Emscripten compiler binary and other tool binaries are located.

• Changelog

[ryanking13]

Could you elaborate on this?

[agriyakhetarpal]

This is because the Emscripten version was being mis-found during concurrent builds. For example, in pypa/cibuildwheel#2800, we were previously relying on the symlinks. When running the test suite in parallel, it would find Emscripten 3.1.58 when we would need 4.0.9 and cause weird failures in the tests. I kept the comment a bit vague, though, indeed. Happy to add more details if you need me to?

[ryanking13]

I see. Makes sense, thanks!

[ryanking13]

Thanks!

[agriyakhetarpal]

Thanks for the review! Leaving a note here about the comment, if we want to take a look at this later:

• We compute the Emscripten directory from pyodide_root. We know that the on-disk layout is:

  {xbuildenv_root}/
      xbuildenv  ->  {version}/          # symlink, which is updated on each install
      {version}/
          xbuildenv/
              pyodide-root/              # which is `self.pyodide_root`
          emsdk/
              upstream/
                  emscripten/            # what we want
  

• We compute this from self.pyodide_root (which is already a fully resolved, version-specific path) rather than via Makefile.envs or DEFAULT_CONFIG for two reasons:

  1. Makefile.envs does not know about the emsdk location.
  2. The xbuildenv symlink is rewritten on each pyodide xbuildenv install call. If multiple builds for different Pyodide versions run concurrently (e.g. in cibuildwheel, when running the test suite in parallel), re-resolving through the symlink at a later point could yield the wrong versioned directory. Using self.pyodide_root, which was resolved at construction time, gives us a stable, race-free path.

[ryanking13]

The xbuildenv symlink is rewritten on each pyodide xbuildenv install call. If multiple builds for different Pyodide versions run concurrently (e.g. in cibuildwheel, when running the test suite in parallel), re-resolving through the symlink at a later point could yield the wrong versioned directory. Using self.pyodide_root, which was resolved at construction time, gives us a stable, race-free path.

Thanks for the investigation. Actually, I guess this is something that we need to think more carefully about, not just for emscripten but for other config variables as well. I personally never considered concurrent builds when designing xbuildenv installation. If cibuildwheel is using parallel builds a lot, we should be make pyodide-build more concurrent-robust.