Support buildng multiple outputs from a recipe

[agriyakhetarpal]

This feature request is based on a requirement that came up in #109 – it would be great if we could also support the conda-forge feature for multiple outputs for builds, which would use the same build environment for building multiple variations of a package.

The proposed implementation could be as follows:

• add an outputs: field to the recipe that allows subfields such as script:, about: post:, more requirements: (and sub-requirements), etc.
• build in the order they are specified, building the first listed output and proceed to the next one if it passes
  • we can stop if it fails; I don't think we need a --keep-going option or building in parallel at the moment
• the graph builder can keep the same logic, more or less – it's just that one recipe will generate $n$ packages instead of just one

The rationale is that many recipes could benefit from this – for example:

• packages such as NumPy/SciPy/etc. and various Meson-based packages can then use their "install tags" integration with Meson to build WASM wheels for scipy and scipy-tests, which can be installed together in the same environment. this can be used to control build sizes better than stripping away wheels.
• it allows packages to ship multiple variants of their builds: we could ship NumPy + OpenBLAS and NumPy with Netlib BLAS from the same recipe file, just with different build scripts listed together.
• for libraries: SuiteSparse, for example, can be instructed to build different components separately for packages that need only a specific one.

Reference:

• https://github.com/conda-forge/scipy-feedstock/blob/1e665407f6222093df606d9e27553072b7222fc9/recipe/meta.yaml
• https://github.com/conda-forge/suitesparse-feedstock/blob/d0a371ec49dfb0a9dbbcea5815c0f4c71e87d462/recipe/recipe.yaml

[ryanking13]

Thanks for the proposal, @agriyakhetarpal. It sounds like an interesting idea to me. However, I think the format of the outputs can be more formalized before we proceed to implementation, for instance, how it will work with the lockfile, etc.

[agriyakhetarpal]

Coming back to this from pyodide/pyodide#5563, I think the format will remain largely the same as the current recipes, as we would not like or need the added complexity that comes with conda-forge recipes.

For the lockfile: two outputs would build two different packages with two different names and checksums, which would be placed in the dist/ folder of the recipe dir, next to meta.yaml, so I think we should be fine.

Here is what a sample recipe for NumPy and numpy-tests could look like:

package:
  name: numpy
  version: 2.2.3
  tag:
    - min-scipy-stack
  top-level:
    - numpy
source:
  url: https://files.pythonhosted.org/packages/source/n/numpy/numpy-2.2.3.tar.gz
  sha256: dbdc15f0c81611925f382dfa97b3bd0bc2c1ce19d4fe50482cb0ddc12ba30020

outputs:
  - name: numpy
    build:
      unvendor-tests: false # we do this on our own using Meson's install tags
      # numpy uses vendored meson, so we need to pass the cross file manually
      backend-flags: |
        setup-args=-Dallow-noblas=true
        setup-args=--cross-file=${MESON_CROSS_FILE}
        install-args=--tags=runtime,python-runtime,devel
        build-dir=shared_build
      # numpy creates numpy/distutils/__pycache__ directory during the build.
      # It breaks our test because there is a .pyc in the directory.
      post: |
        rm -rf numpy/distutils/__pycache__
      cflags: |
        -Wno-return-type
      cross-build-env: true
      cross-build-files:
        - numpy/_core/include/numpy/numpyconfig.h
        - numpy/_core/include/numpy/_numpyconfig.h
        - numpy/_core/lib/libnpymath.a
        - numpy/random/lib/libnpyrandom.a
  
  - name: numpy-tests
    requirements:
      host:
        - numpy
      run:
        - numpy
    build:
      unvendor-tests: false # we do this on our own using Meson's install tags
      # rename the package before we build
      script: sed -i 's/name = "numpy"/name = "numpy-tests"/' pyproject.toml
      # numpy uses vendored meson, so we need to pass the cross file manually
      # reuse the build directory from the previous output
      backend-flags: |
        setup-args=-Dallow-noblas=true
        setup-args=--cross-file=${MESON_CROSS_FILE}
        install-args=--tags=tests
        build-dir=shared_build
      # numpy creates numpy/distutils/__pycache__ directory during the build.
      # It breaks our test because there is a .pyc in the directory.
      post: |
        rm -rf numpy/distutils/__pycache__
      cflags: |
        -Wno-return-type

about:
  home: https://www.numpy.org
  PyPI: https://pypi.org/project/numpy
  summary: NumPy is the fundamental package for array computing with Python.
  license: BSD-3-Clause
extra:
  recipe-maintainers:
    - agriyakhetarpal

Thus, the difference, and the aspects of this format are as follows:

• NumPy is included in the xbuildenv, but numpy-tests is not
• the about: and extra: sections are shared for both outputs
• numpy-tests has a build-time dependency and runtime dependency on NumPy. The build-time dependency exists solely to not allow NumPy and numpy-tests to be built together, as we don't want that to happen – we wish to reuse the NumPy build files for numpy-tests, which we marked as persistent using the build-dir config setting.
• all packages listed under outputs: are built from the same source directory, and patches, if any, are applied only once
• the outputs are built based on how the graph builder resolves them, and not in order (which also ties into point 3 above).

Of course, this is quite tailored to support this NumPy and numpy-tests scenario, so I think we could keep it Pyodide-maintainer-only and expand it to other packages before we mark it as generally available to all recipe maintainers.

I'm not sure if there are any other implications of this format that we will need to consider, as only an implementation can tell –unless you can foresee some. I have an alternative idea for a persistent/shared build directory that I can put up a PR for in a moment.

[ryanking13]

Thanks for the detailed example. It generally looks good to me. One thing I am not sure about is whether we should promote people to use outputs key instead of using build and requirements key in top-level. The numpy case is very special, and I don't think people would normally need to output multiple wheels, so I think it is good to keep the original syntax for other packages.

numpy-tests has a build-time dependency and runtime dependency on NumPy. The build-time dependency exists solely to not allow NumPy and numpy-tests to be built together, as we don't want that to happen – we wish to reuse the NumPy build files for numpy-tests, which we marked as persistent using the build-dir config setting.

I think it is better to specify the build order explicitly (or simply build in order in the list), instead of relying on the requirements. The numpy key in requirements would be tricky to handle as we both have numpy in the top-level, and under the outputs section.

[agriyakhetarpal]

One thing I am not sure about is whether we should promote people to use outputs key instead of using build and requirements key in top-level. The numpy case is very special, and I don't think people would normally need to output multiple wheels, so I think it is good to keep the original syntax for other packages.

Yes, I agree. Such syntax should be opt-in, and the existing syntax should be recommended unless there is a case for using this syntax. Currently, I only have Meson-based packages such as NumPy, SciPy, scikit-image, Matplotlib, and some libraries like SuiteSparse (it builds various subpackages like COLAMD, BTF, KLU from the base SuiteSparse-config) that would be potential candidates.

numpy-tests has a build-time dependency and runtime dependency on NumPy. The build-time dependency exists solely to not allow NumPy and numpy-tests to be built together, as we don't want that to happen – we wish to reuse the NumPy build files for numpy-tests, which we marked as persistent using the build-dir config setting.

I think it is better to specify the build order explicitly (or simply build in order in the list), instead of relying on the requirements. The numpy key in requirements would be tricky to handle as we both have numpy in the top-level, and under the outputs section.

Yes, I thought so, too, but didn't mention it as I wasn't sure how it would play out with our graph builder. Thanks! Building in order sounds good to me – that is what conda-forge is doing. However, we'll still need numpy-tests to depend on NumPy, so we will need to figure out how that would work with our setup. For example, the conda install -c conda-forge scipy-tests command also prompts the user to install SciPy, as scipy-tests depends on scipy.

[ryanking13]

Yes, I thought so, too, but didn't mention it as I wasn't sure how it would play out with our graph builder. Thanks! Building in order sounds good to me – that is what conda-forge is doing. However, we'll still need numpy-tests to depend on NumPy, so we will need to figure out how that would work with our setup. For example, the conda install -c conda-forge scipy-tests command also prompts the user to install SciPy, as scipy-tests depends on scipy.

Right, I think that's something we should be careful about when implementing the feature.

• If another package specifies numpy as a dependency, should we build both numpy and numpy-tests or only numpy? How can we determine it?
• Should we provide a way to build only numpy not numpy-tests? Or build both of them everytime?
• Should pyodide build-recipes numpy-tests --install only include numpy-tests in the lockfile, or both numpy and numpy-tests?

I mean, we don't need to decide everything before proceeding to the implementation, but I think it would be helpful to think about the subtle cases we might have to deal with later.

[agriyakhetarpal]

Thanks for asking these questions!

• If another package specifies numpy as a dependency, should we build both numpy and numpy-tests or only numpy? How can we determine it?

If another package specifies NumPy as a host: dependency, we should build only numpy and not numpy-tests. If a package depends on NumPy at runtime, we should build both numpy and numpy-tests, as they might be using code from numpy.testing for their in-tree tests, which would be included only in the latter.

However, I see two problems coming immediately from this, and I don't have a solution yet:

• we are progressing on unvendoring the recipes but haven't fully done so yet – which means that in most cases, numpy and numpy-tests would already have been built.
• The numpy/numpy-testscase is highly special, as hundreds of packages depend on NumPy. We are able to make a distinction here because we know the difference between the contents ofnumpyandnumpy-tests`, and also why they should exist (smaller binary size). We can't say the same about packages which will depend on package $B$ and package $C$, which are outputs of a recipe for a package $A$, as it depends on what the packages are for.

Should we provide a way to build only numpy not numpy-tests? Or build both of them everytime?

I think this is also linked to the previous question in some way. In this particular case, we would build both of them every time, as the caching from the persistence of the build dir would ensure that the second output (numpy-tests) would build soon. I would be inclined towards building all packages every time. This situation is better handled when CI time is not a concern, i.e., when recipes are unvendored.

Should pyodide build-recipes numpy-tests --install only include numpy-tests in the lockfile, or both numpy and numpy-tests?

Based on my answer to the second question, as numpy-tests would depend on numpy, this command would build both packages and include both of them in the lockfile.

I mean, we don't need to decide everything before proceeding to the implementation, but I think it would be helpful to think about the subtle cases we might have to deal with later.

I agree, though I'm happy to wait until we finalise it before working on something concrete!

[agriyakhetarpal]

I have an alternative idea for a persistent/shared build directory that I can put up a PR for in a moment.

In the meantime, I have put this up at #195, and I think it works. While it's not complete, I'm happy to hear your feedback.