From bf33e73d5fe25b831c872793a7a6831875028aeb Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Fri, 22 Jul 2022 18:01:13 -0500 Subject: [PATCH 1/9] Create cfep-22.md --- cfep-22.md | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 cfep-22.md diff --git a/cfep-22.md b/cfep-22.md new file mode 100644 index 0000000..c5f4a99 --- /dev/null +++ b/cfep-22.md @@ -0,0 +1,81 @@ + + + + + + + + + +
Title ABI pinning via SONAME in build string
Status Draft | Proposed | Accepted | Rejected | Deferred | Implemented
Author(s) Daniel Ching <full.name@gmail.com>
Created Jul 22, 2022
Updated Jul 22, 2022
Discussion link to the PR where the CFEP is being discussed, NA is circulated initially
Implementation link to the PR for the implementation, NA if not availble
+ +## Abstract + +Conda lacks a built-in method of tracking ABI separately from API. This is fine for python, whose ABI/API are effectively the same, but for compiled libraries it is not. +Trying to figure out what compatability guarantees that a project offers between API and ABI is annoying because you have to contact upstream maintainers and +manually monitor for changes. However, projects which care about ABI stability already offer this information in the form of the SONAME which is the version of the +ABI and is commonly added to the name of the shared library. This proposal is a standard procedure for adding the SONAME into the build string so that +the conda solver can prevent ABI breaks automatically. + +## Motivation + +You have to ask upstream about compatability guarantees. +Many projects already track this information with the SONAME. + +https://github.com/conda-forge/libavif-feedstock/pull/1#issuecomment-986310764 +https://github.com/conda-forge/dav1d-feedstock/pull/1/files#r927851556 +https://github.com/conda-forge/libwebp-feedstock/blob/615b5309a76ac96409394aa100ec11bb1c7ea150/recipe/meta.yaml#L14 + + +## Backward Compatability + +This proposal does not break exisiting packages. It is a feature that only helps future package builds. + +## Alternatives + +In theory breaking ABI changes can be introduced without changing the API. Reordering struct elements for example, +but project managers would in practice not also make a new API release for this change. + +## Sample Implementation + +Related: What do you think about putting the ABI major version in the build string? i.e. +david=1.0.0=b6hsdfbiwd where the SO major version is 6. Then we could pin to the API and ABI separately. + +run_export: + - {{ pin_subpackage( name|lower ) }} b{{ so_name_major }}h* + +This would be templated out to + +run_exports: + - david >=1.0.0,<2 b6h* + +or maybe + +run_exports: + - david >=1.0.0,<2 + - david * b6h* + +So then the run export handles both API and ABI. The ABI version is between "b" and "h" because if it was just b6*, then we could run into issues when we reach b60. + +The more C++ projects that I become the maintainer of, the more I run into the problem that conda doesn't have a built in way for tracking ABI separately from API (because conda's roots are in python where you don't worry about ABI). + + +## Other sections + +Other relevant sections of the proposal. Common sections include: + + * Specification -- The technical details of the proposed change. + * Motivation -- Why the proposed change is needed. + * Rationale -- Why particular decisions were made in the proposal. + * Backwards Compatibility -- Will the proposed change break existing + packages or workflows. + * Alternatives -- Any alternatives considered during the design. + * Sample Implementation -- Links to prototype or a sample implementation of + the proposed change. + * FAQ -- Frequently asked questions (and answers to them). + * Resolution -- A short summary of the decision made by the community. + * Reference -- Any references used in the design of the CFEP. + +## Copyright + +All CFEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). From 520c8cbe6cdc236009a0547167d5d53e9f0bc5fc Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Mon, 25 Jul 2022 15:34:01 -0500 Subject: [PATCH 2/9] Add links to related discussions --- cfep-22.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/cfep-22.md b/cfep-22.md index c5f4a99..6d1ec95 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -5,7 +5,11 @@ Author(s) Daniel Ching <full.name@gmail.com> Created Jul 22, 2022 Updated Jul 22, 2022 - Discussion link to the PR where the CFEP is being discussed, NA is circulated initially + Discussion + https://github.com/conda-forge/conda-forge.github.io/issues/610 + https://github.com/conda-forge/conda-forge.github.io/issues/157 + https://github.com/conda-forge/conda-forge.github.io/issues/150 + Implementation link to the PR for the implementation, NA if not availble From 6f42e94b83c934cd38604a9e1fb9616f5a0565b9 Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Tue, 26 Jul 2022 15:51:28 -0500 Subject: [PATCH 3/9] Add more details to the proposal --- cfep-22.md | 155 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 124 insertions(+), 31 deletions(-) diff --git a/cfep-22.md b/cfep-22.md index 6d1ec95..88bca50 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -1,68 +1,161 @@ - - + + - + - +
Title ABI pinning via SONAME in build string
Status Draft | Proposed | Accepted | Rejected | Deferred | Implemented
Author(s) Daniel Ching <full.name@gmail.com>
Status Draft
Author(s) Daniel Ching <carterbox@no.reply.github.com>
Created Jul 22, 2022
Updated Jul 22, 2022
Updated Jul 26, 2022
Discussion https://github.com/conda-forge/conda-forge.github.io/issues/610 https://github.com/conda-forge/conda-forge.github.io/issues/157 https://github.com/conda-forge/conda-forge.github.io/issues/150
Implementation link to the PR for the implementation, NA if not availble
Implementation NA
## Abstract -Conda lacks a built-in method of tracking ABI separately from API. This is fine for python, whose ABI/API are effectively the same, but for compiled libraries it is not. -Trying to figure out what compatability guarantees that a project offers between API and ABI is annoying because you have to contact upstream maintainers and -manually monitor for changes. However, projects which care about ABI stability already offer this information in the form of the SONAME which is the version of the -ABI and is commonly added to the name of the shared library. This proposal is a standard procedure for adding the SONAME into the build string so that -the conda solver can prevent ABI breaks automatically. +Conda lacks a built-in method of tracking ABI separately from API. This is fine +for python, whose ABI/API are effectively the same, but for compiled libraries +it is not. Trying to figure out what ABI compatability guarantees that a +project offers between API releases is annoying because you have to contact +upstream maintainers and manually monitor for changes. However, projects which +care about ABI stability already offer this information in the form of the +SONAME which is the version of the ABI and is commonly added to the name of the +shared library. This proposal is a standard procedure for adding the SONAME +into the build string, so that the exisiting conda solver can choose a +compatible ABI automatically. ## Motivation -You have to ask upstream about compatability guarantees. -Many projects already track this information with the SONAME. +Conda doesn't have a built in way for tracking ABI separately from API which +causes hardship for recipe maintainers in an ecosystem where packages are +required to link dynamically. In order to prevent ABI breaks from a future API +release, maintainers need to know ahead of time at which API release, the ABI +will change. This means either asking upstream package maintainers about +compatability guarantees or exporting a pin to the minor version. -https://github.com/conda-forge/libavif-feedstock/pull/1#issuecomment-986310764 -https://github.com/conda-forge/dav1d-feedstock/pull/1/files#r927851556 -https://github.com/conda-forge/libwebp-feedstock/blob/615b5309a76ac96409394aa100ec11bb1c7ea150/recipe/meta.yaml#L14 +One of the approaches is imperfect because maintainers could change their +policy or not have a policy. The other is inflexible because always pinning to +the minor API version may not be necessary. One approach may lead to broken +packages, and the other causes extra downstream builds and may occasionally +cause unsolvable environments. + +## Specification + +Recipes whose libraries include a separate ABI version should add the major ABI +version to the build string between `v` and `so`. For example, `v2so` for +version `2`. This substring in the build string should be used in the +run_export to constrain the pinning to the same ABI major version in addition +to cosntraints on the API version. +Recipes whose libraries have an API version only should export a pin to the +patch level (`x.x.x`) in order to prevent ABI breaks. + +Recipes whose package version is the same as the ABI version do not need to +modify their build string. + +## Sample Implementation + +```yaml +{% set name = "libavif" %} +{% set build = 0 %} +# NOTE: Humans must also update the library version in the tests section of this recipe +{% set version = "0.10.1" %} +# Look in the libavif top level CMakeLists.txt for the updated ABI version. +# ABI is updated separately from API version. +{% set so_version = "14.0.1" %} +{% set so_major_version = so_version.split('.')[0] %} + +package: + name: {{ name|lower }} + version: {{ version }} + +build: + number: {{ build }} + string: "v{{ so_name_major }}soh{{ PKG_HASH }}_{{ build }}" + run_exports: + - {{ pin_subpackage(name|lower) }} *v{{ so_name_major }}so* + +test: + commands: + - test -f ${PREFIX}/lib/libavif.so.{{ so_major_version }} # [linux] + - test -f ${PREFIX}/lib/libavif.so.{{ so_version }} # [linux] + +``` + +## Rationale + +Adding the ABI version to the build string allows the run_export to handle both +API and ABI using existing conda capabilities. Not needing new conda features +saves engineering effort and is implementable today. Build strings are already +used to track features such as the CUDA version and MPI variants. + +The ABI version is sandwiched between `v` and `so` because these letters are +not part of hexadecimal and to prevent matching collisions with other build +string tags or higher numbers. i.e. `6*` matches with `60`, but `6so*` does +not. + +Using a combination of build string and run_exports to add ABI information +requires no changes from downstream packages except using the latest build. +They will be guarded from ABI breakages as soon as they rebuild with the +updated package. ## Backward Compatability -This proposal does not break exisiting packages. It is a feature that only helps future package builds. +This proposal does not break exisiting packages. It is a feature that only +helps future package builds. + ## Alternatives -In theory breaking ABI changes can be introduced without changing the API. Reordering struct elements for example, -but project managers would in practice not also make a new API release for this change. +### API Pinning Only -## Sample Implementation +In theory, breaking ABI changes can be introduced without changing the API +(reordering struct elements for example). In practice, project managers would +always make a new API release for this change, so ABI breaks can be avoided by +pinning down to the patch version. This is the most conservative approach, but +is the least flexible. Recipe maintainers can pin to minor API versions if the +upstream package makes any such promises about not breaking the ABI. + +### Adding ABI name to package name + +This alternatives renames outputs to `{{ name|lower }}{{ so_major_version }}`. +This is not backward compatible and would require migrating all downstream +packages to the new output names. + +### Prepending ABI to package version + +This would mean versioning packages to `{{so_major_version}}.{{verison}}`. This +approach may not be backward compatible with already published package versions +if the ABI version is lower than the API version and would require migrating +all downstream feedstocks. -Related: What do you think about putting the ABI major version in the build string? i.e. -david=1.0.0=b6hsdfbiwd where the SO major version is 6. Then we could pin to the API and ABI separately. +### Exporting a separate ABI package -run_export: - - {{ pin_subpackage( name|lower ) }} b{{ so_name_major }}h* +The conda-forge pybind11 package does this currently. An empty package called +pybind11_abi tracks the ABI version and is used as a run_constraint and +run_export. This approach is backward compatible, but requires additions to the +recipe (additional outputs, run_constraints, and exports). -This would be templated out to +### Modifying conda to automatically track ABI via SONAME -run_exports: - - david >=1.0.0,<2 b6h* +This approach requires new software features on conda instead of relying on +existing features. -or maybe +# Reference -run_exports: - - david >=1.0.0,<2 - - david * b6h* +https://github.com/conda-forge/conda-forge.github.io/issues/610 -So then the run export handles both API and ABI. The ABI version is between "b" and "h" because if it was just b6*, then we could run into issues when we reach b60. +https://github.com/conda-forge/conda-forge.github.io/issues/157 -The more C++ projects that I become the maintainer of, the more I run into the problem that conda doesn't have a built in way for tracking ABI separately from API (because conda's roots are in python where you don't worry about ABI). +https://github.com/conda-forge/conda-forge.github.io/issues/150 +https://github.com/conda-forge/libavif-feedstock/pull/1#issuecomment-986310764 + +https://github.com/conda-forge/dav1d-feedstock/pull/1/files#r927851556 + +https://github.com/conda-forge/libwebp-feedstock/blob/615b5309a76ac96409394aa100ec11bb1c7ea150/recipe/meta.yaml#L14 ## Other sections From ea8176bd15df128b599efe893b6b0293a8ba2767 Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Tue, 26 Jul 2022 16:02:17 -0500 Subject: [PATCH 4/9] Add note about multiple libraries --- cfep-22.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/cfep-22.md b/cfep-22.md index 88bca50..8c464af 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -55,6 +55,11 @@ patch level (`x.x.x`) in order to prevent ABI breaks. Recipes whose package version is the same as the ABI version do not need to modify their build string. +Recipes with multiple library artifacts whose ABIs are tracked separately +should split these libraries into separate outputs of one recipe. A metapackage +may be used for installation convenience, but the metapackge must also +enumerate the run_exports for each subpackage. + ## Sample Implementation ```yaml From cc507f80c9fd2f7ae3ac62f566f9f1089587aa18 Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Wed, 27 Jul 2022 11:16:53 -0500 Subject: [PATCH 5/9] Fix spelling mistake Co-authored-by: Dominik Kutra --- cfep-22.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cfep-22.md b/cfep-22.md index 8c464af..a25af46 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -131,7 +131,7 @@ packages to the new output names. ### Prepending ABI to package version -This would mean versioning packages to `{{so_major_version}}.{{verison}}`. This +This would mean versioning packages to `{{so_major_version}}.{{version}}`. This approach may not be backward compatible with already published package versions if the ABI version is lower than the API version and would require migrating all downstream feedstocks. From 5c30cb994f2e16b745a0b9883a771d52693cb3fa Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Wed, 27 Jul 2022 15:09:49 -0500 Subject: [PATCH 6/9] Add upside to ABI in package name --- cfep-22.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cfep-22.md b/cfep-22.md index a25af46..5d26807 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -127,7 +127,9 @@ upstream package makes any such promises about not breaking the ABI. This alternatives renames outputs to `{{ name|lower }}{{ so_major_version }}`. This is not backward compatible and would require migrating all downstream -packages to the new output names. +packages to the new output names. However, it does enable installing multiple +ABI versions simultaneously (if you don't mind refactoring to prevent +clobbering). ### Prepending ABI to package version From 547c4d4029c38ab233cc2ac810f8c169cc49cbe1 Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Wed, 27 Jul 2022 16:18:37 -0500 Subject: [PATCH 7/9] NEW: Add more detailed explanation/example of ABI in name alternate --- cfep-22.md | 60 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 50 insertions(+), 10 deletions(-) diff --git a/cfep-22.md b/cfep-22.md index 5d26807..3cbd5b2 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -70,7 +70,7 @@ enumerate the run_exports for each subpackage. # Look in the libavif top level CMakeLists.txt for the updated ABI version. # ABI is updated separately from API version. {% set so_version = "14.0.1" %} -{% set so_major_version = so_version.split('.')[0] %} +{% set so_name_major = so_version.split('.')[0] %} package: name: {{ name|lower }} @@ -84,8 +84,8 @@ build: test: commands: - - test -f ${PREFIX}/lib/libavif.so.{{ so_major_version }} # [linux] - - test -f ${PREFIX}/lib/libavif.so.{{ so_version }} # [linux] + - test -f ${PREFIX}/lib/libavif.so.{{ so_name_major }} # [linux] + - test -f ${PREFIX}/lib/libavif.so.{{ so_version }} # [linux] ``` @@ -123,17 +123,57 @@ pinning down to the patch version. This is the most conservative approach, but is the least flexible. Recipe maintainers can pin to minor API versions if the upstream package makes any such promises about not breaking the ABI. -### Adding ABI name to package name +### New output with ABI in package name -This alternatives renames outputs to `{{ name|lower }}{{ so_major_version }}`. -This is not backward compatible and would require migrating all downstream -packages to the new output names. However, it does enable installing multiple -ABI versions simultaneously (if you don't mind refactoring to prevent -clobbering). +Proposed in [this +issue](https://github.com/conda-forge/conda-forge.github.io/issues/157), this +alternative renames outputs to `{{ name|lower ~ so_name_major }}`. This method +is currently utilized for +[libgfortran](https://github.com/conda-forge/ctng-compilers-feedstock/blob/main/recipe/meta.yaml#L542) +and is common in official linux distributions. It also enables installing +multiple ABI versions of a library simultaneously (remember to refactor to +prevent clobbering). The downside of this alternative is that it requires +multiple recipe outputs which is more difficult to implement). + +```yaml +{% set name = "libavif" %} +{% set build = 0 %} +# NOTE: Humans must also update the library version in the tests section of this recipe +{% set version = "0.10.1" %} +# Look in the libavif top level CMakeLists.txt for the updated ABI version. +# ABI is updated separately from API version. +{% set so_version = "14.0.1" %} +{% set so_name_major = so_version.split('.')[0] %} + +package: + name: {{ name|lower ~ so_name_major }} + version: {{ version }} + +build: + number: {{ build }} + run_exports: + - {{ pin_subpackage(name|lower ~ so_name_major) }} + +test: + commands: + - test -f ${PREFIX}/lib/libavif.so.{{ so_name_major }} # [linux] + - test -f ${PREFIX}/lib/libavif.so.{{ so_version }} # [linux] + +outputs: + - {{ name|lower ~ so_name_major }} + # This metapackage allows recipes to continue using the unversioned name + - {{ name|lower }} + build: + run_exports: + - {{ pin_subpackage(name|lower ~ so_name_major) }} + requirements: + run: + - {{ name|lower ~ so_name_major }} +``` ### Prepending ABI to package version -This would mean versioning packages to `{{so_major_version}}.{{version}}`. This +This would mean versioning packages to `{{so_name_major}}.{{version}}`. This approach may not be backward compatible with already published package versions if the ABI version is lower than the API version and would require migrating all downstream feedstocks. From a239b0d1b66cea25b1ddf0c2ab0d29a1dccff7f9 Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Wed, 27 Jul 2022 16:20:52 -0500 Subject: [PATCH 8/9] STY: Remove stray parenthesis --- cfep-22.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cfep-22.md b/cfep-22.md index 3cbd5b2..b67e8b7 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -133,7 +133,7 @@ is currently utilized for and is common in official linux distributions. It also enables installing multiple ABI versions of a library simultaneously (remember to refactor to prevent clobbering). The downside of this alternative is that it requires -multiple recipe outputs which is more difficult to implement). +multiple recipe outputs which is more difficult to implement. ```yaml {% set name = "libavif" %} From 7a5b36fffd522bea8361dc5f55a33ef97fd7d1da Mon Sep 17 00:00:00 2001 From: Daniel Ching Date: Mon, 8 Aug 2022 14:02:28 -0500 Subject: [PATCH 9/9] Add more detailed example named-package recipe --- cfep-22.md | 57 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/cfep-22.md b/cfep-22.md index b67e8b7..a7e6688 100644 --- a/cfep-22.md +++ b/cfep-22.md @@ -127,13 +127,14 @@ upstream package makes any such promises about not breaking the ABI. Proposed in [this issue](https://github.com/conda-forge/conda-forge.github.io/issues/157), this -alternative renames outputs to `{{ name|lower ~ so_name_major }}`. This method -is currently utilized for +alternative produces a new output for the shared library with the name `{{ +name|lower ~ so_name_major }}`. This method is currently utilized for [libgfortran](https://github.com/conda-forge/ctng-compilers-feedstock/blob/main/recipe/meta.yaml#L542) -and is common in official linux distributions. It also enables installing -multiple ABI versions of a library simultaneously (remember to refactor to -prevent clobbering). The downside of this alternative is that it requires -multiple recipe outputs which is more difficult to implement. +and is common in official linux distributions. It enables installing multiple +ABI versions of a library simultaneously, but this requires separating the +headers/docs from the shared libraries to prevent clobbering when multiple ABIs +are installed in the same environment. This kind of output separation is more +difficult to implement. ```yaml {% set name = "libavif" %} @@ -146,7 +147,7 @@ multiple recipe outputs which is more difficult to implement. {% set so_name_major = so_version.split('.')[0] %} package: - name: {{ name|lower ~ so_name_major }} + name: {{ name|lower }}-splitme version: {{ version }} build: @@ -154,21 +155,39 @@ build: run_exports: - {{ pin_subpackage(name|lower ~ so_name_major) }} -test: - commands: - - test -f ${PREFIX}/lib/libavif.so.{{ so_name_major }} # [linux] - - test -f ${PREFIX}/lib/libavif.so.{{ so_version }} # [linux] +# build all outputs, but don't install in "splitme" recipe outputs: - - {{ name|lower ~ so_name_major }} - # This metapackage allows recipes to continue using the unversioned name + # The unversioned name output contains docs, bins, and headers according to + # the API version. - {{ name|lower }} - build: - run_exports: - - {{ pin_subpackage(name|lower ~ so_name_major) }} - requirements: - run: - - {{ name|lower ~ so_name_major }} + + script: install-headers-docs-bins.sh + + requirements: + run: + - {{ pin_subpackage(name|lower ~ so_name_major, exact=True) }} + + test: + commands: + - test -f ${PREFIX}/include/libavif.h # [linux] + # The top-level package must not contain libraries to prevent clobbering + - ! test -f ${PREFIX}/lib/libavif.so.{{ so_name_major }} # [linux] + - ! test -f ${PREFIX}/lib/libavif.so.{{ so_version }} # [linux] + - ! test -f ${PREFIX}/lib/libavif.so # [linux] + + # This ABI-named secondary output contains the shared libaries only + - {{ name|lower ~ so_name_major }} + + script: install-libraries-only.sh + + test: + commands: + - test -f ${PREFIX}/lib/libavif.so.{{ so_name_major }} # [linux] + - test -f ${PREFIX}/lib/libavif.so.{{ so_version }} # [linux] + # Versioned libraries only to prevent clobbering + - ! test -f ${PREFIX}/lib/libavif.so # [linux] + - ! test -f ${PREFIX}/include/libavif.h # [linux] ``` ### Prepending ABI to package version