From e0ca3e128fd0161bddd2295399cdb3dcda746af5 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 23 Apr 2025 15:13:54 -0400 Subject: [PATCH 01/14] PEP 808: Partially Dynamic Metadata Signed-off-by: Henry Schreiner --- .github/CODEOWNERS | 1 + peps/pep-0808.rst | 323 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 324 insertions(+) create mode 100644 peps/pep-0808.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f20767aed5c..8b3689269c9 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -680,6 +680,7 @@ peps/pep-0801.rst @warsaw peps/pep-0802.rst @AA-Turner peps/pep-0803.rst @encukou peps/pep-0804.rst @pradyunsg +peps/pep-0808.rst @FFY00 # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst new file mode 100644 index 00000000000..eaea061dbef --- /dev/null +++ b/peps/pep-0808.rst @@ -0,0 +1,323 @@ +PEP: 808 +Title: Partially dynamic project metadata +Author: Henry Schreiner +Sponsor: Filipe Laíns +PEP-Delegate: Paul Moore +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 19-Sep-2025 + + + +Abstract +======== + +This PEP relaxes the constraint on dynamic metadata listed in the ``[project]`` +section in ``pyproject.toml`` to allow the static portion of mixed metadata to +be defined in the normal location. + +This allows users to opt into allowing a backend to extend metadata while still +keeping the static portions of the metadata defined in the standard location in +``pyproject.toml``, and allows inspection tools to still be able to process the +static portions of the metadata. + + +Motivation +========== + +In the core metadata specification originally set out in PEP 621, metadata can +be specified in three ways. First, it can be listed in the ``[project]`` table. +This makes it statically inferable, meaning any tool (not just the build +backend) can reliably compute the value. Second, a field can be listed in the +``project.dynamic`` list, which allows the build backend to compute the value. +Finally, a value could be missing from both the ``project`` table and the +``project.dynamic`` list, in which case the matching metadata is guaranteed to +be empty. + +This system provided (at least) two benefits to Python packaging. A standard +specification that all major backends have now adopted make teaching much +easier; a single tutorial is now sufficient to cover the metadata portion of +configuring any backend. Users can now switch from a general purpose backend to +a specialized backend without changing their static metadata. Tooling like +schema validation tools can verify and catch configuration mistakes. + +The second benefit is improved support for static tools that read the source +files looking for metadata. This is useful for dependency chain analysis, such +as creating "used by" and "uses" graphs. It is used for code quality tooling to +detect the minimum supported version of Python. It is used by `cibuildwheel +`__ to automatically avoid building wheels that are not +supported. It is not used, however, to avoid wheel builds when the SDist is +available; that was addressed by METADATA 2.2, and a ``Dynamic`` field in the +SDist metadata that lets a tool know if the metadata can change when making a +wheel - this is an easy mistake to make due to the similarity of the names. + +Due to the rapidly increasing popularity of PEP 621 metadata, support from all +major backends, and a rise of backends supporting complex compiled extensions, +an issue with the restrictions applied in PEP 621 is becoming more apparent. In +PEP 621, the metadata choice is all-or-nothing; metadata must be completely +static, or listed in the dynamic field and completely absent from the static +definition. For the most common use cases, this is fine; there is little +benefit to set the ``version`` statically if you are going to override it +dynamically. If you are using a custom README processor to filter or modify the +README for proper display, it's not a big deal to have to specify the +configuration in a custom ``tool.*`` section. But there is a specific class of +cases where the all-or-nothing approach is problematic: lists of items where +the backend needs to add items are currently forced to be fully dynamically +specified (that is, in a backend-specific configuration location). This causes +both of the original benefits (standard location and static tooling support) to +be lost. + +Rationale +========= + + +PEP 621 includes the following statement: + + In an earlier version of this PEP, tools were allowed to extend data for + fields. For instance, build back-ends could take the version number and add + a local version for when they built the wheel. Tools could also add more + trove classifiers for things like the license or supported Python versions. + + In the end, though, it was thought better to start out stricter and + contemplate loosening how static the data could be considered based on + real-world usage. + +In this PEP, we are proposing a limited and explicit loosening of the +``[project]`` table and ``project.dynamic`` list based on real world use cases. + +Every list and table with arbitrary keys will now be allowed to be specified +both statically, in the ``[project]`` table, and in the ``project.dynamic`` +list. If it is present in both places, the build backend can extend list items +and add new keys, but not modify existing list items or strings. We are also +including one special case; the ``license`` field when specified as a string is +an SPDX expression, which has a similar concept of extension with ``AND``. + + +Use Cases +--------- + +There is an entire class of metadata fields where advanced use cases +would really benefit from a relaxation of this rule. + +- Pinning dependency requirements when building the wheel +- Generating extra scripts from a build system (proposed for scikit-build-core) +- Adding entry points dynamically (validate-pyproject-schema-store could have + used this) +- Adding dependencies or optional dependencies based on configuration (such as + making an all dependency, or reading dependencies from dependency-groups, for + example) +- Adding classifiers; some backends can compute classifiers from other places + and inject them (Poetry) +- Adding license files to the wheel based on what libraries are linked in +- Adding licenses based on vendored/linked code (the setuptools repo might be + able to use this.) +- Adding SBom's when building - PEP 770 had to remove the pyproject.toml field + specifically because you _want_ the build tool to add these, so the + ``[project]`` table setting would be useless, you'd almost never be able to use + it. + +All of these use cases have a similar feature: they are adding something +dynamically to a fixed list (possibly a narrower pin for the dependency case). +With the exception of the recently added ``license`` field, these are all lists +or tables that need extending. + +You can implement these today, but it requires providing a completely separate +configuration location for the non-extended portion, and static analysis tools +lose the ability to detect anything. Since the current solution is to move all +the metadata out of the standard field, this proposal will increase the +availability of metadata for static tooling. + + +Example +------- + +For example, let's say you want to allow an imaginary build backend +(``my-build-backend``) to pin to the supported build of PyTorch. Before this +PEP, you could do this: + +.. code-block:: toml + + [project] + dynamic = ["dependencies"] + + [tool.my-build-backend] + original-dependencies = ["torch", "packaging"] + pin-to-build-versions = ["torch=={exact}"] + +Static tooling no longer can tell that ``torch`` and ``packaging`` are runtime +dependencies, and the build backend had to duplicate the dependency table, +making it harder for users to learn and read; the standardized place proposed +by PEP 621 and adopted by all major build backends is lost. + +With this PEP, this could now be specified like this: + +.. code-block:: toml + + [project] + dependencies = ["torch", "packaging"] + dynamic = ["dependencies"] + + [tool.my-build-backend] + pin-to-build-versions = ["torch=={exact}"] + +Static tooling can now detect the static dependencies, and the build backend no +longer needs to create and document a new location for the standard +``project.dependencies`` field. + +Future Updates +-------------- + +New fields added to the ``[project]`` table in future PEPs should include how +to handle additive dynamic metadata if they need special handling. Currently, +there is only one special case (``license``), so this is not expected to be +common. + +Loosening this rule to allow purely additive metadata should address many of +the use cases that have been seen in practice. If further changes are needed, +this can be revisited in a future PEP; this PEP neither recommends or precludes +future updates like this. + + +Specification +============= + +Any field that is comprised of a list or a table with arbitrary entries will +now be allowed to be present in both the ``[project]`` table and the +``project.dynamic`` list. If a field is present in both places, then the build +backend is allowed to extend the list or table with new entries, but not remove +entries, or modify the entries in a way that causes them to be removed. Tables +of arrays allow adding a new table entry or extending an existing array. As a +special case, the ``license`` field, when set to a string SPDX expression, can be +extended logically, as well. + +The fields that are arrays or tables with arbitrary entries are: + +* ``authors``, ``maintainers``: New author tables can be added to the list. + Existing authors cannot be modified (list of tables with pre-defined keys). +* ``classifiers``: Classifiers can be added to the list. +* ``dependencies``: New dependencies can be added, including more tightly + constrained existing dependencies. Backends are allowed to simplify + duplicated items with different constraints as long as it is strictly + identical to the original plus the duplicated items. +* ``entry-points``: Entry points can be added, to either new or existing + groups. Existing entry points cannot be changed or removed. +* ``keywords``: Keywords can be added to the list. +* ``license-files``: Files can be added to the list. +* ``license`` (string, special case): The license expression can be extended. + An existing license cannot be logically excluded. +* ``optional-dependencies``: A new extra or new items can be added to a + existing extra. +* ``scripts``, ``gui-scripts``: New scripts can be added. Existing ones cannot + be changed or removed. +* ``urls``: New urls can be added. Existing ones cannot be changed or removed. + +This is entirely opt-in by listing the field in ``dynamic``; without that, the +metadata continues to be entirely static. + +A backend SHOULD warn if a field is specified and it does not know how to +extend that field, to protect against possible user error, and MAY throw an +error instead. It should be noted, however, that mistakenly adding a field to +the ``dynamic`` array is not a serious mistake, as it only limits the ability +of a static tool to ensure completeness, so it is up to the discretion of the +backend if this warrants an error. + +Static analysis tools, when detecting a field is both specified and in the +``project.dynamic`` array, must assume the field could be extended with new +entries when the package is built. + + +Reference Implementation +======================== + +The choice to support dynamic metadata for each field is already left up to +backends, and this PEP simply relaxes restrictions on what a backend is allowed to +do with dynamic metadata. + +The `pyproject-metadata `__ project, which is used by +several build backends, will need to modify the correctness check to account +for the possible extensions; this is in `a draft PR `__. + +The `dynamic-metadata `__ project, which provides a plugin +system that backends can use to share dynamic metadata plugins, was designed to +allow this possibility, and a similar PR to the one above will allow additive +metadata. + +Backwards Compatibility +======================= + +This does not affect any existing ``pyproject.toml``'s, since this was strictly +not allowed before this PEP. + +Security Implications +===================== + +There are no security concerns that are not already present, as this just adds a static +component to existing dynamic metadata support. + +How to Teach This +================= + +The current guides that state metadata must not be listed in both ``[project]`` +and ``project.dynamic`` can be updated to say that some fields can be extended +by ``project.dynamic``. Since dynamic metadata is already an advanced concept, +this will likely not affect most existing tutorial material aimed at +introductory packaging. + +The ``pyproject.toml`` `specification `__ will be updated to +include the behavior of fields when specified and also listed in the dynamic +field. An example of such an update for the license field: + +* When also listed in ``dynamic``: Can be extended with ``AND`` + + +Rejected Ideas +============== + +Special case some fields +------------------------ + +This has come up specifically for the pinning build dependency use case, but +could also be applied to more of the use cases listed. This would not cover all +the use cases seen, though, and an explicit, opt-in approach is better for +static tooling. + + +Include more string fields +-------------------------- + +There are two more string fields, namely ``version`` and ``requires-python`` +(``name`` is not allowed to be specified dynamically). There did not seem to be +a pressing practical need to allow these to be extended as with ``license``, +and logical extensions are not as clear, so they are not included in this PEP. + + +Fully remove restrictions on backends +------------------------------------- + +Another option would be to simply allow backends to do whatever they wanted if +a field is statically defined and in the dynamic array. This would sacrifice +the ability for static tooling to infer anything about the field, and could +potentially confuse users by allowing the backend to ignore or change what they +entered. This is not worse than the status quo for static tooling and dynamic +metadata, but the current proposal improves the ability of static tooling to +infer some things about dynamic fields. Knowing some of the dependencies is +better for most applications than not knowing anything about the dependencies, +for example. + + +References +========== + +.. _cibuildwheel: https://cibuildwheel.pypa.io +.. _pyprojectspec: https://packaging.python.org/en/latest/specifications/pyproject-toml +.. _pyprojectmetadata: https://github.com/pypa/pyproject-metadata +.. _pyprojectmetadatapr: +.. _dynamicmetadata: https://github.com/scikit-build/dynamic-metadata + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 8f4e7346c3fd63de3d3e0c6092eaa49513863499 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 23 Apr 2025 15:45:27 -0400 Subject: [PATCH 02/14] chore: add link to PR Signed-off-by: Henry Schreiner --- peps/pep-0808.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index eaea061dbef..d70cb354765 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -312,7 +312,7 @@ References .. _cibuildwheel: https://cibuildwheel.pypa.io .. _pyprojectspec: https://packaging.python.org/en/latest/specifications/pyproject-toml .. _pyprojectmetadata: https://github.com/pypa/pyproject-metadata -.. _pyprojectmetadatapr: +.. _pyprojectmetadatapr: https://github.com/pypa/pyproject-metadata/pull/241 .. _dynamicmetadata: https://github.com/scikit-build/dynamic-metadata From fe4f6ef2c932537f15636d6f59e7f78baefa59a8 Mon Sep 17 00:00:00 2001 From: Cristian Le Date: Fri, 25 Apr 2025 10:18:45 +0200 Subject: [PATCH 03/14] Clarify the scope in the abstract --- peps/pep-0808.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index d70cb354765..4e0ab5ea73d 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -15,7 +15,8 @@ Abstract This PEP relaxes the constraint on dynamic metadata listed in the ``[project]`` section in ``pyproject.toml`` to allow the static portion of mixed metadata to -be defined in the normal location. +be defined in the normal location if the field is a ``list`` type, a ``dict`` +type or ``license`` by having the dynamic fields extend the static ones. This allows users to opt into allowing a backend to extend metadata while still keeping the static portions of the metadata defined in the standard location in From 17c7445c451cfeb57abcee10420e227aaf95e7a8 Mon Sep 17 00:00:00 2001 From: Cristian Le Date: Fri, 25 Apr 2025 10:58:59 +0200 Subject: [PATCH 04/14] Clarify the torch example --- peps/pep-0808.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 4e0ab5ea73d..844a6b6d50c 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -146,6 +146,13 @@ PEP, you could do this: original-dependencies = ["torch", "packaging"] pin-to-build-versions = ["torch=={exact}"] +Which could effectively expand to + +.. code-block:: toml + + [project] + dependencies = ["torch", "packaging", "torch==2.8.0a0+gitf38dae76"] + Static tooling no longer can tell that ``torch`` and ``packaging`` are runtime dependencies, and the build backend had to duplicate the dependency table, making it harder for users to learn and read; the standardized place proposed @@ -164,7 +171,7 @@ With this PEP, this could now be specified like this: Static tooling can now detect the static dependencies, and the build backend no longer needs to create and document a new location for the standard -``project.dependencies`` field. +``project.dependencies`` field (the ``original-dependencies``). Future Updates -------------- From 7a32f23068742fe7cb4f9ffe6faef066d9c5fae5 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 2 May 2025 15:45:14 -0400 Subject: [PATCH 05/14] Apply suggestions from code review --- peps/pep-0808.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 844a6b6d50c..c2785c610a4 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -16,7 +16,7 @@ Abstract This PEP relaxes the constraint on dynamic metadata listed in the ``[project]`` section in ``pyproject.toml`` to allow the static portion of mixed metadata to be defined in the normal location if the field is a ``list`` type, a ``dict`` -type or ``license`` by having the dynamic fields extend the static ones. +type or ``license`` by having the dynamic fields extend the static ones. This allows users to opt into allowing a backend to extend metadata while still keeping the static portions of the metadata defined in the standard location in @@ -146,7 +146,7 @@ PEP, you could do this: original-dependencies = ["torch", "packaging"] pin-to-build-versions = ["torch=={exact}"] -Which could effectively expand to +Which would effectively expand to .. code-block:: toml @@ -171,7 +171,7 @@ With this PEP, this could now be specified like this: Static tooling can now detect the static dependencies, and the build backend no longer needs to create and document a new location for the standard -``project.dependencies`` field (the ``original-dependencies``). +``project.dependencies`` field (the ``original-dependencies`` field above, for example). Future Updates -------------- From d92b95ab6a95f5a021d4030587dc4dc1d79b6b13 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 2 May 2025 17:31:07 -0400 Subject: [PATCH 06/14] fix: a bit more about back-compat Signed-off-by: Henry Schreiner --- peps/pep-0808.rst | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index c2785c610a4..19632862c7c 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -15,8 +15,8 @@ Abstract This PEP relaxes the constraint on dynamic metadata listed in the ``[project]`` section in ``pyproject.toml`` to allow the static portion of mixed metadata to -be defined in the normal location if the field is a ``list`` type, a ``dict`` -type or ``license`` by having the dynamic fields extend the static ones. +be defined in the normal location if the field is a table, array or ``license`` +by having the dynamic fields extend the static ones. This allows users to opt into allowing a backend to extend metadata while still keeping the static portions of the metadata defined in the standard location in @@ -239,8 +239,8 @@ Reference Implementation ======================== The choice to support dynamic metadata for each field is already left up to -backends, and this PEP simply relaxes restrictions on what a backend is allowed to -do with dynamic metadata. +backends, and this PEP simply relaxes restrictions on what a backend is allowed +to do with dynamic metadata. The `pyproject-metadata `__ project, which is used by several build backends, will need to modify the correctness check to account @@ -257,11 +257,19 @@ Backwards Compatibility This does not affect any existing ``pyproject.toml``'s, since this was strictly not allowed before this PEP. +When users adopt this in a ``pyproject.toml``, the backend must support it; an +error will be correctly generated if it doesn't following the previous +standard. Frontends were never required to throw an error, though some +frontends may need to be updated to benefit from the partially static metadata. + +Using metadata from SDists or wheels is unaffected. The METADATA version does +not need to be incremented. + Security Implications ===================== -There are no security concerns that are not already present, as this just adds a static -component to existing dynamic metadata support. +There are no security concerns that are not already present, as this just adds +a static component to existing dynamic metadata support. How to Teach This ================= From 9b65fd095ae4c1b4745f6d34106d03c9c4ef2efc Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 7 May 2025 00:46:03 -0400 Subject: [PATCH 07/14] feat: fill out more of the PEP Signed-off-by: Henry Schreiner --- peps/pep-0808.rst | 174 ++++++++++++++++++++++++++++++---------------- 1 file changed, 115 insertions(+), 59 deletions(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 19632862c7c..2f04885064b 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -27,17 +27,17 @@ static portions of the metadata. Motivation ========== -In the core metadata specification originally set out in PEP 621, metadata can -be specified in three ways. First, it can be listed in the ``[project]`` table. -This makes it statically inferable, meaning any tool (not just the build -backend) can reliably compute the value. Second, a field can be listed in the -``project.dynamic`` list, which allows the build backend to compute the value. -Finally, a value could be missing from both the ``project`` table and the -``project.dynamic`` list, in which case the matching metadata is guaranteed to -be empty. - -This system provided (at least) two benefits to Python packaging. A standard -specification that all major backends have now adopted make teaching much +In the core metadata specification originally set out in :pep:`621`, metadata +can be specified in three ways. First, it can be listed in the ``[project]`` +table. This makes it statically inferable, meaning any tool (not just the +build backend) can reliably compute the value. Second, a field can be listed in +the ``project.dynamic`` list, which allows the build backend to compute the +value. Finally, a value could be missing from both the ``project`` table and +the ``project.dynamic`` list, in which case the matching metadata is guaranteed +to be empty. + +This system provided two important benefits to Python packaging. A standard +specification that all major backends have now adopted makes teaching much easier; a single tutorial is now sufficient to cover the metadata portion of configuring any backend. Users can now switch from a general purpose backend to a specialized backend without changing their static metadata. Tooling like @@ -46,17 +46,17 @@ schema validation tools can verify and catch configuration mistakes. The second benefit is improved support for static tools that read the source files looking for metadata. This is useful for dependency chain analysis, such as creating "used by" and "uses" graphs. It is used for code quality tooling to -detect the minimum supported version of Python. It is used by `cibuildwheel -`__ to automatically avoid building wheels that are not -supported. It is not used, however, to avoid wheel builds when the SDist is -available; that was addressed by METADATA 2.2, and a ``Dynamic`` field in the -SDist metadata that lets a tool know if the metadata can change when making a -wheel - this is an easy mistake to make due to the similarity of the names. - -Due to the rapidly increasing popularity of PEP 621 metadata, support from all +detect the minimum supported version of Python. It is used by cibuildwheel_ to +automatically avoid building wheels that are not supported. It is not used, +however, to avoid wheel builds when the SDist is available; that was addressed +by METADATA 2.2, which a ``Dynamic`` field in the SDist metadata that lets a +tool know if the metadata can change when making a wheel - this is an easy +mistake to make due to the similarity of the names. + +Due to the rapidly increasing popularity of the project table, support from all major backends, and a rise of backends supporting complex compiled extensions, -an issue with the restrictions applied in PEP 621 is becoming more apparent. In -PEP 621, the metadata choice is all-or-nothing; metadata must be completely +an issue with the restrictions applied in :pep:`621` is becoming more apparent. +In PEP 621, the metadata choice is all-or-nothing; metadata must be completely static, or listed in the dynamic field and completely absent from the static definition. For the most common use cases, this is fine; there is little benefit to set the ``version`` statically if you are going to override it @@ -73,7 +73,7 @@ Rationale ========= -PEP 621 includes the following statement: +:pep:`621` includes the following statement: In an earlier version of this PEP, tools were allowed to extend data for fields. For instance, build back-ends could take the version number and add @@ -85,38 +85,46 @@ PEP 621 includes the following statement: real-world usage. In this PEP, we are proposing a limited and explicit loosening of the -``[project]`` table and ``project.dynamic`` list based on real world use cases. +restrictions on the ``[project]`` table and ``project.dynamic`` list. -Every list and table with arbitrary keys will now be allowed to be specified -both statically, in the ``[project]`` table, and in the ``project.dynamic`` -list. If it is present in both places, the build backend can extend list items -and add new keys, but not modify existing list items or strings. We are also -including one special case; the ``license`` field when specified as a string is -an SPDX expression, which has a similar concept of extension with ``AND``. +Every list and every table with arbitrary keys will now be allowed to be +specified both statically, in the ``[project]`` table, and in the +``project.dynamic`` list. If it is present in both places, the build backend +can extend list items and add new keys, but not modify existing list items or +strings. We are also including one special case; the ``license`` field when +specified as a string is an SPDX expression, which has a similar concept of +extension with ``AND``. Use Cases --------- There is an entire class of metadata fields where advanced use cases -would really benefit from a relaxation of this rule. - -- Pinning dependency requirements when building the wheel -- Generating extra scripts from a build system (proposed for scikit-build-core) -- Adding entry points dynamically (validate-pyproject-schema-store could have - used this) +would really benefit from a relaxation of this rule. Here are some use +cases that have come up: + +- Pinning dependency requirements when building the wheel. When building + PyTorch_ extensions, for example, the version you build with adds a constraint + to the wheel you create that is not present with the SDist. +- Generating extra scripts from a build system (this is a currently proposed in + scikit-build-core_). +- Adding entry points dynamically (validate-pyproject-schema-store_ could have + used this to generate an entry point for each schema present in the package.) - Adding dependencies or optional dependencies based on configuration (such as making an all dependency, or reading dependencies from dependency-groups, for - example) + example). Adding constraints can also be useful; pybind11_ uses adds a ``global`` + extra that pins ``pybind11-global==``, as both packages are in the + same repository and released in sync. - Adding classifiers; some backends can compute classifiers from other places - and inject them (Poetry) -- Adding license files to the wheel based on what libraries are linked in -- Adding licenses based on vendored/linked code (the setuptools repo might be + and inject them (Poetry_ being the best known example). +- Adding license files to the wheel based on what libraries are linked in (this + is an active discussion in followup to :pep:`639`). +- Similarly, adding licenses based on vendored code (the setuptools_ repo might be able to use this.) -- Adding SBom's when building - PEP 770 had to remove the pyproject.toml field - specifically because you _want_ the build tool to add these, so the - ``[project]`` table setting would be useless, you'd almost never be able to use - it. +- Adding SBom's when building - :pep:`770` had to remove the ``pyproject.toml`` + field specifically because you _want_ the build tool to add these, so the + ``[project]`` table setting would be useless, you'd almost never be able to + use it. All of these use cases have a similar feature: they are adding something dynamically to a fixed list (possibly a narrower pin for the dependency case). @@ -130,11 +138,11 @@ the metadata out of the standard field, this proposal will increase the availability of metadata for static tooling. -Example -------- +Example: pinning +---------------- For example, let's say you want to allow an imaginary build backend -(``my-build-backend``) to pin to the supported build of PyTorch. Before this +(``my-build-backend``) to pin to the supported build of PyTorch_. Before this PEP, you could do this: .. code-block:: toml @@ -151,12 +159,12 @@ Which would effectively expand to .. code-block:: toml [project] - dependencies = ["torch", "packaging", "torch==2.8.0a0+gitf38dae76"] + dependencies = ["torch", "packaging", "torch==2.8.0"] Static tooling no longer can tell that ``torch`` and ``packaging`` are runtime dependencies, and the build backend had to duplicate the dependency table, making it harder for users to learn and read; the standardized place proposed -by PEP 621 and adopted by all major build backends is lost. +by :pep:`621` and adopted by all major build backends is lost. With this PEP, this could now be specified like this: @@ -171,7 +179,35 @@ With this PEP, this could now be specified like this: Static tooling can now detect the static dependencies, and the build backend no longer needs to create and document a new location for the standard -``project.dependencies`` field (the ``original-dependencies`` field above, for example). +``project.dependencies`` field (the ``original-dependencies`` field above, for +example). + + +Example: licenses +----------------- + +As another example, a package could specify it's license(s): + +.. code-block:: toml + + [project] + license = "MIT" + license-files = ["LICENSE"] + + dynamic = ["license", "license-files"] + + [tool.my-build-backend] + license-search-sdist = ["vendor/*"] + linked-wheel-licenses = true + + +Now a build-backend can search for licenses in the vendor folder, and add them +automatically to the license expression and/or file list. If it supports this +when creating the wheel, it can specify the necessary fields in the ``Dynamic`` +``METADATA`` (2.2+) field and then add the licenses that are linked in when +creating the wheel. While you can do this today, you can't specify the +project's own license statically when you do this; this PEP allows both. + Future Updates -------------- @@ -210,14 +246,14 @@ The fields that are arrays or tables with arbitrary entries are: identical to the original plus the duplicated items. * ``entry-points``: Entry points can be added, to either new or existing groups. Existing entry points cannot be changed or removed. +* ``scripts``, ``gui-scripts``: New scripts can be added. Existing ones cannot + be changed or removed. * ``keywords``: Keywords can be added to the list. * ``license-files``: Files can be added to the list. * ``license`` (string, special case): The license expression can be extended. An existing license cannot be logically excluded. * ``optional-dependencies``: A new extra or new items can be added to a existing extra. -* ``scripts``, ``gui-scripts``: New scripts can be added. Existing ones cannot - be changed or removed. * ``urls``: New urls can be added. Existing ones cannot be changed or removed. This is entirely opt-in by listing the field in ``dynamic``; without that, the @@ -230,11 +266,13 @@ the ``dynamic`` array is not a serious mistake, as it only limits the ability of a static tool to ensure completeness, so it is up to the discretion of the backend if this warrants an error. +Build backends MUST error if a string field is specified twice with differing +contents, such as the same script being specified statically and dynamically. + Static analysis tools, when detecting a field is both specified and in the ``project.dynamic`` array, must assume the field could be extended with new entries when the package is built. - Reference Implementation ======================== @@ -242,11 +280,11 @@ The choice to support dynamic metadata for each field is already left up to backends, and this PEP simply relaxes restrictions on what a backend is allowed to do with dynamic metadata. -The `pyproject-metadata `__ project, which is used by +The pyproject-metadata_ project, which is used by several build backends, will need to modify the correctness check to account for the possible extensions; this is in `a draft PR `__. -The `dynamic-metadata `__ project, which provides a plugin +The dynamic-metadata_ project, which provides a plugin system that backends can use to share dynamic metadata plugins, was designed to allow this possibility, and a similar PR to the one above will allow additive metadata. @@ -261,6 +299,8 @@ When users adopt this in a ``pyproject.toml``, the backend must support it; an error will be correctly generated if it doesn't following the previous standard. Frontends were never required to throw an error, though some frontends may need to be updated to benefit from the partially static metadata. +Some frontends and other tooling may need updating, such as schema +validation, just like other ``pyproject.toml`` PEPs. Using metadata from SDists or wheels is unaffected. The METADATA version does not need to be incremented. @@ -290,8 +330,8 @@ field. An example of such an update for the license field: Rejected Ideas ============== -Special case some fields ------------------------- +Special case some fields without adding dynamic +----------------------------------------------- This has come up specifically for the pinning build dependency use case, but could also be applied to more of the use cases listed. This would not cover all @@ -306,6 +346,9 @@ There are two more string fields, namely ``version`` and ``requires-python`` (``name`` is not allowed to be specified dynamically). There did not seem to be a pressing practical need to allow these to be extended as with ``license``, and logical extensions are not as clear, so they are not included in this PEP. +Fixed key tables, like the deprecated ``license.text``/``license.file`` or +``readme.text``/``readme.file`` also have no clear benefit being partially +dynamic. Fully remove restrictions on backends @@ -322,15 +365,28 @@ better for most applications than not knowing anything about the dependencies, for example. +Add a general mechanism to specify dynamic-metadata +--------------------------------------------------- + +This PEP does not cover methods to specify dynamic metadata; that continues to +be entirely up to the backend. An earlier draft proposal did this, but it was +deemed better to develop that as a library (dynamic-metadata_, for the curious) +instead. This may be revisited in the future. + References ========== .. _cibuildwheel: https://cibuildwheel.pypa.io .. _pyprojectspec: https://packaging.python.org/en/latest/specifications/pyproject-toml -.. _pyprojectmetadata: https://github.com/pypa/pyproject-metadata +.. _pyproject-metadata: https://github.com/pypa/pyproject-metadata .. _pyprojectmetadatapr: https://github.com/pypa/pyproject-metadata/pull/241 -.. _dynamicmetadata: https://github.com/scikit-build/dynamic-metadata - +.. _dynamic-metadata: https://github.com/scikit-build/dynamic-metadata +.. _PyTorch: https://pytorch.org/ +.. _scikit-build-core: https://github.com/scikit-build/scikit-build-core +.. _validate-pyproject-schema-store: https://pypi.org/project/validate-pyproject-schema-store/ +.. _pybind11: https://github.com/pybind/pybind11 +.. _Poetry: https://python-poetry.org/ +.. _setuptools: https://github.com/pypa/setuptools Copyright ========= From 0a24ea7722518ed5c122494b250f2a9af7b725fa Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 7 May 2025 01:48:17 -0400 Subject: [PATCH 08/14] Add LecrisUT to the authors --- peps/pep-0808.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 2f04885064b..14055003a35 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -1,6 +1,7 @@ PEP: 808 Title: Partially dynamic project metadata -Author: Henry Schreiner +Author: Henry Schreiner , + Cristian Le Sponsor: Filipe Laíns PEP-Delegate: Paul Moore Status: Draft From b04dabeb116b0cd47ce407df10ad84be233e5b8b Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 22 May 2025 18:17:56 -0400 Subject: [PATCH 09/14] fix: remove narrowing/simplification Signed-off-by: Henry Schreiner --- peps/pep-0808.rst | 66 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 19 deletions(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 14055003a35..5d005f89397 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -155,12 +155,21 @@ PEP, you could do this: original-dependencies = ["torch", "packaging"] pin-to-build-versions = ["torch=={exact}"] -Which would effectively expand to +Which would effectively expand to the following SDist metadata: -.. code-block:: toml +.. code-block:: text - [project] - dependencies = ["torch", "packaging", "torch==2.8.0"] + Dynamic: Requires-Dist + Requires-Dist: packaging + Requires-Dist: torch + +Which would then could make a wheel with this: + +.. code-block:: text + + Requires-Dist: packaging + Requires-Dist: torch + Requires-Dist: torch==2.8.0 Static tooling no longer can tell that ``torch`` and ``packaging`` are runtime dependencies, and the build backend had to duplicate the dependency table, @@ -223,6 +232,12 @@ the use cases that have been seen in practice. If further changes are needed, this can be revisited in a future PEP; this PEP neither recommends or precludes future updates like this. +Terminology +=========== + +The keywords "MUST", "MUST NOT", "REQUIRED", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" +in this document are to be interpreted as described in :rfc:`2119`. Specification ============= @@ -230,11 +245,11 @@ Specification Any field that is comprised of a list or a table with arbitrary entries will now be allowed to be present in both the ``[project]`` table and the ``project.dynamic`` list. If a field is present in both places, then the build -backend is allowed to extend the list or table with new entries, but not remove -entries, or modify the entries in a way that causes them to be removed. Tables -of arrays allow adding a new table entry or extending an existing array. As a -special case, the ``license`` field, when set to a string SPDX expression, can be -extended logically, as well. +backend is allowed to insert entries into the list or table, but not remove +entries, reorder entries, or modify the entries. Tables of arrays allow adding +a new table entry or extending an existing array according to the rules above. +As a special case, the ``license`` field, when set to a string SPDX expression, +can be extended logically, as well. The fields that are arrays or tables with arbitrary entries are: @@ -242,9 +257,7 @@ The fields that are arrays or tables with arbitrary entries are: Existing authors cannot be modified (list of tables with pre-defined keys). * ``classifiers``: Classifiers can be added to the list. * ``dependencies``: New dependencies can be added, including more tightly - constrained existing dependencies. Backends are allowed to simplify - duplicated items with different constraints as long as it is strictly - identical to the original plus the duplicated items. + constrained existing dependencies. * ``entry-points``: Entry points can be added, to either new or existing groups. Existing entry points cannot be changed or removed. * ``scripts``, ``gui-scripts``: New scripts can be added. Existing ones cannot @@ -257,8 +270,8 @@ The fields that are arrays or tables with arbitrary entries are: existing extra. * ``urls``: New urls can be added. Existing ones cannot be changed or removed. -This is entirely opt-in by listing the field in ``dynamic``; without that, the -metadata continues to be entirely static. +To add items, users must opt-in by listing the field in ``dynamic``; without +that, the metadata continues to be entirely static. A backend SHOULD warn if a field is specified and it does not know how to extend that field, to protect against possible user error, and MAY throw an @@ -267,12 +280,14 @@ the ``dynamic`` array is not a serious mistake, as it only limits the ability of a static tool to ensure completeness, so it is up to the discretion of the backend if this warrants an error. -Build backends MUST error if a string field is specified twice with differing -contents, such as the same script being specified statically and dynamically. - Static analysis tools, when detecting a field is both specified and in the -``project.dynamic`` array, must assume the field could be extended with new -entries when the package is built. +``project.dynamic`` array, SHOULD assume the field is incomplete, allowing for +new entries to be present when the package is built. + +The ``Dynamic`` field, as specified in :pep:`643`, is unaffected by this PEP, +and backends can continue to fill it as they chose. However, a backend MUST +ensure that both the SDist and the wheel metadata include the static metadata +portion of the project table. Reference Implementation ======================== @@ -365,6 +380,19 @@ infer some things about dynamic fields. Knowing some of the dependencies is better for most applications than not knowing anything about the dependencies, for example. +Allow simplifications +--------------------- + +An earlier draft of this PEP had a clause allowing backends to simplify some +types of fields; most notably dependency specifiers would have allowed +"tightening", such as ``torch`` being replaced by ``torch>=1.2``, for example. +. This was removed due to it being impossible to ensure a variation will +resolve identically on all resolvers within the current specification, and to +simplify the contract with backends. Any other simplifications would be purely +cosmetic, and so were left out. The order in the current PEP is now required to +match the original static metadata, with the dynamic portion only allowing +insertions. + Add a general mechanism to specify dynamic-metadata --------------------------------------------------- From f9ab6da1ff2f71746488f84517e45f74501a2321 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 22 May 2025 18:23:21 -0400 Subject: [PATCH 10/14] fix: mention toga Signed-off-by: Henry Schreiner --- peps/pep-0808.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 5d005f89397..89906c93bfb 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -115,7 +115,9 @@ cases that have come up: making an all dependency, or reading dependencies from dependency-groups, for example). Adding constraints can also be useful; pybind11_ uses adds a ``global`` extra that pins ``pybind11-global==``, as both packages are in the - same repository and released in sync. + same repository and released in sync. toga_ is a collection of packages that + currently is unable to set any static dependencies due to the same sort of + pinning problem. - Adding classifiers; some backends can compute classifiers from other places and inject them (Poetry_ being the best known example). - Adding license files to the wheel based on what libraries are linked in (this @@ -416,6 +418,7 @@ References .. _pybind11: https://github.com/pybind/pybind11 .. _Poetry: https://python-poetry.org/ .. _setuptools: https://github.com/pypa/setuptools +.. _toga: https://github.com/beeware/toga Copyright ========= From 7852ce95ad3f0131335fcaaa67f5fec9b36ef611 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 24 Sep 2025 13:54:29 -0400 Subject: [PATCH 11/14] refactor: address a few bits of feedback Signed-off-by: Henry Schreiner --- peps/pep-0808.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 89906c93bfb..69e33951a86 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -198,7 +198,7 @@ example). Example: licenses ----------------- -As another example, a package could specify it's license(s): +As another example, a package could specify its license(s): .. code-block:: toml @@ -275,12 +275,10 @@ The fields that are arrays or tables with arbitrary entries are: To add items, users must opt-in by listing the field in ``dynamic``; without that, the metadata continues to be entirely static. -A backend SHOULD warn if a field is specified and it does not know how to -extend that field, to protect against possible user error, and MAY throw an -error instead. It should be noted, however, that mistakenly adding a field to -the ``dynamic`` array is not a serious mistake, as it only limits the ability -of a static tool to ensure completeness, so it is up to the discretion of the -backend if this warrants an error. +A backend SHOULD error if a field is specified and it does not support +extending that field, to protect against possible user error. We recommend +being as strict as possible to avoid unnecessary entries in the ``dynamic`` +list. Static analysis tools, when detecting a field is both specified and in the ``project.dynamic`` array, SHOULD assume the field is incomplete, allowing for @@ -344,6 +342,10 @@ field. An example of such an update for the license field: * When also listed in ``dynamic``: Can be extended with ``AND`` +It should also be noted that specifying something in dynamic will require any +tool that requires the full metadata to invoke the backend even if it is +partially statically specified, so it should not be used unless necessary. + Rejected Ideas ============== From 4f6fe37df234a766d73d0a3612d0db7b0afa4500 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 24 Sep 2025 16:48:27 -0400 Subject: [PATCH 12/14] refactor: remove special case Signed-off-by: Henry Schreiner --- peps/pep-0808.rst | 65 ++++++++++------------------------------------- 1 file changed, 13 insertions(+), 52 deletions(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 69e33951a86..718a0dd9cd8 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -16,8 +16,8 @@ Abstract This PEP relaxes the constraint on dynamic metadata listed in the ``[project]`` section in ``pyproject.toml`` to allow the static portion of mixed metadata to -be defined in the normal location if the field is a table, array or ``license`` -by having the dynamic fields extend the static ones. +be defined in the normal location if the field is a table or array by having +the dynamic fields extend the static ones. This allows users to opt into allowing a backend to extend metadata while still keeping the static portions of the metadata defined in the standard location in @@ -92,9 +92,7 @@ Every list and every table with arbitrary keys will now be allowed to be specified both statically, in the ``[project]`` table, and in the ``project.dynamic`` list. If it is present in both places, the build backend can extend list items and add new keys, but not modify existing list items or -strings. We are also including one special case; the ``license`` field when -specified as a string is an SPDX expression, which has a similar concept of -extension with ``AND``. +strings. Use Cases @@ -122,8 +120,6 @@ cases that have come up: and inject them (Poetry_ being the best known example). - Adding license files to the wheel based on what libraries are linked in (this is an active discussion in followup to :pep:`639`). -- Similarly, adding licenses based on vendored code (the setuptools_ repo might be - able to use this.) - Adding SBom's when building - :pep:`770` had to remove the ``pyproject.toml`` field specifically because you _want_ the build tool to add these, so the ``[project]`` table setting would be useless, you'd almost never be able to @@ -131,8 +127,6 @@ cases that have come up: All of these use cases have a similar feature: they are adding something dynamically to a fixed list (possibly a narrower pin for the dependency case). -With the exception of the recently added ``license`` field, these are all lists -or tables that need extending. You can implement these today, but it requires providing a completely separate configuration location for the non-extended portion, and static analysis tools @@ -195,40 +189,10 @@ longer needs to create and document a new location for the standard example). -Example: licenses ------------------ - -As another example, a package could specify its license(s): - -.. code-block:: toml - - [project] - license = "MIT" - license-files = ["LICENSE"] - - dynamic = ["license", "license-files"] - - [tool.my-build-backend] - license-search-sdist = ["vendor/*"] - linked-wheel-licenses = true - - -Now a build-backend can search for licenses in the vendor folder, and add them -automatically to the license expression and/or file list. If it supports this -when creating the wheel, it can specify the necessary fields in the ``Dynamic`` -``METADATA`` (2.2+) field and then add the licenses that are linked in when -creating the wheel. While you can do this today, you can't specify the -project's own license statically when you do this; this PEP allows both. - Future Updates -------------- -New fields added to the ``[project]`` table in future PEPs should include how -to handle additive dynamic metadata if they need special handling. Currently, -there is only one special case (``license``), so this is not expected to be -common. - Loosening this rule to allow purely additive metadata should address many of the use cases that have been seen in practice. If further changes are needed, this can be revisited in a future PEP; this PEP neither recommends or precludes @@ -250,8 +214,6 @@ now be allowed to be present in both the ``[project]`` table and the backend is allowed to insert entries into the list or table, but not remove entries, reorder entries, or modify the entries. Tables of arrays allow adding a new table entry or extending an existing array according to the rules above. -As a special case, the ``license`` field, when set to a string SPDX expression, -can be extended logically, as well. The fields that are arrays or tables with arbitrary entries are: @@ -266,8 +228,6 @@ The fields that are arrays or tables with arbitrary entries are: be changed or removed. * ``keywords``: Keywords can be added to the list. * ``license-files``: Files can be added to the list. -* ``license`` (string, special case): The license expression can be extended. - An existing license cannot be logically excluded. * ``optional-dependencies``: A new extra or new items can be added to a existing extra. * ``urls``: New urls can be added. Existing ones cannot be changed or removed. @@ -338,9 +298,7 @@ introductory packaging. The ``pyproject.toml`` `specification `__ will be updated to include the behavior of fields when specified and also listed in the dynamic -field. An example of such an update for the license field: - -* When also listed in ``dynamic``: Can be extended with ``AND`` +field. It should also be noted that specifying something in dynamic will require any tool that requires the full metadata to invoke the backend even if it is @@ -359,13 +317,16 @@ the use cases seen, though, and an explicit, opt-in approach is better for static tooling. -Include more string fields --------------------------- +Include string fields +--------------------- + +Some string fields could also be extended. Most notably, the ``license`` field +would benefit from being extendable, and due to the semantics of SPDX +expressions, extension could be defined through ``AND``. This was not added to +this PEP to keep it purely focused on arrays and tables. -There are two more string fields, namely ``version`` and ``requires-python`` -(``name`` is not allowed to be specified dynamically). There did not seem to be -a pressing practical need to allow these to be extended as with ``license``, -and logical extensions are not as clear, so they are not included in this PEP. +The other string fields, namely ``version`` and ``requires-python`` (``name`` +is not allowed to be specified dynamically), have less reason to be extended. Fixed key tables, like the deprecated ``license.text``/``license.file`` or ``readme.text``/``readme.file`` also have no clear benefit being partially dynamic. From f19c86f4408ea037542f790a8477fde10a8c6c3f Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 24 Sep 2025 17:08:01 -0400 Subject: [PATCH 13/14] Update peps/pep-0808.rst Co-authored-by: Paul Moore --- peps/pep-0808.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0808.rst b/peps/pep-0808.rst index 718a0dd9cd8..cddf3a0d2c6 100644 --- a/peps/pep-0808.rst +++ b/peps/pep-0808.rst @@ -323,7 +323,7 @@ Include string fields Some string fields could also be extended. Most notably, the ``license`` field would benefit from being extendable, and due to the semantics of SPDX expressions, extension could be defined through ``AND``. This was not added to -this PEP to keep it purely focused on arrays and tables. +this PEP because that would require individual fields to have custom semantics. The other string fields, namely ``version`` and ``requires-python`` (``name`` is not allowed to be specified dynamically), have less reason to be extended. From 95f738600fbd4716b306840408399c1d9451ea23 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 3 Oct 2025 22:24:30 -0400 Subject: [PATCH 14/14] Update CODEOWNERS Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 8b3689269c9..19f5d1221a5 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -680,6 +680,7 @@ peps/pep-0801.rst @warsaw peps/pep-0802.rst @AA-Turner peps/pep-0803.rst @encukou peps/pep-0804.rst @pradyunsg +# ... peps/pep-0808.rst @FFY00 # ... peps/pep-2026.rst @hugovk