From 4e066a20567ab09d6252116d367a449f86425d76 Mon Sep 17 00:00:00 2001 From: Klaus Zimmermann Date: Tue, 23 Sep 2025 18:02:43 +0200 Subject: [PATCH 1/2] PEP 780: Update to use sys.abi_info --- peps/pep-0780.rst | 123 ++++++++++++++++++++++++---------------------- 1 file changed, 64 insertions(+), 59 deletions(-) diff --git a/peps/pep-0780.rst b/peps/pep-0780.rst index b488e3872cc..54158f3b9ed 100644 --- a/peps/pep-0780.rst +++ b/peps/pep-0780.rst @@ -22,15 +22,16 @@ Post-History: Abstract ======== -This PEP defines using ABI features as environment markers for project -dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508` -(later moved to :ref:`packaging:dependency-specifiers`) introduced environment -markers to specify dependencies based on rules that describe when the -dependency should be used. This PEP extends the environment markers to allow -specifying dependencies based on specific ABI features of the Python -interpreter. For this, it defines a set of `ABI Features`_ and specifies how -they are made available for `environment markers `_ -as a new marker variable, ``sys_abi_features``. +This PEP defines the use of information about the interpreter ABI as +environment markers for project dependencies through a new ``sys_abi_info`` +environment marker variable. :pep:`508` (later moved to +:ref:`packaging:dependency-specifiers`) introduced environment markers to +specify dependencies based on rules that describe when the dependency should +be used. This PEP extends environment markers to allow specifying dependencies +based on specific information about the ABI of the Python interpreter. For +this, it defines a set of `ABI Features`_ and specifies how they are made +available for `environment markers `_ via a new marker +variable, ``sys_abi_info``. Motivation ========== @@ -57,23 +58,19 @@ marker for free-threading`_ Discourse thread: wheels. Picking up the wrong Cython version is causing a lot of obscure build failures and runtime crashes. It would be beneficial if the metadata could express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_). -- CFFI has no support for free-threading yet, and `Armin Rigo, one of the - maintainers, has stated - `__ - that it may be a good idea to fork ``cffi``, implement support for - free-threading, and only come back to the CFFI project with a single large PR - that adds support after the functionality "is reasonably well-tested (either - as tests or, better in this case, tested by being in use in various other - projects)". There are a lot of projects that depend on ``cffi``. They are - likely fine to start depending on a fork for free-threading only, however +- CFFI has only recently gained support for free-threading, and for a + significant period of time, it looked likely that support would first be + added in a fork. There are a lot of projects that depend on ``cffi``, which + are likely fine to start depending on a fork for free-threading only, however depending on a fork for >=3.13 or for all Python versions seems like a much - larger ask, and more disruptive for distribution packagers. + larger ask, and more disruptive for distribution packagers. While the + situation has been resolved with CFFI 2.0.0, other projects may face similar + issues. -While these concrete examples may be addressed later this year by Cython and -CFFI making compatible releases, the same issue is going to repeat further up -the stack. The free-threading rollout is expected to take several years, and an -environment marker for free-threading will make that rollout significantly -easier. +While concrete examples may be addressed with compatible releases, the same +issue is going to repeat further up the stack. The free-threading rollout is +expected to take several years, and an environment marker for free-threading +will make that rollout significantly easier. Another important ABI feature that is not yet covered by environment markers is the bitness of the interpreter. In most cases, the ``sys_platform`` or @@ -135,7 +132,7 @@ Relation to Other PEPs This PEP extends environment markers with set semantics for ABI features. :pep:`751#additions-to-marker-expression-syntax` includes a similar extension for lock file specific environment markers; although the two have been -developed indepedently, they are compatible where they overlap in terms of the +developed independently, they are compatible where they overlap in terms of the new set semantics. Specification @@ -149,49 +146,57 @@ and "**OPTIONAL**"" in this document are to be interpreted as described in ABI Features ------------ -ABI features are intrinsic properties of the Python interpreter, expressed as -simple, understandable strings. However, not all features are equally -applicable to all Python interpreters or Python versions. For example, the -distinction between free-threaded and GIL-enabled interpreters is only relevant -for CPython 3.13 onwards, but the bitness of the interpreter is relevant for -all interpreters. +ABI features are intrinsic properties of the Python interpreter. +However, not all features are equally applicable to all Python interpreters or +Python versions. For example, the distinction between free-threaded and +GIL-enabled interpreters is only relevant for CPython 3.13 onwards, but the +bitness of the interpreter, i.e. the number of bits in a pointer, is relevant +for all interpreters. All interpreters MUST handle the following ABI features as stated. ABI features that are restricted to particular interpreters MUST NOT be provided by other -interpreters. The features are subdivided into groups and for each group there -MUST be exactly one feature present, except when the group is marked as -optional, in which case there MUST be at most one feature present. +interpreters. -``free-threading`` or ``gil-enabled`` (only CPython) - If the Python interpreter is free-threaded, the ``free-threading`` feature - MUST be present and the ``gil-enabled`` feature MUST NOT be present. - Otherwise, the ``gil-enabled`` feature MUST be present and the - ``free-threading`` feature MUST NOT be present. +In Python 3.15 and later, the ABI features are determined by the new +``sys.abi_info`` attribute. In earlier Python versions, the ABI features are +the same as for Python 3.15 and given as follows: -``debug`` (only CPython, optional) +``pointer_bits`` + The bitness of the interpreter, that is, the number of bits in a pointer, + typically ``32`` or ``64`` [#bitness]_. + +``free_threaded`` (only CPython) + If the Python interpreter is free-threaded, ``free_threaded`` MUST be + ``True``, otherwise it MUST be ``False``. + +``debug`` (only CPython) This ABI feature is reserved for the ``--with-pydebug`` build of CPython. If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities, - the ``debug`` feature MUST be present. On POSIX systems, this corresponds + the ``debug`` feature MUST be ``True``. On POSIX systems, this corresponds to the Python expression ``"d" in sys.abiflags``. -``32-bit`` or ``64-bit`` (optional) - The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit - build [#bitness]_. If the bitness is unknown or neither 32-bit nor 64-bit, - this feature MUST NOT be present. +``byteorder`` + Indicates the byte order of the interpreter. The value MUST be either + ``"little"`` or ``"big"``. -.. _pep-780-sys_abi_features: +.. _pep-780-sys_abi_info: -The ``sys_abi_features`` Environment Marker +The ``sys_abi_info`` Environment Marker ------------------------------------------- To make ABI features available in dependency specifications, a new environment -marker variable, ``sys_abi_features``, is added to the format of dependency -specifiers. +marker variable, ``sys_abi_info``, is added to the format of dependency +specifiers. It is built to reflect the value of ``sys.abi_info``, which is a +new attribute of the ``sys`` module introduced in Python 3.15 as a simple +namespace. To represent its fields in the environment marker variable, they are +expressed as a frozen set of strings of the form ``f"{name}::{value}"``, where +``name`` is the name of the field and ``value`` is the string representation of +its value. To do this, we need to extend the grammar laid out in :pep:`508` and maintained in the :ref:`packaging:dependency-specifiers` and document the possible values. -The grammar is extended to include the ``sys_abi_features`` marker variable by +The grammar is extended to include the ``sys_abi_info`` marker variable by augmenting the definition of ``env_var`` as follows:: env_var = ('python_version' | 'python_full_version' | @@ -199,7 +204,7 @@ augmenting the definition of ``env_var`` as follows:: 'platform_system' | 'platform_version' | 'platform_machine' | 'platform_python_implementation' | 'implementation_name' | 'implementation_version' | - 'sys_abi_features' | + 'sys_abi_info' | 'extra' # ONLY when defined by a containing layer ) @@ -212,10 +217,10 @@ Like the grammar, also the overview table of environment markers in * - Marker - Python equivalent - Sample values - * - ``sys_abi_features`` - - no direct equivalent available - - ``{'free-threading', '64-bit'}``, - ``{'gil-enabled', 'debug', '32-bit'}`` + * - ``sys_abi_info`` + - ``sys.abi_info`` + - ``{'free_threaded::True', 'pointer_bits::64'}``, + ``{'free_threaded::False', 'debug::True', 'pointer_bits::32'}`` With these additions, ABI features can be used in dependency specifications via the ``in`` operator to test for the presence of a feature, or the ``not in`` @@ -229,22 +234,22 @@ Require Cython Pre-release for Free-Threaded Python To require a pre-release of Cython only for a free-threaded Python interpreter, the following dependency specification can be used:: - cython >3.1.0a1; "free-threading" in sys_abi_features - cython ==3.0.*; "free-threading" not in sys_abi_features + cython >3.1.0a1; "free_threaded::True" in sys_abi_info + cython ==3.0.*; "free_threaded::False" in sys_abi_info Require SciPy Unless on 32-bit ``win32`` ---------------------------------------- To require SciPy unless on a 32-bit interpreter on Windows, the following dependency specification can be used:: - scipy; platform_system != "Windows" or "32-bit" not in sys_abi_features + scipy; platform_system != "Windows" or "pointer_bits::32" not in sys_abi_info Require NumPy for a Free-Threaded Interpreter With Debugging Capabilities -------------------------------------------------------------------------- To require NumPy only for a free-threaded interpreter with debugging capabilities, the following dependency can be used:: - numpy; "free-threading" in sys_abi_features and "debug" in sys_abi_features + numpy; "free-threaded::True" in sys_abi_info and "debug::True" in sys_abi_info Backwards Compatibility ======================= From 174ff299937a8231afe1fe3cf62ae90269b274a9 Mon Sep 17 00:00:00 2001 From: Klaus Zimmermann Date: Fri, 26 Sep 2025 15:13:37 +0200 Subject: [PATCH 2/2] Address review --- peps/pep-0780.rst | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/peps/pep-0780.rst b/peps/pep-0780.rst index 54158f3b9ed..bd0b8441557 100644 --- a/peps/pep-0780.rst +++ b/peps/pep-0780.rst @@ -8,7 +8,7 @@ Status: Draft Type: Standards Track Topic: Packaging Created: 21-Mar-2025 -Python-Version: 3.14 +Python-Version: 3.15 Post-History: `05-Aug-2024 `__, `26-Mar-2025 `__, @@ -158,8 +158,8 @@ that are restricted to particular interpreters MUST NOT be provided by other interpreters. In Python 3.15 and later, the ABI features are determined by the new -``sys.abi_info`` attribute. In earlier Python versions, the ABI features are -the same as for Python 3.15 and given as follows: +:external+py3.15:py:data:`sys.abi_info` attribute. In earlier Python versions, +the ABI features are the same as for Python 3.15 and given as follows: ``pointer_bits`` The bitness of the interpreter, that is, the number of bits in a pointer, @@ -170,10 +170,8 @@ the same as for Python 3.15 and given as follows: ``True``, otherwise it MUST be ``False``. ``debug`` (only CPython) - This ABI feature is reserved for the ``--with-pydebug`` build of CPython. - If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities, - the ``debug`` feature MUST be ``True``. On POSIX systems, this corresponds - to the Python expression ``"d" in sys.abiflags``. + This ABI feature corresponds to the + :external+py3.15:py:attr:`sys.abi_info.debug` field. ``byteorder`` Indicates the byte order of the interpreter. The value MUST be either @@ -182,7 +180,7 @@ the same as for Python 3.15 and given as follows: .. _pep-780-sys_abi_info: The ``sys_abi_info`` Environment Marker -------------------------------------------- +--------------------------------------- To make ABI features available in dependency specifications, a new environment marker variable, ``sys_abi_info``, is added to the format of dependency