Documentation/classification of Stable ABI type flags

[da-woods]

Type flags are ultimately macros, and macros tend to be badly documented about whether they're in the Limited API/Stable ABI. However, some of them are documented as part of the stable ABI (Py_TPFLAGS_BASETYPE, Py_TPFLAGS_HAVE_GC, Py_TPFLAGS_DEFAULT, Py_TPFLAGS_METHOD_DESCRIPTOR, Py_TPFLAGS_ITEMS_AT_END, Py_TPFLAGS_HAVE_VECTORCALL). But not all.

Looking at the implementation is also inconsistent.

• There's a group that are specifically excluded from the Limited API https://github.com/python/cpython/blob/d6f77e6a3fa46fa31694d288a3081d5ee76e9d07/Include/object.h#L535-L364)
• There's one that's conditionally excluded based on version (https://github.com/python/cpython/blob/d6f77e6a3fa46fa31694d288a3081d5ee76e9d07/Include/object.h#L580-L586)
• However the rest are unconditionally excluded (including some which are only added in recent memory).

My interest is mostly in documenting them a bit better - the fact that some as listed as part of the Stable ABI and some aren't strongly implies that the ones that aren't are unavailable. I created a PR which added most of the missing ones (python/cpython#143345) but Petr suggested I come here.

Petr suggests:

• the *_SUBCLASS should not be included; you should use PyUnicode_Check and similar (which should be added to Limited API/Stable ABI if they're not there already).

That seems reasonable to me

• READY & READYING shouldn't be included either -- AFAIK, this detail of the class creation machinery isn't that important for heap types, as there's no hooks to customize behaviour between allocation and the PyType_Ready call. We shouldn't need to expose the bits.

That seems reasonable to me.

• For DISALLOW_INSTANTIATION & IMMUTABLETYPE, we're probably stuck exposing them as flags. But, code that checks these, for something else than optimizations, is probably doing something wrong.

Again, seems reasonable - they're mostly useful in class creation and it's pretty unusual that you need to query them. So arguably they could be in the Stable ABI for that only.

The other one that's useful is HEAPTYPE - in this case it's useful to be able to query it because PyType_GetSlot changes behaviour in older versions if it's a heap-type. It isn't useful to be able to set it.

---

There's also the possibility of changing the Py_LIMITED_API guards if things shouldn't be exposed - _Py_TPFLAGS_MATCH_SELF is an example which is exposed even though it has an underscore and the other pattern matching flags are deliberately hidden.

---

Essentially the "some documented, some not" situation is confusing and I think it'd be good to improve that. At least for some useful subset.

[encukou]

Practically, this issue is about bringing the Limited API definition (stable_abi.toml) and the header files (Py_LIMITED_API) in sync. I didn't get around to doing that for macros when implementing PEP 652 (which matches my "suggestions", except for the newer DISALLOW_INSTANTIATION and IMMUTABLETYPE).

I suggest adding deprecation warnings for *_SUBCLASS, READY & READYING the if Py_LIMITED_API is defined. As I see it, the fact that they're available with Py_LIMITED_API is a bug, but there's no immediate reason to force people to change working code; hiding these implementation details is a clean-up for the long term.

---

There's also the possibility of changing the Py_LIMITED_API guards if things shouldn't be exposed - _Py_TPFLAGS_MATCH_SELF is an example which is exposed even though it has an underscore and the other pattern matching flags are deliberately hidden.

That shouldn't need a discussion -- it's fair game to hide those, as long as e.g. macros or static inline functions don't use them.

[encukou]

The other one that's useful is HEAPTYPE - in this case it's useful to be able to query it because PyType_GetSlot changes behaviour in older versions if it's a heap-type. It isn't useful to be able to set it.

The change was in 3.10.
I'd go with deprecation as for the other flags, but, we could also just hide the flag for Py_LIMITED_API >= 3.15.