Integrate pytest-subtests #13738

nicoddemus · 2025-09-20T11:59:58Z

This PR copies the files from pytest-subtests and performs minimal integration.
I'm opening this to gauge whether everyone is on board with integrating this feature into the core.

Why?

Pros

subtests is a standard unittest feature, so it makes sense for pytest to support it as well.
Provides a simple alternative to parametrization.
Adds the ability to generate new test cases at runtime during the test execution, which is not possible with parametrization.
While it can exist as an external plugin, it requires many hacks, and better report integration is not easily achievable without core integration (off the top of my head: issues with terminal reporting, last failed and stepwise support, among others).

Cons

Adds another maintenance burden to the core.

TODO ✅

If everyone is on board, I will take the time this week to polish it and get it ready to merge ASAP:

✅ Cleanup the implementation: Currently it relies on monkey-patching, which should no longer be needed since we can modify the core code directly.
✅ Documentation: The feature should be documented as experimental -- meaning that while the feature itself is solid, we are still working out details such as how to best report subtest failures, integration with other plugins, etc.
✅ Fix lint and testing failures (obviously).

Can you explain the rationale for these options -- did someone ask for them in pytest-subtests?

I admit the difference between the options is a bit obscure, I wonder if they can't be folded into a single option?

I wonder if this shouldn't be an ini option rather than a cli option? I imagine someone would want to set this if it's too noisy, in which case it more of a project setting. But maybe it's more of a user-preference thing, then it's a CLI flag...

Maybe it should be configurable at the test level, using the fixture itself? (Probably not)

Should we document the options in the tutorial?

These were external contributions: pytest-dev/pytest-subtests#81 and pytest-dev/pytest-subtests#198.

I'm OK with changing them somehow into a single option, perhaps an option to disable all subtest progress unless it is failure?

I don't have experience with subtests to say myself. Generally less flags is better :)

pytest-dev/pytest-subtests#81
This allows the native pytest column calculations to not be disrupted and minimizes unneeded output for large CI systems.

Is the "disruption" still a thing?

Otherwise seems to want to minimize output. Though for shortletter it's a bit dubious I must say (it is quite "short"). I suspect what they mean is "we semantically don't view subtests as something that should be reported".

pytest-dev/pytest-subtests#198
I find it extremely annoying that each subtest success is printed to stdout

Wants to minimize output.

an option to disable all subtest progress unless it is failure?

IIUC this is mostly what they want.

Perhaps we should leverage the new "verbosity_" options:

https://docs.pytest.org/en/stable/reference/reference.html#confval-verbosity_assertions

We could include a verbosity_subtests which defaults to only show subtests failure, with additional verbosity being used to also show passed subtests.

A few thoughts on this:

Since the main motivation appears to be adding a feature unittest already provides, IMO it makes sense to follow its behavior: no extra output if all subtests pass.

CLI tools usually disable verbose output by default unless requested. Examples:

pytest -v ...

python3 -m unittest -v ...

PYTHONWARNINGS=always python3 foo.py

cp -v foo bar

The PR I made in Add --no-subtest-reports CLI opt pytest-subtests#199 goes in the opposite direction: it adds --no-subtest-reports to suppress verbosity. I think that should not be replicated here.

Subtests are mainly for context on failure. If all pass, knowing the exact execution is usually irrelevant. If you do care, it probably means it should be a separate test, not a subtest.

Just my 2 cents, and thanks to @nicoddemus for this very welcome feature.

src/_pytest/subtests.py

nicoddemus · 2025-10-18T14:02:12Z

Now the top-level test will fail with a message in case it contains failed subtests but otherwise does not contain failed assertions on its own:

__________________________________ test_foo ___________________________________
Contains 2 failed subtests

In addition, enable the plugin in `pytest/__init__.py` and `config/__init__.py`.

bluetech · 2025-10-19T09:25:31Z

testing/test_subtests.py

+                    r"test_skip_with_failure.py::T::test_foo \[custom message\] \(i=0\) SUBSKIPPED"
+                    r" \(skip subtest i=0\) .*",


Nit: I think it would be better to skip lint/formatter complaining here than to break the line, as it's less readable this way.

bluetech · 2025-10-19T09:26:21Z

doc/en/how-to/subtests.rst

+
+* Each subtest is reported with the ``,`` character.
+* Subtests are reported first and the "top-level" test is reported at the end on its own.
+* Subtest failures are reported as ``SUBFAIL``.


Suggested change

* Subtest failures are reported as ``SUBFAIL``.

* Subtest failures are reported as ``SUBFAILED``.

bluetech · 2025-10-19T09:36:19Z

src/_pytest/subtests.py

+    from typing_extensions import Self
+
+
+def pytest_addoption(parser: Parser) -> None:


I don't have experience with subtests to say myself. Generally less flags is better :)

pytest-dev/pytest-subtests#81
This allows the native pytest column calculations to not be disrupted and minimizes unneeded output for large CI systems.

Is the "disruption" still a thing?

Otherwise seems to want to minimize output. Though for shortletter it's a bit dubious I must say (it is quite "short"). I suspect what they mean is "we semantically don't view subtests as something that should be reported".

pytest-dev/pytest-subtests#198
I find it extremely annoying that each subtest success is printed to stdout

Wants to minimize output.

an option to disable all subtest progress unless it is failure?

IIUC this is mostly what they want.

bluetech · 2025-10-19T09:38:53Z

src/_pytest/subtests.py

+
+            def test(subtests):
+                for i in range(5):
+                    with subtests.test(msg="custom message", i=i):


Per comment above

Suggested change

with subtests.test(msg="custom message", i=i):

with subtests.test("custom message", i=i):

bluetech · 2025-10-19T09:47:36Z

src/_pytest/subtests.py

+    # capsys or capfd are active, subtest should not capture.
+    capman = request.config.pluginmanager.getplugin("capturemanager")
+    capture_fixture_active = getattr(capman, "_capture_fixture", None)
+
+    if option == "sys" and not capture_fixture_active:
+        fixture = CaptureFixture(SysCapture, request, _ispytest=True)
+    elif option == "fd" and not capture_fixture_active:
+        fixture = CaptureFixture(FDCapture, request, _ispytest=True)
+    else:
+        fixture = None


To me the code is clearer written like this:

Suggested change

# capsys or capfd are active, subtest should not capture.

capman = request.config.pluginmanager.getplugin("capturemanager")

capture_fixture_active = getattr(capman, "_capture_fixture", None)

if option == "sys" and not capture_fixture_active:

fixture = CaptureFixture(SysCapture, request, _ispytest=True)

elif option == "fd" and not capture_fixture_active:

fixture = CaptureFixture(FDCapture, request, _ispytest=True)

else:

fixture = None

capman = request.config.pluginmanager.getplugin("capturemanager")

if getattr(capman, "_capture_fixture", None):

# capsys or capfd are active, subtest should not capture.

fixture = None

elif option == "sys":

fixture = CaptureFixture(SysCapture, request, _ispytest=True)

elif option == "fd":

fixture = CaptureFixture(FDCapture, request, _ispytest=True)

else:

fixture = None

bluetech · 2025-10-19T09:49:18Z

src/_pytest/subtests.py

+
+@dataclasses.dataclass()
+class SubtestsReporterPlugin:
+    NAME: ClassVar[str] = "subtests-reporter"


Suggested change

NAME: ClassVar[str] = "subtests-reporter"

NAME: ClassVar = "subtests-reporter"

bluetech · 2025-10-19T09:51:13Z

src/_pytest/subtests.py

+
+
+def pytest_configure(config: Config) -> None:
+    config.pluginmanager.register(SubtestsReporterPlugin(), SubtestsReporterPlugin.NAME)


Why is this a separate plugin? It seems like it can be on the top-level plugin, using config.stash for contains_failed_subtests.

using config.stash for contains_failed_subtests.

Would actually be better to use item.stash instead of a dict on config keyed by nodeid.

bluetech · 2025-10-19T09:53:09Z

src/_pytest/subtests.py

+            # Top-level test, fail it it contains failed subtests and it has passed.
+            if (
+                report.passed
+                and (count := self.contains_failed_subtests.get(report.nodeid, 0)) > 0


It's a defaultdict so the get isn't needed.

bluetech · 2025-10-19T09:56:20Z

changelog/1367.feature.rst

+**Support for subtests** has been added.
+
+:ref:`subtests <subtests>` are an alternative to parametrization, useful in situations where the parametrization values are not all known at collection time.
+


Perhaps should mention that it integrates/replaces pytest-subtests with some differences?

bluetech

A few more comments.

bluetech · 2025-10-20T19:01:20Z

src/_pytest/subtests.py

+        try:
+            if exc_val is not None:
+                exc_info = ExceptionInfo.from_exception(exc_val)
+            else:
+                exc_info = None
+        finally:
+            self._exit_stack.close()


I don't think ExceptionInfo.from_exception is something that can fail, so is there a reason for wrapping in try/finally?

bluetech · 2025-10-20T19:06:30Z

src/_pytest/subtests.py

+            item=self.request.node, call=call_info
+        )
+        sub_report = SubtestReport._from_test_report(
+            report, SubtestContext(msg=self.msg, kwargs=self.kwargs.copy())


Is the kwargs.copy() necessary? SubtestContext uses Mapping (so promises not to mutate) and this function "owns" self.kwargs and doesn't mutate it, so should be OK not to copy.

bluetech · 2025-10-20T19:26:56Z

src/_pytest/subtests.py

+        self._captured_output.update_report(sub_report)
+        self._captured_logs.update_report(sub_report)


Take it or leave it: I prefer to reduce mutation when possible, so instead of update_report I would pass these two objects to SubtestReport._from_test_report and do it directly there. Since removing update_report will mess up NullCapturedLogs, I'd replace it will None and an if check. Slightly less nice but pretty ordinary.

bluetech · 2025-10-20T19:34:18Z

src/_pytest/subtests.py

+
+
+def pytest_configure(config: Config) -> None:
+    config.pluginmanager.register(SubtestsReporterPlugin(), SubtestsReporterPlugin.NAME)


using config.stash for contains_failed_subtests.

Would actually be better to use item.stash instead of a dict on config keyed by nodeid.

bluetech · 2025-10-20T20:08:43Z

src/_pytest/subtests.py

+        self._captured_logs.update_report(sub_report)
+
+        with self.suspend_capture_ctx():
+            self.ihook.pytest_runtest_logreport(report=sub_report)


An obscure bug: the logging plugin "live logging" support has stateful calls like self.log_cli_handler.set_when("logreport"). With subtests, it ends up getting a pytest_runtest_logreport while the call of the parent test is still ongoing, it ends up messing the headers ("live log logreport" instead of "live log call"):

def test(subtests): logging.error('START') with subtests.test(): logging.error('FIRST') assert False logging.error('MIDDLE') with subtests.test(): logging.error('SECOND') assert False logging.error('END') assert False

z.py::test ------------------------------------ live log call ------------------------------------ ERROR root:z.py:6 START ERROR root:z.py:9 FIRST (<subtest>) SUBFAILED [100%] --------------------------------- live log logreport ---------------------------------- ERROR root:z.py:12 MIDDLE ERROR root:z.py:15 SECOND z.py::test (<subtest>) SUBFAILED [100%] --------------------------------- live log logreport ---------------------------------- ERROR root:z.py:18 END z.py::test FAILED [100%]

Dunno how much to bother with this...

bluetech · 2025-10-20T20:17:52Z

src/_pytest/subtests.py

+        if exc_val is not None:
+            if self.request.session.shouldfail:
+                return False
+        return True


There's a complication here, this misses the reraise handling that CallInfo.from_call has:

reraise: tuple[type[BaseException], ...] = (Exit,) if not item.config.getoption("usepdb", False): reraise += (KeyboardInterrupt,)

So subtests.test() ends up swallowing pytest.exit() and Ctrl-C. Can test it with:

import pytest def test(subtests): with subtests.test(): pytest.exit() # To test exit # import time; time.sleep(5) # To test Ctrl-C print('AFTER') def test2(): pass

There's also a question of pytest.skip() behavior. Currently it skips the subtest. Maybe that's what we want? Didn't think about it too much.

bluetech · 2025-10-20T20:22:44Z

src/_pytest/subtests.py

+        handler.setFormatter(logging_plugin.formatter)
+
+        captured_logs = CapturedLogs(handler)
+        with catching_logs(handler):


Maybe should respect --log-level?

Suggested change

with catching_logs(handler):

with catching_logs(handler, level=logging_plugin.log_level):

bluetech · 2025-10-20T20:48:05Z

In pytest-dev/pytest-subtests#44 I saw something I didn't think about, nested subtests, e.g.

import pytest

def test(subtests):
    with subtests.test("a"):
        with subtests.test("b"):
            1 / 0
        1 / 0

unittest doesn't seem to do anything special with them. Only thing I can think of is to automatically nest the subtest name, e.g. a::b or such. Not sure if it's desirable.

nicoddemus changed the title ~~[DRAFT] Integrate pytest subtests~~ [DRAFT] Integrate pytest-subtests Sep 20, 2025

nicoddemus commented Sep 20, 2025

View reviewed changes

src/_pytest/subtests.py Outdated Show resolved Hide resolved

nicoddemus force-pushed the integrate-pytest-subtests branch from b43ab38 to 97ee032 Compare September 22, 2025 23:07

nicoddemus mentioned this pull request Sep 23, 2025

minor: do not import pytest in unittest.py #13748

Merged

nicoddemus force-pushed the integrate-pytest-subtests branch from 97ee032 to 6b5831f Compare September 26, 2025 13:34

psf-chronographer bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Sep 26, 2025

nicoddemus force-pushed the integrate-pytest-subtests branch 2 times, most recently from 5f56d81 to c93c0e0 Compare September 26, 2025 14:00

nicoddemus marked this pull request as ready for review September 26, 2025 14:00

nicoddemus changed the title ~~[DRAFT] Integrate pytest-subtests~~ Integrate pytest-subtests Sep 26, 2025

nicoddemus force-pushed the integrate-pytest-subtests branch from c93c0e0 to b569c93 Compare September 29, 2025 22:17

bluetech reviewed Sep 30, 2025

View reviewed changes

nicoddemus force-pushed the integrate-pytest-subtests branch from b569c93 to 506c75b Compare October 11, 2025 21:23

nicoddemus requested a review from bluetech October 11, 2025 21:23

nicoddemus force-pushed the integrate-pytest-subtests branch 2 times, most recently from 42f910d to ec3144f Compare October 11, 2025 21:32

RonnyPfannschmidt added this to the 9.0 milestone Oct 12, 2025

bluetech mentioned this pull request Oct 14, 2025

Set __module__ = "pytest" for all public class types? #13812

Open

bluetech reviewed Oct 14, 2025

View reviewed changes

Add pytest-subtests files changes

2172c09

In addition, enable the plugin in `pytest/__init__.py` and `config/__init__.py`.

nicoddemus added 8 commits October 18, 2025 11:02

subtests: remove direct pytest import

f2f066d

Force using xdist plugin and fix linting

2f5cf34

Replace attr by dataclass

159217d

Add docs

016c7c3

Cleanup internal hacks

7b09c6c

Code review

cf908c1

Code review

66313f2

Docs

a63b021

nicoddemus force-pushed the integrate-pytest-subtests branch from d8c7655 to 1c31ff1 Compare October 18, 2025 14:02

Make top-level tests fail when there are failing subtests

ee3d5db

nicoddemus force-pushed the integrate-pytest-subtests branch from 1c31ff1 to ee3d5db Compare October 18, 2025 14:13

nicoddemus requested review from RonnyPfannschmidt and bluetech October 18, 2025 14:22

bluetech reviewed Oct 19, 2025

View reviewed changes

nicoddemus mentioned this pull request Oct 20, 2025

Silence SUBPASS pytest-dev/pytest-subtests#198

Closed

bluetech reviewed Oct 20, 2025

View reviewed changes

		from typing_extensions import Self


		def pytest_addoption(parser: Parser) -> None:

		r"test_skip_with_failure.py::T::test_foo \[custom message\] \(i=0\) SUBSKIPPED"
		r" \(skip subtest i=0\) .*",

	* Subtest failures are reported as ``SUBFAIL``.
	* Subtest failures are reported as ``SUBFAILED``.

	with subtests.test(msg="custom message", i=i):
	with subtests.test("custom message", i=i):

	NAME: ClassVar[str] = "subtests-reporter"
	NAME: ClassVar = "subtests-reporter"



		def pytest_configure(config: Config) -> None:
		config.pluginmanager.register(SubtestsReporterPlugin(), SubtestsReporterPlugin.NAME)

		Support for subtests has been added.

		:ref:`subtests <subtests>` are an alternative to parametrization, useful in situations where the parametrization values are not all known at collection time.

		self._captured_output.update_report(sub_report)
		self._captured_logs.update_report(sub_report)

	with catching_logs(handler):
	with catching_logs(handler, level=logging_plugin.log_level):

Uh oh!

Integrate pytest-subtests #13738

Are you sure you want to change the base?

Integrate pytest-subtests #13738

Uh oh!

Conversation

nicoddemus commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

Pros

Cons

TODO ✅

Related

Uh oh!

Uh oh!

Pierre-Sassoulas commented Sep 20, 2025

Uh oh!

RonnyPfannschmidt commented Sep 20, 2025

Uh oh!

RonnyPfannschmidt commented Sep 20, 2025

Uh oh!

webknjaz commented Sep 20, 2025

Uh oh!

nicoddemus commented Sep 26, 2025

Uh oh!

bluetech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RonnyPfannschmidt commented Oct 12, 2025

Uh oh!

nicoddemus commented Oct 12, 2025

Uh oh!

RonnyPfannschmidt commented Oct 12, 2025

Uh oh!

nicoddemus commented Oct 12, 2025

Uh oh!

bluetech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giampaolo Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicoddemus commented Oct 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicoddemus commented Sep 20, 2025 •

edited

Loading

giampaolo Oct 20, 2025 •

edited

Loading

bluetech commented Oct 20, 2025 •

edited

Loading