-
Notifications
You must be signed in to change notification settings - Fork 28
Description
I have two problems here and not sure if I misunderstand the purpose of pytest-subtests or if there really is a bug.
I have a unittest like this
expect_folder = pathlib.Path.cwd() / 'Beverly'
self.assertTrue(expect_folder.exists())
expect = [
expect_folder / '_Elke.pickle',
expect_folder / '_Foo.pickle',
expect_folder / 'Bar.pickle',
expect_folder / 'Wurst.pickle',
]
for fp in expect:
with self.subTest(str(fp)):
self.assertTrue(fp.exists())
You see assertTrue() is called five times.
The pytest output
===================================================================================== short test summary info ==
BFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
SUBFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
SUBFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
SUBFAIL tests/test_bandas_datacontainer.py::FolderModeFS::test_build_container - AssertionError: False is not true
=================================================================================== 4 failed, 1 passed in 1.97s ==
Problem 1 : passed but not passed
The message 4 failed, 1 passed might technically be correct but it is wrong from the users perspective. There is one test defined by the def test_...() method. It doesn't matter how often self.assert...() is called in there. One fail means the whole test fails. Saying 1 passed is a bug.
Problem 2 : sub test message missing
I do use subtest (self.subTest(str(fp))) to better see which part of the test failed. But the output just tells me four times AssertionError: False is not true without more details about which subtest is involved. I would expect something like AssertionError: False is not true (/Beverly/Foo.pickle) as an output.