Skip to content

Conversation

jenstroeger
Copy link
Owner

@jenstroeger jenstroeger commented Nov 18, 2024

The interesting thing about TAP (Test Anything Protocol) is that test output from different sources can be consolidated and analyzed as one. This change adds a pytest plugin which generated TAP output for tests, thus enabling TAP compatible test output.

However, I think before we would consider this change we need to track PR python-tap/pytest-tap#91 (and maybe PR python-tap/pytest-tap#92).

@jenstroeger
Copy link
Owner Author

With pytest-tap v3.5 released we can now capture and dump the output of individual tests as well:

> make test
Run unit tests...........................................................Passed
- hook id: pytest
- duration: 0.84s

TAP version 13
1..3
ok 1 src/package/something.py::[doctest] package.something.Something.do_something
# --- Captured Log ---
# 
# --- Captured Out ---
# 
# --- Captured Err ---
#
ok 2 tests/test_something.py::test_something
# --- Captured Log ---
# 
# --- Captured Out ---
# 
# --- Captured Err ---
#
ok 3 docs/source/index.rst::[doctest] index.rst
# --- Captured Log ---
# 
# --- Captured Out ---
# 
# --- Captured Err ---
#

However, some pytest plugins also print stats after the tests have been run:

---------- coverage: platform darwin, python 3.13.1-final-0 ----------
Name                       Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------
src/package/__init__.py        1      0      0      0   100%
src/package/something.py       4      0      0      0   100%
----------------------------------------------------------------------
TOTAL                          5      0      0      0   100%

Required test coverage of 100.0% reached. Total coverage: 100.00%
============================ Hypothesis Statistics =============================

tests/test_something.py::test_something:

  - during generate phase (0.00 seconds):
    - Typical runtimes: ~ 0-1 ms, of which < 1ms in data generation
    - 2 passing examples, 0 failing examples, 0 invalid examples

  - Stopped because nothing left to do


============================== slowest durations ===============================
0.01s call     tests/test_something.py::test_something
0.00s call     src/package/something.py::package.something.Something.do_something
0.00s setup    src/package/something.py::package.something.Something.do_something
0.00s call     docs/source/index.rst::index.rst
0.00s teardown src/package/something.py::package.something.Something.do_something
0.00s setup    docs/source/index.rst::index.rst
0.00s teardown docs/source/index.rst::index.rst
0.00s setup    tests/test_something.py::test_something
0.00s teardown tests/test_something.py::test_something
============================== 3 passed in 0.08s ===============================

and those are still missing. We’ll have to dig around some more and check if we can capture those as well, and output them…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant