Skip to content

Tracker: add "repro_snippets" to failing tests #391

@ev-br

Description

@ev-br

Hypothesis reported failures are sometimes hard to diagnose, even though our assertions go to incredible lengths to give useful diagnostics. It is often not easy to reproduce an exact set of parameters which trigger a test failure.
As suggested in #379, we want to emit on failure the exact inputs which triggered it. [1]

A typical test case looks like this:

@given(arg=hypothesis_strategy)
def test_something(arg):
    arg2 = further_hypothesis_magic

    out = xp.function_under_test(arg1, arg2)
    assertions_on_out

and a "low-tech" solution is to convert it into

@given(arg=hypothesis_strategy)
def test_something(arg):
    arg2 = further_hypothesis_magic

    repro_snippet = f"xp.function_under_test({arg1!r}, {arg2!r})"   # exactly the r.h.s. of `out = ` below
    try:
        # as before
        out = xp.function_under_test(arg1, arg2)
        assertions_on_out
    except Exception as exc:
       # attach a failing snippet and reraise
       exc.add_note(repro_snippet)
       raise

See #384 for a worked example.

[1] gh-379 gives two ideas: declare "dependencies" of each test, and emit a "reproducible snippet" on failure. This issue only tracks progress towards the latter.


The plan is transform all tests in this manner:

  • a test module per commit
  • list the commits in .git-blame-ignore-revs for git blame to skip the whitespace changes.

Here's the tracker:

These test modules are different enough to not need "repro snippets", as their use of hypothesis is minimal to not at all:

  • test_signatures.py : does not use hypothesis
  • test_inspection_functions.py: does not use hypothesis
  • test_has_names.py
  • test_constants.py

cc @cbourjau @mtsokol

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions