Support generator case functions with multiple 'yield'

[jenstroeger]

Following issue #229 and the discussion there, adding generators to case function requires discussion and some work:

• What about pytest marks ? marking a single subcase (1 yield) / marking all subcases (all of the yields) ?
• Should we consider the "return" of a generator ?
• How to manage pytest ids one by one ? All at once ?
• How to manage the compliance with the @case decorator ? (ids, filters, tags, etc.). Maybe the best way is to not create a new decorator but to have extra arguments in this decorator ?

Let the conversation begin 🤓

[smarie]

Thanks @jenstroeger ! I am curious to see the thoughts of the community of users here

[jenstroeger]

I think the feature itself would be useful and offer a lot of flexibility to improve automated, larger test sets — for example I’d create a generator which randomly picks a large number of example cases from a Faker provider.¹ How nice would that be 🤓

As for implementing that in your package and answering the above questions, I’m not fluent enough with the details of the pytest implementation… but hopefully the community can help out so this feature can come alive soon 🤞🏼

Footnotes

1. In fact, this feature request is somewhat motivated by the discussion here: HypothesisWorks/hypothesis#3614. ↩

[jenstroeger]

@smarie any more thoughts on this?

As for an example, I think using pytest-repeat combined with a case function that produces random data using Faker would be neat. Something like this:

# The case function returns a generator.
def case_names() -> Generator[str]:
    fake = Faker()
    while True:
        yield fake.name()

# We need to specify where to draw test cases from. The decorator order probably matters.
# Now the `cases` argument is a generator object from which values can be pulled.
@parametrize("name", cases=case_names())
@pytest.mark.repeat(10)
def test_something_faker(name: str) -> None:
    assert something_to_test(name)

So, basically, if the case function is a generator then just next() a value from it (I think even allow StopIteration to raise and fail testing) and pass that value to the test function. This way, I can use the repeat() decorator, or I can use the same case function multiple times and every time it produces a different value.

[jenstroeger]

@smarie any thoughts on this?

[smarie]

Hi @jenstroeger , sorry for the delay and thanks for reminding me about this !

Looking at #229 and #319 I renamed the two issues to make it clearer that the challenge proposed here is much more ambitious/disruptive.

The example that you suggest is different from the one I thought you initially had. I thought that you were suggesting that all yielded values from the case function would be appended to the parameters list. After thinking about this, I am not convinced that the complexity of the pattern is worth adressing:

• it breaks the symmetry between cases functions that require fixtures (that therefore need to be turned into fixtures), and case functions that do not (that are turned into lazy_value)
• it breaks the independence paradigm of tests : one failure happening when generating one parameter would trigger subsequent failures in all other tests requiring this case function
• there is the situation of infinite generators, that would trigger strange behaviours

Finally, I suspect that most if not all usage scenarios have a simple workaround using existing mechanisms. For example, your example above could be done like this:

from pytest_cases import parametrize_with_cases

fake = Faker()

def case_names():
    return fake.name()

@parametrize_with_cases("name", cases=case_names)
@pytest.mark.repeat(10)
def test_something_faker(name: str) -> None:
    assert something_to_test(name)

So I suggest to reject this feature proposal for now, and to possibly focus on solving #229, which is "just" about having setup/teardown capabilities in case functions. This is a low-hanging fruit since some case functions are already converted to fixtures automatically by the framework - we would just need to detect that a case function is a generator, and to force-turn it into a fixture then.

[jenstroeger]

Well, then let’s move forward with #299 and perhaps that’s sufficient for my needs. I’d like to play with the feature, though, so can you guesstimate when an update to this package will be available?