Using @mark.incremental and metafunc.parametrize in a pytest test class
<p>the purpose of @mark.incremental is that if one test fails, the tests afterwards are marked as expected to fail.</p>

<p>However, when I use this in conjuction with parametrization I get undesired behavior.</p>

<p>For example, in the case of this fake code:</p>

<pre><code>//conftest.py:

def pytest_generate_tests(metafunc):
    metafunc.parametrize("input", [True, False, None, False, True])

def pytest_runtest_makereport(item, call):
    if "incremental" in item.keywords:
        if call.excinfo is not None:
            parent = item.parent
            parent._previousfailed = item

def pytest_runtest_setup(item):
    if "incremental" in item.keywords:
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed (%s)" %previousfailed.name)

//test.py:
@pytest.mark.incremental
class TestClass:
    def test_input(self, input):
        assert input is not None
    def test_correct(self, input):
        assert input==True
</code></pre>

<p>I'd expect the test class to run </p>

<ul>
<li><p>test_input on True, </p></li>
<li><p>followed by test_correct on True, </p></li>
<li><p>followed by test_input on False, </p></li>
<li><p>followed by test_correct on False,</p></li>
<li><p>folowed by test_input on None, </p></li>
<li><p>followed by (xfailed) test_correct on None, etc etc.</p></li>
</ul>

<p>Instead, what happens is that the test class </p>

<ul>
<li>runs test_input on True, </li>
<li>then runs test_input on False, </li>
<li>then runs test_input on None, </li>
<li>then marks everything from that point onwards as xfailed (including the test_corrects).</li>
</ul>

<p>What I am assuming is happening is that parametrization takes priority over proceeding through functions in a class. The question is if it is possible to override this behaviour or work around it somehow, as the current situation makes marking a class as incremental completely useless to me.</p>

<p>(is the only way to handle this to copy-paste the code for the class over and over, each time with different parameters? The thought is repulsive to me)</p>