Skip to content

Retry returns failing results, given there is one test result reporting about passing the test #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Aemilivs opened this issue Jan 28, 2020 · 2 comments

Comments

@Aemilivs
Copy link

Description

Given that I run test with retry attribute with the value of N and some of those times test fails, mixed results are returned to MSTest framework, and as the result test is marked as failed.

Steps to reproduce

Create test with retry attribute, with the value of 5
Fail test 4 times out of 5

Expected behavior

Test run is marked as successful

Actual behavior

Test run is not marked as successful

@Aemilivs
Copy link
Author

From my point of view, the only question that is raised is what about test results with failed runs? I don't see any way keeping them and marking tests as passed at the same time.

@AJua
Copy link

AJua commented Mar 10, 2020

I want this behavior too!

Here I have a proposal - add an optional parameter on RetryAttribute. It will override failed test outcome if the maximum retry attempts is not reached. The default behavior is keep its failed test outcome.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using MSTest.TestFramework.Extensions.AttributeEx;
using MSTest.TestFramework.Extensions.TestMethodEx;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethodEx]
        [Retry(5)]
        [Retry(5, UnitTestOutcome.Inconclusive)]
        public void TestMethod1()
        {
            // Test logic that causes the test to fail.
        }
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants