Test | Add flaky test quarantine zone #3856

mdaigle · 2025-12-19T20:54:09Z

At the moment, we have a number of flaky tests in our pipelines (a couple of which I'll take credit for) that cause random failures, necessitate pipeline reruns, and cheapen the value of our regression testing suite. When tests fail regularly, we pay less attention when we see a failing test. A common pattern to deal with flaky tests is to establish a "quarantine zone" where these tests can live temporarily until they're improved. The quarantine zone is separate from the main build pipelines and does not block main builds, but runs regularly so that we can keep an eye on how flaky tests are performing.

To establish a quarantine zone, I'm planning to use the "flaky" category introduced in this PR: #3488. Tests in this category will be run in a separate testing stage immediately after the corresponding Unit/Functional/Manual testing stage (e.g. Unit => UnitFlaky => Functional => FunctionalFlaky ...). The flaky test stage will not block the pipeline, and failures will be ignored using the continueOnError property: steps.task definition | Microsoft Learn

The ActiveIssue tag will remain reserved for tests that cannot pass due to a platform specific bug, pipeline issue, or other limitation that causes the test to always fail.

I started off by tagging our top offenders based on this dashboard. As we discover more flaky tests, the DRI (or anyone) can open a PR adding the flaky tag to the offending tests. Flaky tests can be addressed either via prioritization during sprint planning or when working on a related section of code.

…be evaluated?

Copilot

Pull request overview

This PR simplifies test filtering to support a quarantine zone for flaky tests by updating the --filter expressions in test execution targets. The changes remove platform-specific category exclusions and standardize filtering to only exclude tests marked as "failing" or "flaky" across all test types (Functional and Manual).

Simplifies test filters to exclude only "failing" and "flaky" categories
Removes platform-specific category filters that are no longer needed
Applies consistent filtering logic across Functional and Manual test targets for both Windows and Unix platforms

Copilot · 2025-12-19T20:57:08Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
+        --filter "category!=failing&amp;category!=flaky"


The filter encoding is inconsistent with the Unit test targets. Unit tests use percent-encoded ampersands (%26) while Functional and Manual tests use XML entity encoding (&). For consistency and to avoid potential filtering issues, the encoding should match across all test targets. Consider using the same encoding pattern (%26) as used in the Unit test filters at lines 235 and 256.

Copilot · 2025-12-19T20:57:08Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
+        --filter "category!=failing&amp;category!=flaky"


The filter encoding is inconsistent with the Unit test targets. Unit tests use percent-encoded ampersands (%26) while Functional and Manual tests use XML entity encoding (&). For consistency and to avoid potential filtering issues, the encoding should match across all test targets. Consider using the same encoding pattern (%26) as used in the Unit test filters at lines 235 and 256.

Copilot · 2025-12-19T20:57:09Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
+        --filter "category!=failing&amp;category!=flaky"


The filter encoding is inconsistent with the Unit test targets. Unit tests use percent-encoded ampersands (%26) while Functional and Manual tests use XML entity encoding (&). For consistency and to avoid potential filtering issues, the encoding should match across all test targets. Consider using the same encoding pattern (%26) as used in the Unit test filters at lines 235 and 256.

Copilot · 2025-12-19T20:57:09Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
+        --filter "category!=failing&amp;category!=flaky"


The filter encoding is inconsistent with the Unit test targets. Unit tests use percent-encoded ampersands (%26) while Functional and Manual tests use XML entity encoding (&). For consistency and to avoid potential filtering issues, the encoding should match across all test targets. Consider using the same encoding pattern (%26) as used in the Unit test filters at lines 235 and 256.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 12 comments.

Copilot · 2025-12-19T22:44:48Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+      platform: '${{parameters.platform }}'
+      configuration: '${{parameters.buildConfiguration }}'
+      ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:      
+        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'


The msbuildArguments value has mismatched quotes. The single quote after the closing parenthesis should come at the end of the line, not before the Filter parameter. This will cause a syntax error in the pipeline. The correct format should be: msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'

Copilot · 2025-12-19T22:44:48Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+      platform: '${{parameters.platform }}'
+      configuration: '${{parameters.buildConfiguration }}'
+      ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:
+        msbuildArguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'


The msbuildArguments value has mismatched quotes. The single quote after the closing parenthesis should come at the end of the line, not before the Filter parameter. This will cause a syntax error in the pipeline. The correct format should be: msbuildArguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'

Copilot · 2025-12-19T22:44:48Z

build.proj

+      <FilterStatement>"$(Filter)"</FilterStatement>    
+      <FilterStatement Condition="'$(FilterStatement)' == ''">"category!=failing&amp;category!=flaky"</FilterStatement>


The FilterStatement logic has a flaw. When Filter is empty or not provided, line 226 sets FilterStatement to "$(Filter)" which evaluates to "" (a string containing two double quotes). However, line 227's condition checks if FilterStatement equals an empty string '', which will never be true. This means the default filter will never be applied when Filter is empty. The condition should check for '""' or the initial assignment should be changed to not include quotes around an empty Filter value.

Copilot · 2025-12-19T22:44:49Z

build.proj

+      <FilterStatement>"$(Filter)"</FilterStatement>    
+      <FilterStatement Condition="'$(FilterStatement)' == ''">"category!=failing&amp;category!=flaky"</FilterStatement>


The FilterStatement logic has a flaw. When Filter is empty or not provided, line 249 sets FilterStatement to "$(Filter)" which evaluates to "" (a string containing two double quotes). However, line 250's condition checks if FilterStatement equals an empty string '', which will never be true. This means the default filter will never be applied when Filter is empty. The condition should check for '""' or the initial assignment should be changed to not include quotes around an empty Filter value.

Copilot · 2025-12-19T22:44:49Z

build.proj

+      <FilterStatement>"$(Filter)"</FilterStatement>    
+      <FilterStatement Condition="'$(FilterStatement)' == ''">"category!=failing&amp;category!=flaky"</FilterStatement>


The FilterStatement logic has a flaw. When Filter is empty or not provided, line 328 sets FilterStatement to "$(Filter)" which evaluates to "" (a string containing two double quotes). However, line 329's condition checks if FilterStatement equals an empty string '', which will never be true. This means the default filter will never be applied when Filter is empty. The condition should check for '""' or the initial assignment should be changed to not include quotes around an empty Filter value.

Copilot · 2025-12-19T22:44:50Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+        ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:   
+          msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+        ${{ else }}: # x86
+          msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'


The msbuildArguments value has mismatched quotes. The single quote after the closing parenthesis should come at the end of the line, not before the Filter parameter. This will cause a syntax error in the pipeline. The correct format should be: msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'

Copilot · 2025-12-19T22:44:50Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+      ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:      
+        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+      ${{ else }}: # x86
+        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'


The msbuildArguments value has mismatched quotes. The single quote after the closing parenthesis should come at the end of the line, not before the Filter parameter. This will cause a syntax error in the pipeline. The correct format should be: msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'

Copilot · 2025-12-19T22:44:50Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+      ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:
+        msbuildArguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+      ${{ else }}: # x86
+        msbuildArguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'


The msbuildArguments value has mismatched quotes. The single quote after the closing parenthesis should come at the end of the line, not before the Filter parameter. This will cause a syntax error in the pipeline. The correct format should be: msbuildArguments: '-t:RunManualTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'

Copilot · 2025-12-19T22:44:51Z

build.proj

+      <FilterStatement>"$(Filter)"</FilterStatement>    
+      <FilterStatement Condition="'$(FilterStatement)' == ''">"category!=failing&amp;category!=flaky"</FilterStatement>


The FilterStatement logic has a flaw. When Filter is empty or not provided, line 275 sets FilterStatement to "$(Filter)" which evaluates to "" (a string containing two double quotes). However, line 276's condition checks if FilterStatement equals an empty string '', which will never be true. This means the default filter will never be applied when Filter is empty. The condition should check for '""' or the initial assignment should be changed to not include quotes around an empty Filter value.

Copilot · 2025-12-19T22:44:51Z

build.proj

+      <FilterStatement>"$(Filter)"</FilterStatement>    
+      <FilterStatement Condition="'$(FilterStatement)' == ''">"category!=failing&amp;category!=flaky"</FilterStatement>


The FilterStatement logic has a flaw. When Filter is empty or not provided, line 300 sets FilterStatement to "$(Filter)" which evaluates to "" (a string containing two double quotes). However, line 301's condition checks if FilterStatement equals an empty string '', which will never be true. This means the default filter will never be applied when Filter is empty. The condition should check for '""' or the initial assignment should be changed to not include quotes around an empty Filter value.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

Copilot · 2025-12-19T23:23:24Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+    - task: MSBuild@1
+      displayName: 'Run Flaky Unit Tests ${{parameters.msbuildArchitecture }}'
+      inputs:
+        solution: build.proj
+        msbuildArchitecture: ${{parameters.msbuildArchitecture }}
+        platform: '${{parameters.platform }}'
+        configuration: '${{parameters.buildConfiguration }}'
+        ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:   
+          msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+        ${{ else }}: # x86
+          msbuildArguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'
+      continueOnError: true


The regular Unit Tests task is missing the retryCountOnTaskFailure property that was removed from the diff. While the flaky unit tests don't need retry logic (they use continueOnError), the regular unit tests should still have retry capability. The original configuration had retryCountOnTaskFailure: 1 for unit tests, which should be retained on the regular unit test task at line 64.

Copilot · 2025-12-19T23:23:25Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+  - task: MSBuild@1
+    displayName: 'Run Flaky Functional Tests ${{parameters.msbuildArchitecture }}'
+    inputs:
+      solution: build.proj
+      msbuildArchitecture: ${{parameters.msbuildArchitecture }}
+      platform: '${{parameters.platform }}'
+      configuration: '${{parameters.buildConfiguration }}'
+      ${{ if eq(parameters.msbuildArchitecture, 'x64') }}:      
+        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:Filter="category=flaky"'
+      ${{ else }}: # x86
+        msbuildArguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:DotnetPath=${{parameters.dotnetx86RootPath }} -p:Filter="category=flaky"'
+    continueOnError: true


The regular Functional Tests task is missing the retryCountOnTaskFailure property that was removed from the diff. The original configuration had retryCountOnTaskFailure: 1 for functional tests, which should be retained on the regular functional test task at line 88.

Copilot · 2025-12-19T23:23:25Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+    - task: DotNetCoreCLI@2
+      displayName: 'Run Flaky Unit Tests'
+      inputs:
+        command: custom
+        projects: build.proj
+        custom: msbuild
+        arguments: '-t:RunUnitTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:platform=${{parameters.platform }} -p:Configuration=${{parameters.buildConfiguration }} -p:Filter="category=flaky"'
+        verbosityRestore: Detailed
+        verbosityPack: Detailed
+      continueOnError: true


The regular Unit Tests task is missing the retryCountOnTaskFailure property that was removed from the diff. The original configuration had retryCountOnTaskFailure: 1 for unit tests, which should be retained on the regular unit test task at line 141.

Copilot · 2025-12-19T23:23:25Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

+  - task: DotNetCoreCLI@2
+    displayName: 'Run Flaky Functional Tests'
+    inputs:
+      command: custom
+      projects: build.proj
+      custom: msbuild
+      arguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:platform=${{parameters.platform }} -p:Configuration=${{parameters.buildConfiguration }} -p:Filter="category=flaky"'
+      verbosityRestore: Detailed
+      verbosityPack: Detailed
+    continueOnError: true


The regular Functional Tests task is missing the retryCountOnTaskFailure property that was removed from the diff. The original configuration had retryCountOnTaskFailure: 1 for functional tests, which should be retained on the regular functional test task at line 163.

Copilot · 2025-12-19T23:23:26Z

build.proj

  <!-- Run all Functional tests applicable to Unix. -->
  <Target Name="RunFunctionalTestsUnix" Condition="'$(IsEnabledWindows)' != 'true'">
-    <PropertyGroup>
+      <PropertyGroup>


Inconsistent indentation: The opening <PropertyGroup> tag has an extra space of indentation compared to other similar targets. It should be indented with 4 spaces to align with the same element in other targets like RunFunctionalTestsWindows at line 274.

Suggested change

<PropertyGroup>

<PropertyGroup>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 13 comments.

Copilot · 2025-12-20T00:19:16Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
+        --filter "$(FilterStatement)"


The filter has been changed from "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'nonnetcoreapptests', 'nonlinuxtests', or 'nonuaptests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests&category!=flaky"

Copilot · 2025-12-20T00:19:17Z

eng/pipelines/common/templates/steps/run-all-tests-step.yml

      custom: msbuild
      arguments: '-t:RunFunctionalTests -p:TF=${{parameters.targetFramework }} -p:TestSet=${{parameters.testSet }} -p:ReferenceType=${{parameters.referenceType }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.nugetPackageVersion }} -p:platform=${{parameters.platform }} -p:Configuration=${{parameters.buildConfiguration }}'
      verbosityRestore: Detailed
      verbosityPack: Detailed


The diff shows that retryCountOnTaskFailure: 1 and condition: and(succeeded(), ne(variables['Agent.OS'], 'Windows_NT')) were removed from the Functional Tests task. This means functional tests will run even if previous tasks failed and won't retry on transient failures. These changes appear unintentional and should be restored.

Suggested change

verbosityPack: Detailed

verbosityPack: Detailed

condition: and(succeeded(), ne(variables['Agent.OS'], 'Windows_NT'))

retryCountOnTaskFailure: 1

Copilot · 2025-12-20T00:19:17Z

eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml

    command: test
    projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
-    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
+    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'


The filter has been changed from excluding nonnetfxtests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetfxtests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Framework or Windows. The filter should maintain the original exclusions: "category!=nonnetfxtests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot · 2025-12-20T00:19:17Z

eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml

    command: test
    projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
-    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter  category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests  --collect "Code Coverage"'
+    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter  "category!=failing&category!=flaky" --collect "Code Coverage"'


The filter has been changed from excluding nonnetcoreapptests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetcoreapptests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Core or Windows. The filter should maintain the original exclusions: "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot · 2025-12-20T00:19:18Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
+        --filter "$(FilterStatement)"


The filter has been changed from "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'nonnetcoreapptests', 'nonlinuxtests', or 'nonuaptests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests&category!=flaky"

Copilot · 2025-12-20T00:19:18Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
+        --filter "$(FilterStatement)"


The filter has been changed from "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'non$(TargetGroup)tests' or 'nonwindowstests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot · 2025-12-20T00:19:18Z

eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml

    command: test
    projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
-    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
+    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'


The filter has been changed from excluding nonnetfxtests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetfxtests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Framework or Windows. The filter should maintain the original exclusions: "category!=nonnetfxtests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot · 2025-12-20T00:19:18Z

eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml

    command: test
    projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
-    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }}  -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests"'
+    arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }}  -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky"'


The filter has been changed from excluding nonnetcoreapptests, failing, and nonwindowstests to only excluding failing and flaky tests. This means tests tagged with 'nonnetcoreapptests' or 'nonwindowstests' will now run in this pipeline, which could cause failures if those tests aren't compatible with .NET Core or Windows. The filter should maintain the original exclusions: "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests&category!=flaky"

Copilot · 2025-12-20T00:19:19Z

build.proj

        --collect "Code coverage"
        --results-directory $(ResultsDirectory)
-        --filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
+        --filter "$(FilterStatement)"


The filter has been changed from "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests" to "category!=failing&category!=flaky". This removes important platform and framework-specific test exclusions. Tests tagged with 'non$(TargetGroup)tests' or 'nonwindowstests' will now run even though they aren't compatible with this configuration. The filter should be: "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests&category!=flaky"

edwardneal · 2025-12-20T12:04:07Z

Can we also include the diagnostic testing please? These seemed to be flaky on ARM64.

paulmedynski

Approved, assuming the Copilot comments will be addressed.

Skip flaky tests. Try removing platform filters which should already …

fa22414

…be evaluated?

Copilot AI review requested due to automatic review settings December 19, 2025 20:54

Copilot started reviewing on behalf of mdaigle December 19, 2025 20:54 View session

Copilot AI reviewed Dec 19, 2025

View reviewed changes

mdaigle added 3 commits December 19, 2025 14:07

Make filter configurable in build.proj

dba39e7

Add quarantined testing steps.

13b684c

Fix closing quote.

8a4f8e0

Copilot AI review requested due to automatic review settings December 19, 2025 22:40

Copilot started reviewing on behalf of mdaigle December 19, 2025 22:41 View session

Fix closing quote.

4d14a1e

Copilot AI reviewed Dec 19, 2025

View reviewed changes

mdaigle added 2 commits December 19, 2025 15:00

Fix quote wrapping for filter argument.

ae19c47

Fix continueOnError indent.

29ecebc

Copilot AI review requested due to automatic review settings December 19, 2025 23:20

Copilot started reviewing on behalf of mdaigle December 19, 2025 23:20 View session

Copilot AI reviewed Dec 19, 2025

View reviewed changes

mdaigle changed the title ~~Test | Add quarantine zone~~ Test | Add flaky test quarantine zone Dec 19, 2025

mdaigle added 2 commits December 19, 2025 16:06

Tag flaky tests.

3413a96

Tag xevent test as flaky.

4c4cf1e

Copilot AI review requested due to automatic review settings December 20, 2025 00:13

Copilot started reviewing on behalf of mdaigle December 20, 2025 00:14 View session

mdaigle marked this pull request as ready for review December 20, 2025 00:14

mdaigle requested a review from a team as a code owner December 20, 2025 00:14

Copilot AI reviewed Dec 20, 2025

View reviewed changes

paulmedynski self-assigned this Dec 22, 2025

paulmedynski approved these changes Dec 22, 2025

View reviewed changes

		<FilterStatement>"$(Filter)"</FilterStatement>
		<FilterStatement Condition="'$(FilterStatement)' == ''">"category!=failing&category!=flaky"</FilterStatement>

Test | Add flaky test quarantine zone #3856

Are you sure you want to change the base?

Test | Add flaky test quarantine zone #3856

Uh oh!

Conversation

mdaigle commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

mdaigle commented Dec 19, 2025 •

edited

Loading