Skip to content

Migrate CI to RecordedTests by default #1390

@scbedd

Description

@scbedd

This issue covers the CI cutover/build definition updates that finalize the CI changes that are related to #1208 .

Where we're at

Here's the situation where we're at once #1208 is complete:

  • Any tests that can be recorded are. If a tool is changed, the recorded tests are run as a step in the public build.
  • There are exceptions due to non-http usage or not being factored properly for recordings.
    • Speech (websockets use for synthesizers)
    • PostGres (tcp database connection)
    • ConfidentialLedger (presence of a built-in internal identity client creating a new SSL cert for a connection. need to make this available externally somehow and utilize the testproxy adminclient to set the certificate before invoking the tests.)
    • ISV (blocked on inability to record due to resource requirements)
  • The mcp - pullrequest - live build still triggers and runs relevant livetests for the change (including a default set of 3)

Where we want to go

Be aware that how we are addressing this has changed pretty substantively. Check edits on this issue for past iterations.

We now have recordings for the majority (and all new) tools in the repo. We need to rely on recordings and see where the gaps are in our existing recorded tests + core validation.

  • pull request - live will have it's required check status removed.
  • it will still be available to be run manually through /azp run pullrequest - live

The following tools do not support recorded tests at this time:

Everyone else is cutover to support recorded command tests.

Metadata

Metadata

Assignees

Labels

Central-EngSysneeds-team-triageWorkflow: This issue needs the team to triage.server-Azure.McpAzure.Mcp.Servertools-CoreAZMCP Core functionality that all other tools build on top of

Type

No type

Projects

Status

🐝 Dev

Status

Untriaged

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions