Skip to content

Conversation

Copy link

Copilot AI commented Dec 9, 2025

Description

The weekly live testing pipeline fails with AssertionError: Connection 'azure_ai_project_scope' not found in dev connections. Tests expected a connections.json file that doesn't exist in CI/CD environments. This PR implements the standard Azure SDK pattern for test resource provisioning.

Changes

Added sdk/evaluation/test-resources.json

  • ARM template provisions Azure ML workspace (AI project), Azure OpenAI with gpt-35-turbo deployment, and supporting infrastructure
  • Outputs environment variables: AZURE_AI_PROJECT_*, AZURE_OPENAI_*

Updated sdk/evaluation/azure-ai-evaluation/tests/conftest.py

  • Added _get_connection_from_env() to read from environment variables when connections.json absent
  • Maintains backward compatibility with existing connections.json workflow

Added sdk/evaluation/TESTING.md

  • Documents both local development (connections.json) and CI/CD (ARM template) testing approaches

The pipeline will now use New-TestResources.ps1 to deploy the ARM template, setting environment variables that tests consume.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.
Original prompt

Currently our weekly live testing pipeline fails with the following error:

failed on setup with "AssertionError: Connection 'azure_ai_project_scope' not found in dev connections.
assert 'azure_ai_project_scope' in {}"

E assert 'azure_ai_project_scope' in {}

It seems we are missing the proper testing resources to enable weekly live testing. Follow best practice and documentation to get our tests passing. Link to pipeline: https://dev.azure.com/azure-sdk/internal/_build?definitionId=7163


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 3 commits December 9, 2025 15:39
… variables

Co-authored-by: slister1001 <103153180+slister1001@users.noreply.github.com>
… variable

Co-authored-by: slister1001 <103153180+slister1001@users.noreply.github.com>
Co-authored-by: slister1001 <103153180+slister1001@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix assertion error in weekly live testing pipeline Fix weekly live testing pipeline - add test-resources.json for Azure AI Evaluation Dec 9, 2025
Copilot AI requested a review from slister1001 December 9, 2025 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants