Add comprehensive test coverage analysis for azure-ai-evaluation

[Copilot]

Provides visibility into test coverage across the azure-ai-evaluation package to identify testing gaps and prioritize quality improvements.

Analysis Delivered

TEST_COVERAGE_ANALYSIS.md - Module-by-module coverage breakdown with:

• 278 files analyzed, 29% overall coverage (unit tests only)
• Well-tested: Evaluators (87%), Evaluate engine (77%), Graders (100%)
• Coverage gaps: Red Team (3%), Simulator (62%), Common infrastructure (26%)
• Prioritized recommendations for improvement
• Instructions for regenerating coverage reports

Coverage Report Generation

# Run unit tests with coverage
cd sdk/evaluation/azure-ai-evaluation
pytest tests/unittests --cov=azure.ai.evaluation --cov-report=html --cov-branch

# View interactive report
open htmlcov/index.html

Infrastructure Updates

• Added coverage artifacts to .gitignore: coverage.json, htmlcov/, log files
• HTML coverage report generated locally (278 files with line-by-line analysis)

Critical Findings

Red Team module requires immediate attention - 30 files with 0% coverage represent significant quality risk. Simulator and AI service converters need expanded test suites.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• eastus-8.in.applicationinsights.azure.com
  • Triggering command: /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/bin/pytest pytest tests/unittests --cov=azure.ai.evaluation --cov-report=term --cov-report=html --cov-branch -v ash (dns block)
• sanitized.api.cognitive.microsoft.com
  • Triggering command: /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/bin/pytest pytest tests/unittests --cov=azure.ai.evaluation --cov-report=term --cov-report=html --cov-branch -v ash (dns block)
• settings.sdk.monitor.azure.com
  • Triggering command: /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/sdk/evaluation/azure-ai-evaluation/.tox/whl/bin/pytest /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/sdk/evaluation/azure-ai-evaluation/.tox/whl/bin/pytest -rsfE --junitxml=/home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/sdk/evaluation/azure-ai-evaluation/test-junit-whl.xml --verbose --cov-branch --durations=10 --ignore=azure --ignore=.tox --ignore=build --ignore=.eggs --ignore=samples --log-cli-level=51 /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/sdk/evaluation/azure-ai-evaluation d -n 10 (dns block)
  • Triggering command: /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/bin/pytest pytest tests/unittests --cov=azure.ai.evaluation --cov-report=term --cov-report=html --cov-branch -v ash (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

I need to understand test coverage of the azure-ai-evaluation package.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.