feat: Integration Test Framework for Connector Apps (PART-447)#1200
Closed
tanishkhot wants to merge 18 commits intomainfrom
Closed
feat: Integration Test Framework for Connector Apps (PART-447)#1200tanishkhot wants to merge 18 commits intomainfrom
tanishkhot wants to merge 18 commits intomainfrom
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…RT-453) Add optional keyword-only `description` parameter to all 28 assertion functions. When provided, the description surfaces in failure output alongside expected/actual values, helping developers immediately understand what went wrong without reading source code.
- Target transformed/ subdirectory in load_actual_output() to prevent reading non-JSONL files; falls back to full dir for backward compat - Add expected_file path to GapReport and compare_metadata() so CI failure output shows which baseline file was being compared - Add _write_summary() to BaseIntegrationTest that writes a JSON test summary artifact (configurable via INTEGRATION_TEST_SUMMARY_PATH env) - Add tablePartition to DEFAULT_IGNORED_NESTED_FIELDS for column assets - Add output_subdirectory field to Scenario model
Column names are not unique across tables (e.g., city_id exists in both cities and state_provinces). The comparison now builds a composite lookup key using tableName/viewName/parentName + name (e.g., cities/city_id) when parent context is available. Falls back to name-only for assets without parent references (Database, Schema, Table, etc.).
… (PART-452)
- Add METADATA and CONFIG to APIType enum
- Add _call_metadata() for POST /workflows/v1/metadata
- Add _call_config(), get_config(), update_config() for /workflows/v1/config/{id}
- Add config_action, config_workflow_id, config_payload, schema_base_path fields to Scenario
- Validation: config scenarios require config_action and config_workflow_id
- config_workflow_id supports callables for cross-scenario references
…-452) - New validation.py module with pandera YAML schema validation - Port check_record_count_ge custom check from e2e framework - get_normalised_dataframe() reads JSON/parquet and normalizes to DataFrame - get_schema_file_paths() discovers YAML schemas recursively - validate_with_pandera() orchestrates schema-by-schema validation - Integrated into runner: schema_base_path on Scenario or class triggers pandera validation after workflow completion - format_validation_report() for human-readable error output
1. Pandera validation now waits for workflow completion before running. Extracted polling into _ensure_workflow_completed() called before both metadata comparison and pandera validation. (Flaw 1 — High) 2. Renamed test_scenarios() to _run_all_scenarios() so pytest does not discover it as a phantom test. Removes the misleading extra PASSED line from CI output. (Flaw 2 — Low) 3. Scenario.__post_init__ now validates expected_data file exists at definition time, not runtime. Catches typos immediately instead of after a 300s workflow timeout. (Flaw 5 — Low)
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Contributor
📜 Docstring Coverage ReportRESULT: PASSED (minimum: 30.0%, actual: 80.0%) Detailed Coverage Report |
Contributor
📦 Trivy Vulnerability Scan Results
Report SummaryCould not generate summary table (data length mismatch: 9 vs 8). Scan Result Detailsrequirements.txtVulnerabilities
uv.lockVulnerabilities
|
Contributor
📦 Trivy Secret Scan Results
Report SummaryCould not generate summary table (data length mismatch: 9 vs 8). Scan Result Detailsrequirements.txtuv.lock |
Collaborator
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
Contributor
|
🛠 Full Test Coverage Report: https://k.atlan.dev/coverage/application-sdk/pr/1200 |
11-step guide: understand connector, write scenarios, add metadata validation, create CI workflow, set up VPN, enable merge blocking. Any dev can invoke /write-integration-tests in Claude Code.
…e blocking to Claude skill
OnkarVO7
approved these changes
Apr 14, 2026
Contributor
📦 Example workflows test results
|
Contributor
📦 Example workflows test results
|
Contributor
📦 Example workflows test results
|
5 tasks
Contributor
Author
|
Closing in favor of #1322 which has verified commit signatures (required by branch protection). Same code changes, squashed into a single signed commit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a declarative integration test framework to the Application SDK that enables connector developers to write end-to-end tests against real data sources with zero boilerplate.
Proven on 3 connectors (Postgres, Tableau, ClickHouse) across SQL, REST, and VPN-protected sources — 57 test scenarios, all passing on GitHub Actions.
What's New
Framework (
application_sdk/test_utils/integration/)runner.pyBaseIntegrationTest— auto-discovers credentials fromE2E_*env vars, generates pytest methods per scenario, manages workflow polling and cleanupclient.pyIntegrationTestClient— unified HTTP client for 5 API types: AUTH, METADATA, PREFLIGHT, WORKFLOW, CONFIGassertions.pyequals,contains,any_of,matches, etc.) with custom failure descriptionsmodels.pyScenariodataclass — declarative test definitions with credentials, metadata, connection overridescomparison.pyvalidation.pylazy.pyKey Features
Scenario(name="auth_valid", api="auth", assert_that={"success": equals(True)})and the framework handles everything.E2E_{APP_NAME}_*env vars. Per-scenario overrides viaE2E_{SCENARIO_NAME}_*.any_of(contains("password authentication failed"), contains("does not exist"))— catch specific failure modes, not justsuccess: False.Tickets Completed (9 under PART-447)
Demo PRs (Live on GitHub Actions)
[MISSING] MaterialisedView/mv_9999— SQL change drops assetHow to Use (Connector Developer)
Set
E2E_{APP_NAME}_*env vars, copy the GHA workflow YAML, addint-testlabel to a PR. Done.Test Plan