feat: add destination readback introspection for smoke tests#1000

Merged

Aaron ("AJ") Steers (aaronsteers) merged 49 commits intomainfrom

devin/1774225239-destination-readback-testing

Mar 23, 2026

Commits on Mar 23, 2026

feat: add destination readback introspection for smoke tests

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: address CI lint and type check failures

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: use uv run ruff format for CI-compatible formatting

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: use cache's built-in normalizers and move query methods to CacheBase

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: add get_sql_cache() to Destination, simplify readback to use it

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: case-insensitive key lookup for column stats across DBs

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: eliminate destination_readback.py, fix key casing and secret hydration bugs

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
style: convert docstring from reST to markdown syntax

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: use positional aliases in column stats query to avoid truncation

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: quote table names in readback SQL queries for Snowflake compatibility

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: handle dict config in postgres_destination_to_cache for local readback

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: case-insensitive row_count lookup + fallback to original stream name for CamelCase tables

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: move inline imports to top-level in CacheBase readback methods

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: import sorting in base.py + handle dict config and /local path in duckdb_destination_to_cache

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
style: fix ruff format in duckdb_destination_to_cache

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: handle dict config in motherduck_destination_to_cache

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: strip destinationType key in bigquery and snowflake dest-to-cache converters

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: use dialect-aware identifier quoting in readback SQL (BigQuery backticks)

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: move readback logic from CacheBase to SqlProcessorBase per review feedback

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: restore CI-expected formatting in sql_processor.py

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
style: convert all docstrings from reST to Markdown formatting

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: make stats fields nullable, broaden readback exception handling per review

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
docs: add Markdown docstring style guideline to CONTRIBUTING.md

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: pass schema_name through destination_to_cache, add db/schema to TableStatistics, fix docstring

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: improve get_column_info docstring, share inspector across tables

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: make get_column_stats private (_get_column_stats)

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: rename get_* to fetch_* for DB-hitting introspection methods

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: replace _get_stream_names_from_source with Source.get_selected_streams()

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: delete DestinationReadbackResult, flatten table_statistics into DestinationSmokeTestResult

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: make tables_not_found a dict mapping stream_name to expected table_name

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: add Destination.is_cache_supported, remove try/except from readback

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: run readback regardless of write success (partial success support)

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: improve readback skip log message per review suggestion

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: simplify get_sql_cache signature, remove destination_name/config/version params

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: move translation files from destinations/ to caches/_utils/ to resolve circular import

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: add copyright notice to caches/_utils/__init__.py

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: promote inline imports to top-level now that circular dep is resolved

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
docs: fix stale docstrings - readback runs regardless of write success

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
docs: fix remaining double backticks to use Markdown style

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: wrap readback introspection in try/except to preserve structured result

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: dispose engine after schema_name override to fix stale schema_translate_map

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: fall back to default schema when namespace schema has no tables

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: override destination config schema key for namespace isolation

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
style: ruff format destination_smoke_tests.py

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: force disable_type_dedupe=false for readback and improve config override

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
refactor: remove config schema override — catalog namespace is sufficient

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: skip redundant config validation in _prepare_destination_config

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: use plaintext password directly in snowflake_destination_to_cache

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: use destination's credentials_json for BigQuery cache construction

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed