SNOW-3347488: reduce describe query by reusing parent attributes for cte optimization on and reduce desc query on#4175
Draft
sfc-gh-aling wants to merge 13 commits intomainfrom
Draft
SNOW-3347488: reduce describe query by reusing parent attributes for cte optimization on and reduce desc query on#4175sfc-gh-aling wants to merge 13 commits intomainfrom
sfc-gh-aling wants to merge 13 commits intomainfrom
Conversation
|
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4175 +/- ##
==========================================
- Coverage 95.42% 92.27% -3.16%
==========================================
Files 171 171
Lines 43615 43644 +29
Branches 7459 7470 +11
==========================================
- Hits 41620 40271 -1349
- Misses 1220 2504 +1284
- Partials 775 869 +94 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…t' into fix/infer-alias-types-from-parent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-3347488
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Change 1: General reduce-describe optimization
What: Enhanced
_extract_inferable_attribute_namesto acceptfrom_attributesand resolveAlias(Attribute(...))types by looking up the parent plan's known types by column name. Also enhanced_extract_selectable_attributesto pass from_attributes downstream. This is a general improvement to the metadata inference -- it works regardless of CTE.Files: metadata_utils.py only
Effect: When reduce_describe_query_enabled is ON, more projections can be resolved locally without a DESCRIBE, because
Alias(Attribute)expressions now inherit types from theFROMclause.Change 2: CTE-specific describe reduction
What: Three things:
Added
Alias(Cast(...))inference usingCast.tofor scalar types (skipping structured types to avoid Snowflake type promotion issues)Added
try_infer_attributes_from_flattened_projectionand replaced the unconditional_attributes = NoneinSelectStatement.select()flattening with a smart re-derive attemptAdded unit tests (test_metadata_utils.py) and integ tests (test_reduce_describe_query.py)
Files: metadata_utils.py, select_statement.py, plus new test files
Effect: When CTE optimization is ON, the
select()flattening no longer unconditionally discards cached attributes. Combined withCastinference, this eliminates the extra describe query for self-joins on createDataFrame tables.The two changes are complementary -- Change 1 handles the Alias(Attribute) case (column pass-through), Change 2 handles the Alias(Cast) case (typed columns from createDataFrame with schema). Together they cover the two most common projection patterns that trigger unnecessary describes.