feat: Filter temporary _next schemas during inspection#14
Closed
feat: Filter temporary _next schemas during inspection#14
Conversation
Add filtering logic to ignore Snowflake schemas ending with _next or _NEXT (case-insensitive) to prevent race condition errors when these temporary schemas disappear between inspection and execution. - Add filtering in inspect_schemas() function - Add comprehensive unit tests for filtering logic - Fix code formatting in cli.py and utils.py Resolves GEM-5 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
|
🚀 Package published to TestPyPI! You can test the installation with: pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ tundri==1.3.4View package: https://test.pypi.org/project/tundri/1.3.4/ |
lpillmann
changed the title
Collaborator
Author
|
Closing as the fix is being handled in the permifrost repository instead (Gemma-Analytics/permifrost#11). |
lpillmann
deleted the
cyrus/gem-5-intermitent-snowflake-schemas-_next-or-_next-should-be
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes intermittent errors caused by temporary Snowflake schemas with
_nextsuffix being included in permifrost inspection. Our ingestion framework creates schemas with_nextsuffix that may disappear between when tundri starts and when permifrost executes, causingobject not existserrors.Problem
Tundri inspects Snowflake schemas and passes them to permifrost for permission management. Temporary
_nextschemas created by the ingestion framework are ephemeral — they exist when inspection runs but may be gone by the time permifrost processes them.Solution
Filter out schemas ending with
_next(case-insensitive) in tundri'sinspect_schemas()function so they never enter the execution pipeline.Changes
tundri/inspector.py— Added filtering ininspect_schemas()to skip schemas ending with_NEXT(case-insensitive via.upper().endswith("_NEXT"))tests/test_inspector.py— Added comprehensive unit test covering_next,_NEXT, and_NextvariantsTesting
_nextsuffixes are all filteredRelated
Note
This PR adds filtering on the tundri side. The root fix is in permifrost#11 where the error originates. Both changes work together to fully resolve the issue.