Skip to content

feat(templates): add EntityDef and entity-driven orchestration to Sql…#1297

Closed
AtMrun wants to merge 1 commit intochore/remove-deprecated-v2-modulesfrom
feat/entity-registry
Closed

feat(templates): add EntityDef and entity-driven orchestration to Sql…#1297
AtMrun wants to merge 1 commit intochore/remove-deprecated-v2-modulesfrom
feat/entity-registry

Conversation

@AtMrun
Copy link
Copy Markdown
Collaborator

@AtMrun AtMrun commented Apr 13, 2026

…MetadataExtractor

Introduces declarative entity definitions that replace hardcoded asyncio.gather of 4 entities in SqlMetadataExtractor.run().

  • EntityDef dataclass: name, sql, endpoint, phase, enabled, timeout
  • Phased orchestration: entities grouped by phase, run concurrently within phase, sequentially across phases
  • fetch_entity() dispatches to fetch{name}() methods by convention
  • Full backward compat: empty entities list falls back to default 4 (databases, schemas, tables, columns)
  • Adding a new entity type is one line in the entities list

What a Snowflake extractor looks like:

class SnowflakeExtractor(SqlMetadataExtractor):
    entities = [
        EntityDef(name="databases", phase=1),
        EntityDef(name="schemas",   phase=1),
        EntityDef(name="stages",    phase=2),
        EntityDef(name="streams",   phase=2),
    ]

Changelog

  • to be added

Additional context (e.g. screenshots, logs, links)

  • to be added

Checklist

  • Additional tests added
  • All CI checks passed
  • Relevant documentation updated

Copyleft License Compliance

  • Have you used any code that is subject to a Copyleft license (e.g., GPL, AGPL, LGPL)?
  • If yes, have you modified the code in the context of this project? please share additional details.

…MetadataExtractor

Introduces declarative entity definitions that replace hardcoded
asyncio.gather of 4 entities in SqlMetadataExtractor.run().

- EntityDef dataclass: name, sql, endpoint, phase, enabled, timeout
- Phased orchestration: entities grouped by phase, run concurrently
  within phase, sequentially across phases
- _fetch_entity() dispatches to fetch_{name}() methods by convention
- Full backward compat: empty entities list falls back to default 4
  (databases, schemas, tables, columns)
- Adding a new entity type is one line in the entities list

What a Snowflake extractor looks like:

    class SnowflakeExtractor(SqlMetadataExtractor):
        entities = [
            EntityDef(name="databases", phase=1),
            EntityDef(name="schemas",   phase=1),
            EntityDef(name="stages",    phase=2),
            EntityDef(name="streams",   phase=2),
        ]
@snykgituser
Copy link
Copy Markdown

snykgituser commented Apr 13, 2026

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@Aryamanz29
Copy link
Copy Markdown
Member

Closing — duplicate of #1298 (same feature, earlier iteration). Branch preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants