Skip to content

Feature/Databricks support #129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 13 commits into
base: main
Choose a base branch
from
Draft

Feature/Databricks support #129

wants to merge 13 commits into from

Conversation

sleblanc23
Copy link
Contributor

@sleblanc23 sleblanc23 commented Apr 16, 2025

Description & motivation

This PR, along with these changes in edu_wh, makes EDU compatible with Databricks.

TO DO: consider merging in this PR

Breaking changes introduced by this PR:

None

New files:

  • Added several macros that handle differences in function syntax and complications that arise from how Databricks returns values from array functions (always as a named_struct, not a variant).
  • The star macro (name suggestions welcome) implements Snowflake's exclude (which is called except in Databricks) and rename (which is not present in Databricks).

Changes to existing files:

  • Cleaned up in the extract_extension macro
  • Modified to the source_edfi3 and flatten_arrays macros to call new array handling macros
  • Substituted a macro call in place of Snowflake-specific array behavior throughout the stage models
  • Rewrote of stg_ef3__contacts and stg_ef3__student_contact_associations (pending decision on PR linked above)

Tests and QC done:

Ran in GSN and SC; confirmed that all affected models are unchanged by comparing row counts and relevant column values

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants