Incremental batch data pipeline using AWS S3 and AWS Glue with date-based partitioning.
-
Updated
Jan 13, 2026 - Python
#
incremental-load
Here are 4 public repositories matching this topic...
End-to-end Metadata-Driven Data Engineering framework built on Azure. Features dynamic SQL/REST API ingestion with range pagination, automated schema mapping, and event-driven orchestration. Implements robust CI/CD via GitHub Actions/YAML and automated failure alerting with Logic Apps. Optimized for scalability and DE best practices.
rest-api data-engineering cicd azure-sql-database azure-data-factory etl-pipeline data-engineering-pipeline azure-logic-apps data-orchestration yaml-templating data-lifecycle-management etl-developer metadata-driven-pipelines cloud-data-engineering adls-gen2 azure-data-engineer incremental-load
-
Updated
Jan 16, 2026
End-to-end data engineering pipeline using Databricks, Amazon S3, Delta Lake and Unity Catalog, with full and incremental loads.
-
Updated
Dec 13, 2025 - Jupyter Notebook
This project implements a comprehensive event-driven data pipeline for e-commerce transactional data processing using Databricks, PySpark, and Delta Lake. The pipeline handles multiple data sources with advanced data engineering patterns including SCD2 (Slowly Changing Dimensions), data validation, enrichment, and automated archiving.
-
Updated
Oct 4, 2025 - Jupyter Notebook
Improve this page
Add a description, image, and links to the incremental-load topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the incremental-load topic, visit your repo's landing page and select "manage topics."