feat: finish incremental design and update docs#54
Merged
anaselmhamdi merged 3 commits intobizon-data:mainfrom Jan 6, 2026
Merged
feat: finish incremental design and update docs#54anaselmhamdi merged 3 commits intobizon-data:mainfrom
anaselmhamdi merged 3 commits intobizon-data:mainfrom
Conversation
- Add get_records_after() to Notion source for incremental sync
- get_pages_after(): Search API with client-side filtering
- get_blocks_markdown_after(): Database query with timestamp filter
- Respects database_ids and database_filters configuration
- Add incremental support to file and logger destinations
- file: append mode with temp file handling
- logger: logs sync mode completion
- Update documentation for humans and AI agents:
- README: expanded incremental sync section with flow diagram
- CLAUDE.md: implementation guidance with code examples
- docs/contributing/adding-sources.md: get_records_after() guide
- docs/contributing/adding-destinations.md: finalize() patterns
- docs/ai-connector-guide.md: incremental sync templates
- Add example configs for incremental sync:
- BigQuery, file, logger destinations
- Notion, HubSpot, GSheets sources
- Add live tests for BigQuery incremental sync
aballiet
approved these changes
Jan 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Complete incremental sync implementation for Notion source and comprehensive documentation for humans and AI agents to implement incremental sync in other connectors.
Type of Change
Changes Made
get_records_after()to Notion source for incremental syncget_pages_after(): Search API with client-side filteringget_blocks_markdown_after(): Database query with combined timestamp + user filtersdatabase_idsanddatabase_filtersconfigurationfinalize()get_records_after()implementation guidefinalize()patterns for sync modesTesting
uv run pytest)Manual testing:
Checklist
make format)