Skip to content

feat: finish incremental design and update docs#54

Merged
anaselmhamdi merged 3 commits intobizon-data:mainfrom
gorgias:finish-incremental-bq
Jan 6, 2026
Merged

feat: finish incremental design and update docs#54
anaselmhamdi merged 3 commits intobizon-data:mainfrom
gorgias:finish-incremental-bq

Conversation

@anaselmhamdi
Copy link
Collaborator

Complete incremental sync implementation for Notion source and comprehensive documentation for humans and AI agents to implement incremental sync in other connectors.

Type of Change

  • New feature

Changes Made

  • Add get_records_after() to Notion source for incremental sync
    • get_pages_after(): Search API with client-side filtering
    • get_blocks_markdown_after(): Database query with combined timestamp + user filters
    • Respects database_ids and database_filters configuration
  • Add incremental support to file and logger destinations
    • file: append mode with temp file handling in finalize()
    • logger: logs sync mode completion
  • Update documentation:
    • README: expanded incremental sync section with flow diagram, config options, examples
    • CLAUDE.md: implementation guidance with code examples for AI agents
    • docs/contributing/adding-sources.md: get_records_after() implementation guide
    • docs/contributing/adding-destinations.md: finalize() patterns for sync modes
    • docs/ai-connector-guide.md: incremental sync templates and decision trees
  • Add example configs for incremental sync:
    • Destinations: BigQuery, file, logger
    • Sources: Notion, HubSpot, GSheets
  • Add live integration tests for BigQuery incremental sync

Testing

  • Unit tests added/updated
  • Manual testing performed
  • Existing tests pass (uv run pytest)

Manual testing:

  • Ran Notion incremental sync with live credentials
  • First run: full refresh behavior (fetched all data)
  • Second run: correctly found only 1 page edited after last_run
  • Verified blocks fetched and appended to destination file

Checklist

  • Code follows project style (make format)
  • Self-review completed
  • Documentation updated (if needed)

  - Add get_records_after() to Notion source for incremental sync
    - get_pages_after(): Search API with client-side filtering
    - get_blocks_markdown_after(): Database query with timestamp filter
    - Respects database_ids and database_filters configuration

  - Add incremental support to file and logger destinations
    - file: append mode with temp file handling
    - logger: logs sync mode completion

  - Update documentation for humans and AI agents:
    - README: expanded incremental sync section with flow diagram
    - CLAUDE.md: implementation guidance with code examples
    - docs/contributing/adding-sources.md: get_records_after() guide
    - docs/contributing/adding-destinations.md: finalize() patterns
    - docs/ai-connector-guide.md: incremental sync templates

  - Add example configs for incremental sync:
    - BigQuery, file, logger destinations
    - Notion, HubSpot, GSheets sources

  - Add live tests for BigQuery incremental sync
@anaselmhamdi anaselmhamdi requested a review from aballiet January 6, 2026 14:18
@anaselmhamdi anaselmhamdi self-assigned this Jan 6, 2026
@anaselmhamdi anaselmhamdi merged commit 189cf19 into bizon-data:main Jan 6, 2026
1 check passed
@anaselmhamdi anaselmhamdi deleted the finish-incremental-bq branch January 6, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants