feat: Sync V2 data coverage — all CSV tables + deferred alive#86

Merged

JustMaier merged 27 commits intomainfrom

feat/sync-v2

Mar 26, 2026

Commits on Mar 26, 2026

feat: Sync V2 foundation — ops types, dedup helper, WAL writer/reader

JustMaier
and
claude
committed
feat: POST /ops endpoint + GET /sync-lag for Sync V2

JustMaier
and
claude
committed
feat: WAL ops processor — converts ops to engine mutations

JustMaier
and
claude
committed
feat: V2 ops poller — reads BitdexOps, deduplicates, POSTs to /ops endpoint

JustMaier
and
claude
committed
feat: YAML trigger config + SQL generator for Sync V2

JustMaier
and
claude
committed
feat: dump pipeline — registry, persistence, server endpoints

JustMaier
and
claude
committed
feat: V2 sync Prometheus metrics (bitdex_sync_* namespace)

JustMaier
and
claude
committed
test: Sync V2 integration tests

JustMaier
and
claude
committed
feat: wire WAL reader thread into server startup

JustMaier
and
claude
committed
feat: CSV→ops adapter + WAL reader thread wiring

JustMaier
and
claude
committed
feat: rewrite ops_processor — BitmapSink dual-path + direct dump

JustMaier
and
claude
committed
docs: update pg-sync-v2 design — pg-sync as data mover, BitDex owns processing

JustMaier
and
claude
committed
fix: add computed field to SortFieldConfig in tests (rebase fix)

JustMaier
and
claude
committed
perf: rayon parallel CSV parsing in direct dump path

JustMaier
and
claude
committed
fix: address Ivanna review — incremental WAL reads + loading mode in apply_accum

JustMaier
and
claude
committed
feat: wire ClickHouse metrics poller into V2 ops pipeline

JustMaier
and
claude
committed
fix: skip WAL phase in direct mode validation harness

JustMaier
and
claude
committed
fix: simplify direct dump — read_lines + loading mode, fix chunked reader hang

JustMaier
and
claude
committed
fix: chunked CSV processing to prevent OOM on 67GB tags.csv

JustMaier
and
claude
committed
wip: computed sort field deps in FieldMeta + gap analysis

JustMaier
and
claude
committed
feat: wire all 10 CSV tables into V2 dump pipeline

JustMaier
and
claude
committed
feat: deferred alive in V2 ops pipeline

JustMaier
and
claude
committed
fix: reduce chunk size to 2M and free enrichment after images

JustMaier
and
claude
committed
fix: deferred alive in dump path + BitmapSink trait

JustMaier
and
claude
committed
perf: bypass Op abstraction in image dump — direct bitmap writes

JustMaier
and
claude
committed
revert: save+unload between phases — needs staging lifecycle work

JustMaier
and
claude
committed
Merge remote-tracking branch 'origin/main' into feat/sync-v2
JustMaier
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Sync V2 data coverage — all CSV tables + deferred alive#86

feat: Sync V2 data coverage — all CSV tables + deferred alive#86
JustMaier merged 27 commits intomainfrom
feat/sync-v2

Commits on Mar 26, 2026

feat: Sync V2 foundation — ops types, dedup helper, WAL writer/reader

feat: POST /ops endpoint + GET /sync-lag for Sync V2

feat: WAL ops processor — converts ops to engine mutations

feat: V2 ops poller — reads BitdexOps, deduplicates, POSTs to /ops endpoint

feat: YAML trigger config + SQL generator for Sync V2

feat: dump pipeline — registry, persistence, server endpoints

feat: V2 sync Prometheus metrics (bitdex_sync_* namespace)

test: Sync V2 integration tests

feat: wire WAL reader thread into server startup

feat: CSV→ops adapter + WAL reader thread wiring

feat: rewrite ops_processor — BitmapSink dual-path + direct dump

docs: update pg-sync-v2 design — pg-sync as data mover, BitDex owns processing

fix: add computed field to SortFieldConfig in tests (rebase fix)

perf: rayon parallel CSV parsing in direct dump path

fix: address Ivanna review — incremental WAL reads + loading mode in apply_accum

feat: wire ClickHouse metrics poller into V2 ops pipeline

fix: skip WAL phase in direct mode validation harness

fix: simplify direct dump — read_lines + loading mode, fix chunked reader hang

fix: chunked CSV processing to prevent OOM on 67GB tags.csv

wip: computed sort field deps in FieldMeta + gap analysis

feat: wire all 10 CSV tables into V2 dump pipeline

feat: deferred alive in V2 ops pipeline

fix: reduce chunk size to 2M and free enrichment after images

fix: deferred alive in dump path + BitmapSink trait

perf: bypass Op abstraction in image dump — direct bitmap writes

revert: save+unload between phases — needs staging lifecycle work

Merge remote-tracking branch 'origin/main' into feat/sync-v2