feat: PG-Sync V2 — ops pipeline with BitmapSink dual-path#85

Merged

JustMaier merged 19 commits intomainfrom

feat/sync-v2

Mar 26, 2026

Commits on Mar 26, 2026

feat: Sync V2 foundation — ops types, dedup helper, WAL writer/reader

JustMaier
and
claude
committed
feat: POST /ops endpoint + GET /sync-lag for Sync V2

JustMaier
and
claude
committed
feat: WAL ops processor — converts ops to engine mutations

JustMaier
and
claude
committed
feat: V2 ops poller — reads BitdexOps, deduplicates, POSTs to /ops endpoint

JustMaier
and
claude
committed
feat: YAML trigger config + SQL generator for Sync V2

JustMaier
and
claude
committed
feat: dump pipeline — registry, persistence, server endpoints

JustMaier
and
claude
committed
feat: V2 sync Prometheus metrics (bitdex_sync_* namespace)

JustMaier
and
claude
committed
test: Sync V2 integration tests

JustMaier
and
claude
committed
feat: wire WAL reader thread into server startup

JustMaier
and
claude
committed
feat: CSV→ops adapter + WAL reader thread wiring

JustMaier
and
claude
committed
feat: rewrite ops_processor — BitmapSink dual-path + direct dump

JustMaier
and
claude
committed
docs: update pg-sync-v2 design — pg-sync as data mover, BitDex owns processing

JustMaier
and
claude
committed
fix: add computed field to SortFieldConfig in tests (rebase fix)

JustMaier
and
claude
committed
perf: rayon parallel CSV parsing in direct dump path

JustMaier
and
claude
committed
fix: address Ivanna review — incremental WAL reads + loading mode in apply_accum

JustMaier
and
claude
committed
feat: wire ClickHouse metrics poller into V2 ops pipeline

JustMaier
and
claude
committed
fix: skip WAL phase in direct mode validation harness

JustMaier
and
claude
committed
fix: simplify direct dump — read_lines + loading mode, fix chunked reader hang

JustMaier
and
claude
committed
fix: chunked CSV processing to prevent OOM on 67GB tags.csv

JustMaier
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: PG-Sync V2 — ops pipeline with BitmapSink dual-path#85

feat: PG-Sync V2 — ops pipeline with BitmapSink dual-path#85
JustMaier merged 19 commits intomainfrom
feat/sync-v2

Commits on Mar 26, 2026

feat: Sync V2 foundation — ops types, dedup helper, WAL writer/reader

feat: POST /ops endpoint + GET /sync-lag for Sync V2

feat: WAL ops processor — converts ops to engine mutations

feat: V2 ops poller — reads BitdexOps, deduplicates, POSTs to /ops endpoint

feat: YAML trigger config + SQL generator for Sync V2

feat: dump pipeline — registry, persistence, server endpoints

feat: V2 sync Prometheus metrics (bitdex_sync_* namespace)

test: Sync V2 integration tests

feat: wire WAL reader thread into server startup

feat: CSV→ops adapter + WAL reader thread wiring

feat: rewrite ops_processor — BitmapSink dual-path + direct dump

docs: update pg-sync-v2 design — pg-sync as data mover, BitDex owns processing

fix: add computed field to SortFieldConfig in tests (rebase fix)

perf: rayon parallel CSV parsing in direct dump path

fix: address Ivanna review — incremental WAL reads + loading mode in apply_accum

feat: wire ClickHouse metrics poller into V2 ops pipeline

fix: skip WAL phase in direct mode validation harness

fix: simplify direct dump — read_lines + loading mode, fix chunked reader hang

fix: chunked CSV processing to prevent OOM on 67GB tags.csv