aggr-binaries

aggr-binaries indexes large trade archives and builds deterministic, resumable candle binaries.

Core commands:

• index: inventory raw files into SQLite
• process: generate gap-aware binaries + companions
• registry: rebuild derived registry rows from companions
• fixgaps: recover missing trades from persisted gap rows and patch outputs
• clear: delete one market's outputs/state and reindex only that market

Core design constraints:

• determinism (same input => same output)
• resumability (resume from output state)
• scalability (millions of files)

Documentation map

Task docs hold detailed behavior and contracts:

• Index task
• Clear workflow
• Process task
• Registry task
• Fixgaps task
• Merge task (planned)

Requirements

• Node.js >= 22
• sort in PATH (fixgaps rewrite flow)
• unzip in PATH (Kraken direct archive extraction)

Install:

npm install

Configuration

Create config.json (or copy config.example.json) and set:

• root
• dbPath
• batchSize
• flushIntervalSeconds
• includePaths
• outDir
• timeframe

All values can be overridden via CLI flags.

Quickstart

npm start -- index
npm start -- process
npm start -- registry

Selective run example:

npm start -- process --collector RAM --exchange BITMEX --symbol SOLUSD --timeframe 1m

CLI map

• index: docs/index.md
• process: docs/process.md
• registry: docs/registry.md
• fixgaps: docs/fixgaps.md
• clear: docs/index.md (Clear workflow)

Shared flags:

• -r, --root <path>
• -d, --db <path>
• --config <path>
• --no-config
• --include <path> (repeatable)

Input/output at a glance

Input path layout:

{collector}/{bucket}/{exchange}/{symbol}/{YYYY-MM-DD[-HH][.gz]}

Trade line format:

{ts_ms} {price} {size} {side(1=buy)} {liquidation?}

Output per market/timeframe:

{outDir}/{collector}/{exchange}/{symbol}/{timeframe}.bin
{outDir}/{collector}/{exchange}/{symbol}/{timeframe}.json

Binary layout and companion contract are documented in docs/process.md.

Timeline server/client

• Full app: npm run serve
• Client dev: npm run dev:client

Main UI routes:

• /timeline
• /viewer

Operational notes

• DB schema policy is fresh-schema-only; incompatible schemas fail fast with rebuild guidance.
• SQLite write contention (SQLITE_BUSY/SQLITE_LOCKED) is retried with bounded backoff.
• If registry diverges from existing outputs after interruption, run registry for the affected scope.

Development

npm run build:core
npm run build:client
npm run build:tests
npm test