fix: JSONB double-encode + splitBody wiki + parseEmbedding (v0.12.1)

[garrytan]

Summary

Data-correctness hotfix for v0.12.0 Postgres-backed brains. PGLite users were unaffected. Bundles community PRs #187 (@knee5) and #175 (@leonardsellem) with expanded migration scope, schema audit (5 affected JSONB columns vs 3 originally reported), CI grep guard, and an E2E regression test that should have caught the original bug.

JSONB double-encode (Postgres only). Every ${JSON.stringify(value)}::jsonb interpolation in postgres-engine.ts and files.ts caused postgres.js v3 to stringify again on the wire, storing JSONB columns as quoted string literals. Every frontmatter->>'key' returned NULL on Postgres-backed brains. GIN indexes were inert. Switched to sql.json(value) (postgres.js native, OID 3802). Affected columns: pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter. PGLite hid this bug entirely — different driver path.

splitBody truncation. Treated any standalone --- as timeline separator, causing 83% content truncation on wiki corpora (1,991-article wiki, 4,856 of 6,680 wikilinks lost). New behavior requires explicit sentinel: <!-- timeline -->, --- timeline ---, or --- directly before ## Timeline / ## History heading.

inferType wiki subtypes. Added /writing/, /wiki/analysis/, /wiki/guides/, /wiki/hardware/, /wiki/architecture/, /wiki/concepts/. Path order is most-specific-first so projects/blog/writing/essay.md → writing.

pgvector NaN scores (Supabase). getEmbeddingsByChunkIds returned strings instead of Float32Array on Supabase, producing [NaN] cosine scores. Adds parseEmbedding() helper. Throws loud on malformed vectors (no silent NaN); returns null for non-vector strings.

Wikilink extraction. [[page]] and [[page|Display]] syntaxes now extracted alongside standard [text](page.md). resolveSlug() does ancestor-search for wiki KBs that omit ../.

Migration. New gbrain repair-jsonb command + v0_12_1 orchestrator (schema → repair → verify → record). Idempotent — re-running is a no-op. PGLite engines short-circuit cleanly.

CI grep guard at scripts/check-jsonb-pattern.sh fails the build if anyone reintroduces the ${JSON.stringify(x)}::jsonb pattern.

Test Coverage

[+] src/core/markdown.ts — splitBody/serializeMarkdown/inferType
    └── [★★★] markdown.test.ts: 10 splitBody + 5 round-trip + wiki/writing inferType cases
[+] src/core/postgres-engine.ts — sql.json() at putPage/putRawData/logIngest
    └── [★★★] e2e/postgres-jsonb.test.ts: 4 round-trip assertions on real Postgres
[+] src/core/utils.ts — parseEmbedding helper, rowToChunk delegation
    └── [★★★] utils.test.ts: F32A passthrough, pgvector string, null, throw on garbage
[+] src/commands/extract.ts — wikilink extraction, resolveSlug ancestor search
    └── [★★★] extract.test.ts existing coverage
[+] src/commands/files.ts:254 — sql.json metadata
    └── [★★★] e2e/postgres-jsonb.test.ts: files.metadata round-trip
[+] src/commands/migrations/v0_12_1.ts — JSONB repair orchestrator
    └── [★★★] migrations-v0_12_1.test.ts: registry, dry-run, phase exports
[+] src/commands/repair-jsonb.ts — repair command + PGLite short-circuit
    └── [★★★] repair-jsonb.test.ts (PGLite no-op) + e2e/postgres-jsonb.test.ts (real repair)
[+] scripts/check-jsonb-pattern.sh — CI grep guard
    └── [★★] Manual + wired into bun test

Coverage: 8/8 paths (100%). Tests: 1361 → 1415 (+54 new).

E2E suite: 120/120 pass against pgvector/pg16 Docker container. Unit suite: 1415/1415 pass. CI grep guard passes on this diff (no JSON.stringify(x)::jsonb patterns in src/).

Pre-Landing Review

No new issues found. Specialists already comprehensively covered by /plan-eng-review + Codex outside-voice review during planning (25+ findings, 3 material tensions adjudicated). repair-jsonb.ts uses sql.unsafe with table/column names from a hardcoded TARGETS array — no injection vector. Migration is idempotent. parseEmbedding throws loud on malformed input per Codex's no-silent-NaN requirement.

Plan Completion

All 24 planned items DONE. Scope reduced from 9-PR bundle to 2-PR hotfix per Codex outside-voice scope challenge. The remaining 7 PRs (#184, #177, #132, #114, #115, #119, #123) deferred to v0.12.2 follow-up wave per /Users/garrytan/.claude/plans/system-instruction-you-are-working-elegant-squid.md.

TODOS

No items in TODOS.md were specifically completed by this PR (it focused on BrainBench eval work).

Documentation

Documentation was synced to v0.12.1 in commit 998ef82. Six files updated to reflect the JSONB hotfix, splitBody sentinel contract, wiki inferType, and native wikilink extraction.

• README.md — added gbrain repair-jsonb [--dry-run] to the ADMIN command reference
• CLAUDE.md — registered new files, documented splitBody sentinel precedence and inferType wiki subtypes
• INSTALL_FOR_AGENTS.md — fixed stale verification check counts, added v0.12.1 upgrade guidance
• docs/GBRAIN_VERIFY.md — added check perf: parallelize hybrid search pipeline #8 (JSONB Frontmatter Integrity)
• docs/UPGRADING_DOWNSTREAM_AGENTS.md — added v0.12.1 hotfix section explaining the splitBody contract change
• skills/migrate/SKILL.md — documented native [[wikilink]] and [[wikilink|Display]] extraction

CHANGELOG.md was left untouched (already comprehensive). VERSION bumped to 0.12.1.

Test plan

• All unit tests pass (1322/1322)
• All E2E tests pass against real pgvector (120/120, including new test/e2e/postgres-jsonb.test.ts)
• CI grep guard passes on current src/
• PGLite repair-jsonb test confirms no DB connection / 0 rows reported
• Migration dry-run skips all side-effect phases
• Manual smoke on a real Postgres-backed brain: gbrain put a page with frontmatter, query frontmatter->>'key' returns the value
• Manual smoke: gbrain repair-jsonb --dry-run against a brain with double-encoded rows reports the correct count

Attribution

Built on community PRs:

• fix: markdown parsing bugs affecting wiki-style content #187 by @knee5 — splitBody, inferType wiki, JSONB triple-fix in postgres-engine.ts. Re-implemented (PR fix: markdown parsing bugs affecting wiki-style content #187 was CONFLICTING after Minions v7 landed) with expanded migration scope.
• fix: parse pgvector embeddings before cosine rescoring #175 by @leonardsellem — parseEmbedding() helper, getEmbeddingsByChunkIds fix.

Both PRs reported the bugs and proposed the fixes. Codex outside-voice review during planning surfaced the missed page_versions.frontmatter propagation path, dropped the noisy-truncated-diagnostic anti-pattern from scope, and pushed for the engine-aware migration.

🤖 Generated with Claude Code