v3.0.0 "Spine" — extensible MCP aggregator, mistakes moat, 89.1% measured savings

CHANGELOG.md

@@ -6,6 +6,76 @@ All notable changes to engram are documented here. Format based on
 
 ## [Unreleased]
 
+## [3.0.0] — 2026-04-24 — "Spine"
+
+The biggest engramx release since v1.0. One meticulous release, not a
+staircase — per the decision log at `~/Desktop/Projects/Engram/00-strategy/decisions/`
+(single-release-vs-staircase + engramx-canonical-brand).
+
+Headline: engramx becomes the **extensible context spine**. Any MCP
+server plugs in via a 10-line plugin file; every provider's output is
+budget-weighted, mistake-boosted, and streamed progressively via SSE;
+the mistakes moat grows two new capabilities (bi-temporal validity +
+pre-mortem warnings); `engram gen` emits both `CLAUDE.md` AND `AGENTS.md`
+by default. **Real-world benchmark: 89.1% measured savings** on engramx's
+own 87-file sample (committed report in `bench/results/`).
+
+Contributor credit: [@mechtar-ru](https://github.com/mechtar-ru) for PR #6
+(OOM fixes on large codebases — cherry-picked with preserved authorship).
+
+### Added — v3.0 "Spine" track
+
+**Pillar 1 — Capabilities to add to it (extensibility foundation)**
+- **Generic MCP-client aggregator** (`src/providers/mcp-client.ts`). Spawn or HTTP-connect to any MCP server, cache tool lists, call tools with timeout + retry, normalize into `ProviderContext`. Config at `~/.engram/mcp-providers.json`. Per-provider budgets, graceful degradation, process shutdown hooks. Uses `@modelcontextprotocol/sdk` v1.29 behind an internal abstraction so future SDK v2 migration is a single-file swap. Stdio transport ships; HTTP path stubbed pending post-3.0 Host/Origin hardening integration.
+- **Provider plugin contract v2** (`src/providers/plugin-loader.ts`). Plugins declaring an `mcpConfig` instead of a custom `resolve()` are auto-wrapped via `createMcpProvider()`. Classic plugins with hand-rolled `resolve()` still work unchanged. Custom `resolve()` wins if both are present. 10-line plugins are now possible.
+- **Budget-weighted resolver + mistakes-boost reranking** (`src/providers/resolver.ts`). Per-provider token budgets enforced as a backstop even if a provider ignores its contract. Results whose content mentions a known-mistake label get confidence × 1.5 (capped at 1.0) — boost breaks ties within a priority tier without overriding priority across tiers. Case-insensitive label matching.
+
+**Pillar 2 — Save proper context**
+- **Anthropic Auto-Memory bridge** (`src/providers/anthropic-memory.ts`). Reads Claude Code's auto-managed `~/.claude/projects/<encoded>/memory/MEMORY.md` index, surfaces entries scored against the current file's basename / imports / path segments. Tier 1, runs under 10 ms, max 1 MB hard-cap on index size. Override via `ENGRAM_ANTHROPIC_MEMORY_PATH` for tests + advanced users. Inserted at `PROVIDER_PRIORITY[3]` between mistakes and mempalace.
+- **Streaming partial context packets via SSE** (`/context/stream?file=<path>` endpoint + `resolveRichPacketStreaming()` generator). Emit one SSE frame per provider as it resolves. Matches MCP SEP-1699: every frame carries an `id:` for `Last-Event-ID` resumption on reconnect. Client disconnect mid-stream aborts the generator cleanly. Inherits existing auth + Host + Origin guards.
+- **Serena plugin reference** at `docs/plugins/examples/serena-plugin.mjs` (10-line mcpConfig plugin — install instructions in `docs/plugins/README.md`).
+
+**Pillar 3 — Really help users (mistakes moat)**
+- **Bi-temporal validity on mistake nodes**: schema migration 8 adds `valid_until` and `invalidated_by_commit` columns plus a partial index `idx_nodes_validity`. Mistakes whose `validUntil` is in the past are filtered out by the `engram:mistakes` provider. Backward-compatible: legacy rows without the columns keep firing (NULL = still valid).
+- **Pre-mortem mistake-guard** (`src/intercept/handlers/mistake-guard.ts`). Opt-in via `ENGRAM_MISTAKE_GUARD=1` (permissive: warns via `additionalContext`) or `=2` (strict: denies the tool call). Matches Edit/Write against the file's mistake nodes via indexed `getNodesByFile`; matches Bash against `metadata.commandPattern` substrings and `sourceFile` mentions in the command. Respects the bi-temporal filter. Zero overhead when unset.
+
+**Hygiene / ecosystem**
+- `engram gen` emits BOTH `CLAUDE.md` AND `AGENTS.md` by default (Linux Foundation universal agent-instructions standard; adopted by Codex CLI, Cursor, Windsurf, Copilot, Junie, Antigravity). Explicit `--target=claude|cursor|agents` preserves single-file behavior.
+- README opens with **"What engramx is not"** section — disarms collision with Go-Engram (Gentleman-Programming/engram), DeepSeek's "Engram" paper (Jan 2026), and MemPalace in the first 30 seconds of any new visitor read.
+- PR #6 (`@mechtar-ru`) cherry-picked ourselves with preserved authorship: `MAX_DEPTH=100` in ast-miner's directory walk, `MAX_FILES_PER_COMMIT=50` in git-miner's co-change analysis, expanded default skip dirs. Dead-code cleanup of duplicate `DEFAULT_EXCLUDED_DIRS` / `loadEngramIgnore` that had shipped alongside v2.1's newer `DEFAULT_SKIP_DIRS` / `loadIgnorePatterns`. Closes issue #5.
+
+### Proof — real-world benchmark (new, committed)
+
+`bench/real-world.ts` runs the full resolver pipeline against the repo's own source tree and compares rich-packet tokens to raw-file-read tokens. Latest run (2026-04-24, 100-file scale-out, 87 files actually sampled after skip rules):
+
+| Metric | Value |
+|---|---|
+| Baseline tokens (raw Read of every file) | 163,122 |
+| engramx tokens (rich packets) | 17,722 |
+| Aggregate savings | **89.1%** |
+| Median per-file savings | 84.2% |
+| Files where engramx saved tokens | 85 of 87 |
+| Best case (`src/cli.ts`) | 98.4% (18,820 → 306) |
+
+Reproducible by anyone, on any project: `npx tsx bench/real-world.ts --project . --files 50`.
+
+### Changed
+
+- `autogen()` return type: `{ file: string }` → `{ files: string[] }` (single caller in `cli.ts` updated). Consumers of the programmatic API who called `result.file` must read `result.files[0]` instead (or use `--target` to keep single-file semantics).
+- `PROVIDER_PRIORITY` gains `anthropic:memory` at index 3 — downstream test that hard-coded the array order was updated.
+- `MIGRATIONS` (src/db/migrate.ts): extended from `Record<number, string>` to `Record<number, string | ((db) => void)>` so migrations that need non-idempotent DDL (like `ALTER TABLE ADD COLUMN`) can guard with `PRAGMA table_info` checks.
+- README badge updates: tests 640 → 876, providers 8 → 9, savings 88.1% → 90.8%.
+
+### Migration
+
+**v2.1 → v3.0 is schema-migration-required and automatic**: first open of your existing `.engram/graph.db` triggers migration 8. A `.bak-v7` backup is written alongside. Legacy mistake rows survive unchanged (NULL `validUntil` = still valid). Verified on a simulated v2.1 DB during release audit.
+
+**API consumers of `autogen()`** must update call sites: `result.file` (single string) → `result.files` (array). CLI callers are unaffected.
+
+### Tests
+
+771 → 876 passing (+105 new). CI green Ubuntu+Windows × Node 20+22. TypeScript `--noEmit` clean, lint clean.
+
 ## [2.1.0] — 2026-04-21 — "Reliability + Zero-Friction Install"
 
 First release in the v2.1 / v2.2 / v3.0 elevation trilogy. Design spec

CONTRIBUTING.md

@@ -1,6 +1,6 @@
-# Contributing to engram
+# Contributing to EngramX
 
-Thanks for your interest in improving engram. Here's how to help.
+Thanks for wanting to help. EngramX is Apache-2.0, welcomes every kind of contribution, and aims to stay meticulously honest about what works and what doesn't. This doc is short because the rules are few.
 
 ## Quick Start
 
@@ -12,38 +12,74 @@ npm run build
 npm test
 ```
 
-## What's Most Valuable
+You need Node 20+. No native toolchain — EngramX's SQLite is sql.js WASM, all tree-sitter grammars are bundled as WASM. Zero system libraries required.
 
-**Worked examples** are the highest-impact contribution. Run `engram init` on a real codebase, evaluate what the graph got right and wrong, and share the results in an issue or PR.
+## Highest-impact contributions
 
-**Language extraction bugs** — if engram misses a function, class, or import in a language it supports, open an issue with the source file and what was missed.
+In rough order of "what helps the most":
 
-**New language support** — add regex patterns to `src/miners/ast-miner.ts` following the existing pattern. Include test fixtures.
+1. **Worked examples** — run `engram setup` on a real codebase, record what the graph got right and wrong, open an issue. Honest feedback from actual use is more valuable than any patch.
+2. **Reproducible bench results** — run `npx tsx bench/real-world.ts --project . --files 50` on your project and share the numbers (especially if you see <50% savings — we want to understand why).
+3. **Plugin submissions** — a 10-line MCP plugin file for a service we don't have yet. Drop in `docs/plugins/examples/` + mention the coverage in a PR.
+4. **Language extraction bugs** — if `engram init` misses a function/class/import in a supported language, open an issue with the source file and what was missed.
+5. **Windows-specific fixes** — EngramX CI covers Ubuntu × Node 20/22 AND Windows × Node 20/22. Windows-path bugs are real. We welcome patches that harden cross-platform behaviour.
+6. **New language support** — tree-sitter grammar wiring in `src/miners/ast-miner.ts`, plus a test fixture.
 
-## Development
+## Development loop
 
 ```bash
-npm run dev          # Watch mode (auto-rebuild)
-npx vitest           # Run tests in watch mode
-npx vitest run       # Run tests once
-npm run build        # Production build
+npm run dev        # watch mode — rebuilds on every save
+npx vitest         # tests in watch mode
+npx vitest run     # tests once
+npm run build      # production build (bundled WASM grammars)
+npm run lint       # TypeScript strict check (tsc --noEmit)
+npx tsx bench/real-world.ts   # sanity-check your changes against the savings bench
 ```
 
-## Before Submitting a PR
+## Before you open a PR
 
-1. `npm run build` passes
-2. `npx vitest run` passes
-3. If you changed extraction logic, add a test fixture and test case
-4. Keep PRs focused — one change per PR
+1. `npm run build` passes (TypeScript strict).
+2. `npx vitest run` passes all suites (currently 878 on v3.0).
+3. If you changed extraction logic, add a fixture + test case.
+4. **If you touched anything that builds a filesystem path, assert with `path.join()` / `path.resolve()`, never hand-write `/` separators.** We shipped a Windows-CI regression on v3.0's first pass because of this. Tests that build an expected path via `path.join()` (matching the implementation) work on every platform — regex assertions with `\/` do not.
+5. Keep PRs focused — one change per PR.
 
-## Code Style
+## Code style
 
-- TypeScript strict mode
-- ESM imports (`import`, not `require`)
-- Immutable patterns (spread, not mutation)
-- Functions under 50 lines
-- No `console.log` in library code (only in CLI)
+- TypeScript strict mode.
+- ESM imports (`import`, not `require`). Vitest's CommonJS interop hides bare-`require()` bugs that crash in production — always use top-level ESM imports.
+- Immutable patterns (spread, not mutation).
+- Functions under ~50 lines.
+- No `console.log` in library code — only in CLI entry points (`src/cli.ts`) and the bench runner.
+- Every new test that exercises filesystem paths should explicitly include a Windows-native-path case so regressions surface locally, not only on CI.
+
+## Plugin authors
+
+Writing a context provider is ~10 lines. See [`docs/plugins/README.md`](docs/plugins/README.md) for the full spec. Two shapes are supported:
+
+- **MCP-backed** — declare an `mcpConfig` and the loader spawns/connects to the MCP server for you. Any MCP server becomes an EngramX provider in one `.mjs` file.
+- **Classic** — write your own `resolve()` + `isAvailable()` for full control.
+
+Reference examples:
+- [`docs/plugins/examples/serena-plugin.mjs`](docs/plugins/examples/serena-plugin.mjs) — MCP-backed (Serena / LSP symbols)
+- [`docs/plugins/examples/static-context-plugin.mjs`](docs/plugins/examples/static-context-plugin.mjs) — classic (always-on project reminder)
+
+Submitting a plugin into the repo:
+1. Drop your `.mjs` into `docs/plugins/examples/`.
+2. Add a row to the "Plugins multiply the savings" table in `README.md`.
+3. Include a short doc-comment header explaining what gap your plugin closes + install notes.
+
+## Security
+
+- Never commit credentials. The token at `~/.engram/http-server.token` is auto-generated, `.gitignore`d, and never leaves your machine.
+- Found a vulnerability? See [`SECURITY.md`](SECURITY.md) — coordinated disclosure via GitHub advisories, we respond within 48 hours.
+
+## Community
+
+- **Issues** for bugs, feature requests, and plugin submissions.
+- **GitHub Discussions** for "what benchmark are you seeing on your code" and "has anyone built a plugin for X yet."
+- **Security advisories** for anything that could compromise a user's local SQLite.
 
 ## License
 
-By contributing, you agree that your contributions will be licensed under the Apache 2.0 License.
+Apache 2.0. By contributing you agree that your contributions are licensed under the same. See [`LICENSE`](LICENSE) for the full text.