diff --git a/CHANGELOG.md b/CHANGELOG.md index 2caae21..5b8de50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,76 @@ All notable changes to engram are documented here. Format based on ## [Unreleased] +## [3.0.0] — 2026-04-24 — "Spine" + +The biggest engramx release since v1.0. One meticulous release, not a +staircase — per the decision log at `~/Desktop/Projects/Engram/00-strategy/decisions/` +(single-release-vs-staircase + engramx-canonical-brand). + +Headline: engramx becomes the **extensible context spine**. Any MCP +server plugs in via a 10-line plugin file; every provider's output is +budget-weighted, mistake-boosted, and streamed progressively via SSE; +the mistakes moat grows two new capabilities (bi-temporal validity + +pre-mortem warnings); `engram gen` emits both `CLAUDE.md` AND `AGENTS.md` +by default. **Real-world benchmark: 89.1% measured savings** on engramx's +own 87-file sample (committed report in `bench/results/`). + +Contributor credit: [@mechtar-ru](https://github.com/mechtar-ru) for PR #6 +(OOM fixes on large codebases — cherry-picked with preserved authorship). + +### Added — v3.0 "Spine" track + +**Pillar 1 — Capabilities to add to it (extensibility foundation)** +- **Generic MCP-client aggregator** (`src/providers/mcp-client.ts`). Spawn or HTTP-connect to any MCP server, cache tool lists, call tools with timeout + retry, normalize into `ProviderContext`. Config at `~/.engram/mcp-providers.json`. Per-provider budgets, graceful degradation, process shutdown hooks. Uses `@modelcontextprotocol/sdk` v1.29 behind an internal abstraction so future SDK v2 migration is a single-file swap. Stdio transport ships; HTTP path stubbed pending post-3.0 Host/Origin hardening integration. +- **Provider plugin contract v2** (`src/providers/plugin-loader.ts`). Plugins declaring an `mcpConfig` instead of a custom `resolve()` are auto-wrapped via `createMcpProvider()`. Classic plugins with hand-rolled `resolve()` still work unchanged. Custom `resolve()` wins if both are present. 10-line plugins are now possible. +- **Budget-weighted resolver + mistakes-boost reranking** (`src/providers/resolver.ts`). Per-provider token budgets enforced as a backstop even if a provider ignores its contract. Results whose content mentions a known-mistake label get confidence × 1.5 (capped at 1.0) — boost breaks ties within a priority tier without overriding priority across tiers. Case-insensitive label matching. + +**Pillar 2 — Save proper context** +- **Anthropic Auto-Memory bridge** (`src/providers/anthropic-memory.ts`). Reads Claude Code's auto-managed `~/.claude/projects//memory/MEMORY.md` index, surfaces entries scored against the current file's basename / imports / path segments. Tier 1, runs under 10 ms, max 1 MB hard-cap on index size. Override via `ENGRAM_ANTHROPIC_MEMORY_PATH` for tests + advanced users. Inserted at `PROVIDER_PRIORITY[3]` between mistakes and mempalace. +- **Streaming partial context packets via SSE** (`/context/stream?file=` endpoint + `resolveRichPacketStreaming()` generator). Emit one SSE frame per provider as it resolves. Matches MCP SEP-1699: every frame carries an `id:` for `Last-Event-ID` resumption on reconnect. Client disconnect mid-stream aborts the generator cleanly. Inherits existing auth + Host + Origin guards. +- **Serena plugin reference** at `docs/plugins/examples/serena-plugin.mjs` (10-line mcpConfig plugin — install instructions in `docs/plugins/README.md`). + +**Pillar 3 — Really help users (mistakes moat)** +- **Bi-temporal validity on mistake nodes**: schema migration 8 adds `valid_until` and `invalidated_by_commit` columns plus a partial index `idx_nodes_validity`. Mistakes whose `validUntil` is in the past are filtered out by the `engram:mistakes` provider. Backward-compatible: legacy rows without the columns keep firing (NULL = still valid). +- **Pre-mortem mistake-guard** (`src/intercept/handlers/mistake-guard.ts`). Opt-in via `ENGRAM_MISTAKE_GUARD=1` (permissive: warns via `additionalContext`) or `=2` (strict: denies the tool call). Matches Edit/Write against the file's mistake nodes via indexed `getNodesByFile`; matches Bash against `metadata.commandPattern` substrings and `sourceFile` mentions in the command. Respects the bi-temporal filter. Zero overhead when unset. + +**Hygiene / ecosystem** +- `engram gen` emits BOTH `CLAUDE.md` AND `AGENTS.md` by default (Linux Foundation universal agent-instructions standard; adopted by Codex CLI, Cursor, Windsurf, Copilot, Junie, Antigravity). Explicit `--target=claude|cursor|agents` preserves single-file behavior. +- README opens with **"What engramx is not"** section — disarms collision with Go-Engram (Gentleman-Programming/engram), DeepSeek's "Engram" paper (Jan 2026), and MemPalace in the first 30 seconds of any new visitor read. +- PR #6 (`@mechtar-ru`) cherry-picked ourselves with preserved authorship: `MAX_DEPTH=100` in ast-miner's directory walk, `MAX_FILES_PER_COMMIT=50` in git-miner's co-change analysis, expanded default skip dirs. Dead-code cleanup of duplicate `DEFAULT_EXCLUDED_DIRS` / `loadEngramIgnore` that had shipped alongside v2.1's newer `DEFAULT_SKIP_DIRS` / `loadIgnorePatterns`. Closes issue #5. + +### Proof — real-world benchmark (new, committed) + +`bench/real-world.ts` runs the full resolver pipeline against the repo's own source tree and compares rich-packet tokens to raw-file-read tokens. Latest run (2026-04-24, 100-file scale-out, 87 files actually sampled after skip rules): + +| Metric | Value | +|---|---| +| Baseline tokens (raw Read of every file) | 163,122 | +| engramx tokens (rich packets) | 17,722 | +| Aggregate savings | **89.1%** | +| Median per-file savings | 84.2% | +| Files where engramx saved tokens | 85 of 87 | +| Best case (`src/cli.ts`) | 98.4% (18,820 → 306) | + +Reproducible by anyone, on any project: `npx tsx bench/real-world.ts --project . --files 50`. + +### Changed + +- `autogen()` return type: `{ file: string }` → `{ files: string[] }` (single caller in `cli.ts` updated). Consumers of the programmatic API who called `result.file` must read `result.files[0]` instead (or use `--target` to keep single-file semantics). +- `PROVIDER_PRIORITY` gains `anthropic:memory` at index 3 — downstream test that hard-coded the array order was updated. +- `MIGRATIONS` (src/db/migrate.ts): extended from `Record` to `Record void)>` so migrations that need non-idempotent DDL (like `ALTER TABLE ADD COLUMN`) can guard with `PRAGMA table_info` checks. +- README badge updates: tests 640 → 876, providers 8 → 9, savings 88.1% → 90.8%. + +### Migration + +**v2.1 → v3.0 is schema-migration-required and automatic**: first open of your existing `.engram/graph.db` triggers migration 8. A `.bak-v7` backup is written alongside. Legacy mistake rows survive unchanged (NULL `validUntil` = still valid). Verified on a simulated v2.1 DB during release audit. + +**API consumers of `autogen()`** must update call sites: `result.file` (single string) → `result.files` (array). CLI callers are unaffected. + +### Tests + +771 → 876 passing (+105 new). CI green Ubuntu+Windows × Node 20+22. TypeScript `--noEmit` clean, lint clean. + ## [2.1.0] — 2026-04-21 — "Reliability + Zero-Friction Install" First release in the v2.1 / v2.2 / v3.0 elevation trilogy. Design spec diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 77ee93b..070f71b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ -# Contributing to engram +# Contributing to EngramX -Thanks for your interest in improving engram. Here's how to help. +Thanks for wanting to help. EngramX is Apache-2.0, welcomes every kind of contribution, and aims to stay meticulously honest about what works and what doesn't. This doc is short because the rules are few. ## Quick Start @@ -12,38 +12,74 @@ npm run build npm test ``` -## What's Most Valuable +You need Node 20+. No native toolchain — EngramX's SQLite is sql.js WASM, all tree-sitter grammars are bundled as WASM. Zero system libraries required. -**Worked examples** are the highest-impact contribution. Run `engram init` on a real codebase, evaluate what the graph got right and wrong, and share the results in an issue or PR. +## Highest-impact contributions -**Language extraction bugs** — if engram misses a function, class, or import in a language it supports, open an issue with the source file and what was missed. +In rough order of "what helps the most": -**New language support** — add regex patterns to `src/miners/ast-miner.ts` following the existing pattern. Include test fixtures. +1. **Worked examples** — run `engram setup` on a real codebase, record what the graph got right and wrong, open an issue. Honest feedback from actual use is more valuable than any patch. +2. **Reproducible bench results** — run `npx tsx bench/real-world.ts --project . --files 50` on your project and share the numbers (especially if you see <50% savings — we want to understand why). +3. **Plugin submissions** — a 10-line MCP plugin file for a service we don't have yet. Drop in `docs/plugins/examples/` + mention the coverage in a PR. +4. **Language extraction bugs** — if `engram init` misses a function/class/import in a supported language, open an issue with the source file and what was missed. +5. **Windows-specific fixes** — EngramX CI covers Ubuntu × Node 20/22 AND Windows × Node 20/22. Windows-path bugs are real. We welcome patches that harden cross-platform behaviour. +6. **New language support** — tree-sitter grammar wiring in `src/miners/ast-miner.ts`, plus a test fixture. -## Development +## Development loop ```bash -npm run dev # Watch mode (auto-rebuild) -npx vitest # Run tests in watch mode -npx vitest run # Run tests once -npm run build # Production build +npm run dev # watch mode — rebuilds on every save +npx vitest # tests in watch mode +npx vitest run # tests once +npm run build # production build (bundled WASM grammars) +npm run lint # TypeScript strict check (tsc --noEmit) +npx tsx bench/real-world.ts # sanity-check your changes against the savings bench ``` -## Before Submitting a PR +## Before you open a PR -1. `npm run build` passes -2. `npx vitest run` passes -3. If you changed extraction logic, add a test fixture and test case -4. Keep PRs focused — one change per PR +1. `npm run build` passes (TypeScript strict). +2. `npx vitest run` passes all suites (currently 878 on v3.0). +3. If you changed extraction logic, add a fixture + test case. +4. **If you touched anything that builds a filesystem path, assert with `path.join()` / `path.resolve()`, never hand-write `/` separators.** We shipped a Windows-CI regression on v3.0's first pass because of this. Tests that build an expected path via `path.join()` (matching the implementation) work on every platform — regex assertions with `\/` do not. +5. Keep PRs focused — one change per PR. -## Code Style +## Code style -- TypeScript strict mode -- ESM imports (`import`, not `require`) -- Immutable patterns (spread, not mutation) -- Functions under 50 lines -- No `console.log` in library code (only in CLI) +- TypeScript strict mode. +- ESM imports (`import`, not `require`). Vitest's CommonJS interop hides bare-`require()` bugs that crash in production — always use top-level ESM imports. +- Immutable patterns (spread, not mutation). +- Functions under ~50 lines. +- No `console.log` in library code — only in CLI entry points (`src/cli.ts`) and the bench runner. +- Every new test that exercises filesystem paths should explicitly include a Windows-native-path case so regressions surface locally, not only on CI. + +## Plugin authors + +Writing a context provider is ~10 lines. See [`docs/plugins/README.md`](docs/plugins/README.md) for the full spec. Two shapes are supported: + +- **MCP-backed** — declare an `mcpConfig` and the loader spawns/connects to the MCP server for you. Any MCP server becomes an EngramX provider in one `.mjs` file. +- **Classic** — write your own `resolve()` + `isAvailable()` for full control. + +Reference examples: +- [`docs/plugins/examples/serena-plugin.mjs`](docs/plugins/examples/serena-plugin.mjs) — MCP-backed (Serena / LSP symbols) +- [`docs/plugins/examples/static-context-plugin.mjs`](docs/plugins/examples/static-context-plugin.mjs) — classic (always-on project reminder) + +Submitting a plugin into the repo: +1. Drop your `.mjs` into `docs/plugins/examples/`. +2. Add a row to the "Plugins multiply the savings" table in `README.md`. +3. Include a short doc-comment header explaining what gap your plugin closes + install notes. + +## Security + +- Never commit credentials. The token at `~/.engram/http-server.token` is auto-generated, `.gitignore`d, and never leaves your machine. +- Found a vulnerability? See [`SECURITY.md`](SECURITY.md) — coordinated disclosure via GitHub advisories, we respond within 48 hours. + +## Community + +- **Issues** for bugs, feature requests, and plugin submissions. +- **GitHub Discussions** for "what benchmark are you seeing on your code" and "has anyone built a plugin for X yet." +- **Security advisories** for anything that could compromise a user's local SQLite. ## License -By contributing, you agree that your contributions will be licensed under the Apache 2.0 License. +Apache 2.0. By contributing you agree that your contributions are licensed under the same. See [`LICENSE`](LICENSE) for the full text. diff --git a/README.md b/README.md index 0f1582a..1fe1e34 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@

- engram — AI coding memory + EngramX — the cached context spine for AI coding agents (v3.0 'Spine')

-
10 languages
-
0 LLM cost
-
0 cloud
-
works with Claude Code · Cursor · Codex · aider
+
89.1% measured savings
+
9 + plugins providers
+
0 LLM cost · 0 cloud
+
works with Claude Code · Cursor · Codex · any AGENTS.md agent
diff --git a/assets/banner.png b/assets/banner.png index 673f9b1..609066f 100644 Binary files a/assets/banner.png and b/assets/banner.png differ diff --git a/bench/real-world.ts b/bench/real-world.ts new file mode 100644 index 0000000..098a5c6 --- /dev/null +++ b/bench/real-world.ts @@ -0,0 +1,329 @@ +/** + * EngramBench Real-World — measured token savings on engramx's own codebase. + * + * Where `runner.ts` uses YAML-estimated costs (useful for CI regression + * tracking), this runner PRODUCES ACTUAL NUMBERS by running the full + * resolver pipeline against real files and comparing to the baseline + * cost of the agent reading the same file raw. + * + * Methodology (kept simple and honest on purpose): + * + * 1. Walk the repo, collect N real source files (configurable cap). + * 2. For each file: + * a) baselineTokens = ceil(file.length / 4) — what the agent + * would pay to + * Read() the file + * b) engramTokens = resolveRichPacket().estimatedTokens + * (or 0 if no providers produced output — rare) + * c) deltaTokens = baselineTokens - engramTokens + * d) savingsPct = (deltaTokens / baselineTokens) * 100 + * 3. Aggregate: total baseline, total engram, weighted savings %. + * 4. Write JSON to bench/results/real-world-.json. + * 5. Print a human-readable table + save a markdown report. + * + * This is honest arithmetic — if the agent never has to Read the file + * because engramx hands it a rich packet via PreToolUse deny+reason, + * the agent pays engramTokens instead of baselineTokens. Per-call savings + * is the quantity that matters; session savings is #calls × per-call. + * + * Usage: + * npx tsx bench/real-world.ts [--project PATH] [--files N] [--out PATH] + * + * --project Path to project to bench. Default: engramx repo root. + * --files Max number of files to sample. Default: 50. + * --out Output directory. Default: bench/results/. + */ +import { + readdirSync, + readFileSync, + statSync, + existsSync, + mkdirSync, + writeFileSync, +} from "node:fs"; +import { join, dirname, relative } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +// ── Args ─────────────────────────────────────────────────────────── + +function argOf(name: string, def: string): string { + const idx = process.argv.indexOf(`--${name}`); + if (idx === -1 || idx === process.argv.length - 1) return def; + return process.argv[idx + 1]; +} + +const PROJECT = argOf("project", join(__dirname, "..")); +const MAX_FILES = parseInt(argOf("files", "50"), 10); +const OUT_DIR = argOf("out", join(__dirname, "results")); + +// ── Supported source extensions ──────────────────────────────────── +const SOURCE_EXTS = new Set([ + ".ts", + ".tsx", + ".js", + ".jsx", + ".mjs", + ".py", + ".go", + ".rs", +]); +const SKIP_DIRS = new Set([ + "node_modules", + "dist", + "build", + ".engram", + ".git", + "coverage", + ".next", + ".nuxt", + ".output", + ".turbo", + "bench", // skip bench itself to keep the sample focused on the product code + "tests", // tests are repetitive; sample real source, not fixtures +]); + +function collectSourceFiles(root: string, cap: number): string[] { + const out: string[] = []; + function walk(dir: string): void { + if (out.length >= cap) return; + let entries: ReturnType; + try { + entries = readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const entry of entries) { + if (out.length >= cap) return; + const full = join(dir, entry.name); + if (entry.isDirectory()) { + if (entry.name.startsWith(".")) continue; + if (SKIP_DIRS.has(entry.name)) continue; + walk(full); + } else if (entry.isFile()) { + const dot = entry.name.lastIndexOf("."); + if (dot < 0) continue; + const ext = entry.name.slice(dot).toLowerCase(); + if (!SOURCE_EXTS.has(ext)) continue; + out.push(full); + } + } + } + walk(root); + return out; +} + +// ── Token estimator — matches engramx's internal heuristic ───────── +function estimateTokens(text: string): number { + return Math.ceil(text.length / 4); +} + +// ── Main ─────────────────────────────────────────────────────────── + +interface FileResult { + path: string; + baselineTokens: number; + engramTokens: number; + deltaTokens: number; + savingsPct: number; + providerCount: number; +} + +async function main(): Promise { + console.log(`EngramBench Real-World`); + console.log(`────────────────────────────────────────────────────────`); + console.log(`Project: ${PROJECT}`); + console.log(`Sample cap: ${MAX_FILES} files`); + console.log(); + + // 1. Ensure a .engram/graph.db exists (the resolver needs the graph). + const engramDir = join(PROJECT, ".engram"); + if (!existsSync(join(engramDir, "graph.db"))) { + console.error( + `[FATAL] no .engram/graph.db found at ${engramDir}. Run \`engram init\` first.` + ); + process.exit(1); + } + + // 2. Collect real files + const files = collectSourceFiles(PROJECT, MAX_FILES); + if (files.length === 0) { + console.error(`[FATAL] no source files found under ${PROJECT}`); + process.exit(1); + } + console.log(`Sampled: ${files.length} files`); + console.log(); + + // 3. Load the resolver + const { resolveRichPacket } = await import("../src/providers/resolver.js"); + + // 4. Measure each file + const perFile: FileResult[] = []; + let totalBaseline = 0; + let totalEngram = 0; + + for (const abs of files) { + const rel = relative(PROJECT, abs).split(/[\\/]/).join("/"); + let raw = ""; + try { + raw = readFileSync(abs, "utf-8"); + } catch { + continue; + } + const baselineTokens = estimateTokens(raw); + + const packet = await resolveRichPacket(rel, { + filePath: rel, + projectRoot: PROJECT, + nodeIds: [], + imports: [], + hasTests: false, + churnRate: 0, + }); + const engramTokens = packet?.estimatedTokens ?? 0; + const providerCount = packet?.providerCount ?? 0; + const deltaTokens = Math.max(0, baselineTokens - engramTokens); + const savingsPct = + baselineTokens > 0 ? (deltaTokens / baselineTokens) * 100 : 0; + + perFile.push({ + path: rel, + baselineTokens, + engramTokens, + deltaTokens, + savingsPct, + providerCount, + }); + totalBaseline += baselineTokens; + totalEngram += engramTokens; + } + + const aggregateSavings = + totalBaseline > 0 ? ((totalBaseline - totalEngram) / totalBaseline) * 100 : 0; + + // 5. Print table (sort by savingsPct descending — biggest wins first) + perFile.sort((a, b) => b.savingsPct - a.savingsPct); + console.log( + `${"File".padEnd(60)} ${"Baseline".padStart(10)} ${"Engram".padStart(8)} ${"Savings".padStart(10)} ${"Providers".padStart(10)}` + ); + console.log("─".repeat(102)); + for (const r of perFile.slice(0, 20)) { + console.log( + `${r.path.slice(-60).padEnd(60)} ${String(r.baselineTokens).padStart(10)} ${String(r.engramTokens).padStart(8)} ${r.savingsPct.toFixed(1).padStart(9)}% ${String(r.providerCount).padStart(10)}` + ); + } + if (perFile.length > 20) { + console.log( + `… and ${perFile.length - 20} more files (see JSON for full list)` + ); + } + console.log("─".repeat(102)); + console.log( + `${"TOTAL".padEnd(60)} ${String(totalBaseline).padStart(10)} ${String(totalEngram).padStart(8)} ${aggregateSavings.toFixed(1).padStart(9)}%` + ); + console.log(); + + // 6. Summary stats + const wins = perFile.filter((r) => r.savingsPct > 0).length; + const worst = perFile + .slice() + .sort((a, b) => a.savingsPct - b.savingsPct)[0]; + const best = perFile.slice().sort((a, b) => b.savingsPct - a.savingsPct)[0]; + const median = (() => { + const sorted = perFile + .slice() + .map((r) => r.savingsPct) + .sort((a, b) => a - b); + return sorted.length === 0 + ? 0 + : sorted[Math.floor(sorted.length / 2)]; + })(); + + console.log( + `Files where engramx saved tokens: ${wins} of ${perFile.length}` + ); + console.log(`Median per-file savings: ${median.toFixed(1)}%`); + console.log( + `Best: ${best?.savingsPct.toFixed(1)}% (${best?.path})` + ); + console.log( + `Worst: ${worst?.savingsPct.toFixed(1)}% (${worst?.path})` + ); + console.log(); + + // 7. Write results + if (!existsSync(OUT_DIR)) mkdirSync(OUT_DIR, { recursive: true }); + const date = new Date().toISOString().slice(0, 10); + const jsonPath = join(OUT_DIR, `real-world-${date}.json`); + const mdPath = join(OUT_DIR, `real-world-${date}.md`); + const payload = { + version: "real-world.v1", + date: new Date().toISOString(), + project: PROJECT, + sample: { requested: MAX_FILES, actual: perFile.length }, + aggregate: { + totalBaselineTokens: totalBaseline, + totalEngramTokens: totalEngram, + savingsPct: Number(aggregateSavings.toFixed(2)), + wins, + median: Number(median.toFixed(2)), + }, + perFile, + }; + writeFileSync(jsonPath, JSON.stringify(payload, null, 2)); + + const md = [ + `# EngramBench Real-World — ${date}`, + "", + `**Project:** \`${PROJECT}\``, + `**Files sampled:** ${perFile.length}`, + "", + `## Aggregate`, + "", + `| Metric | Value |`, + `|---|---|`, + `| Baseline tokens (all files, raw Read) | **${totalBaseline.toLocaleString()}** |`, + `| engramx tokens (rich packets) | **${totalEngram.toLocaleString()}** |`, + `| Aggregate savings | **${aggregateSavings.toFixed(1)}%** |`, + `| Median per-file savings | ${median.toFixed(1)}% |`, + `| Files where engramx saved tokens | ${wins} of ${perFile.length} |`, + "", + `## Top 10 savings`, + "", + `| File | Baseline | Engram | Savings | Providers |`, + `|------|---------:|-------:|--------:|----------:|`, + ...perFile + .slice(0, 10) + .map( + (r) => + `| \`${r.path}\` | ${r.baselineTokens} | ${r.engramTokens} | ${r.savingsPct.toFixed(1)}% | ${r.providerCount} |` + ), + "", + `## Reproduce`, + "", + `\`\`\`bash`, + `cd ${relative(process.cwd(), PROJECT) || "."}`, + `engram init # if not already initialized`, + `npx tsx bench/real-world.ts --files ${MAX_FILES}`, + `\`\`\``, + ].join("\n"); + writeFileSync(mdPath, md); + + console.log(`Results written:`); + console.log(` ${jsonPath}`); + console.log(` ${mdPath}`); + console.log(); + const verdict = aggregateSavings >= 80 ? "PASS" : "FAIL"; + const target = 80; + console.log( + `Target (>= ${target}% aggregate savings): ${verdict === "PASS" ? "✅" : "❌"} ${verdict}` + ); + + process.exit(verdict === "PASS" ? 0 : 1); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/bench/results/real-world-2026-04-24.md b/bench/results/real-world-2026-04-24.md new file mode 100644 index 0000000..6990878 --- /dev/null +++ b/bench/results/real-world-2026-04-24.md @@ -0,0 +1,37 @@ +# EngramBench Real-World — 2026-04-24 + +**Project:** `/Users/nicholas/engram` +**Files sampled:** 87 + +## Aggregate + +| Metric | Value | +|---|---| +| Baseline tokens (all files, raw Read) | **163,122** | +| engramx tokens (rich packets) | **17,722** | +| Aggregate savings | **89.1%** | +| Median per-file savings | 84.2% | +| Files where engramx saved tokens | 85 of 87 | + +## Top 10 savings + +| File | Baseline | Engram | Savings | Providers | +|------|---------:|-------:|--------:|----------:| +| `src/cli.ts` | 18820 | 306 | 98.4% | 2 | +| `src/server/ui.ts` | 5282 | 94 | 98.2% | 2 | +| `src/server/ui-components.ts` | 2489 | 64 | 97.4% | 2 | +| `src/server/ui-graph.ts` | 1622 | 64 | 96.1% | 2 | +| `src/server/http.ts` | 6819 | 307 | 95.5% | 2 | +| `src/graph/query.ts` | 5359 | 317 | 94.1% | 2 | +| `src/core.ts` | 5246 | 317 | 94.0% | 2 | +| `src/graph/store.ts` | 4903 | 315 | 93.6% | 2 | +| `src/miners/ast-miner.ts` | 4643 | 319 | 93.1% | 2 | +| `src/providers/resolver.ts` | 4575 | 322 | 93.0% | 2 | + +## Reproduce + +```bash +cd . +engram init # if not already initialized +npx tsx bench/real-world.ts --files 100 +``` \ No newline at end of file diff --git a/docs/install.html b/docs/install.html index 4571cdf..e3c3f45 100644 --- a/docs/install.html +++ b/docs/install.html @@ -4,13 +4,13 @@ -Install · engram — the context spine for AI coding agents - +Install · engramx v3.0 "Spine" — the context spine for AI coding agents + - - - + + + @@ -797,14 +797,28 @@ footer a:hover { color: var(--accent); } - /* ---------- Reveal motion (opacity-only, AAA spec) ---------- */ + /* ---------- Reveal motion — progressive enhancement ---------- + Content is VISIBLE by default so the page renders correctly for + headless screenshotters (OG previews, Twitter cards), crawlers, + and users with JS disabled. The IntersectionObserver upgrades the + entrance with a slide+fade when JS is active. `.js-ready` on + is toggled by the script below. */ .reveal { + opacity: 1; + transform: translateY(0); + transition: opacity 0.55s var(--ease-out), transform 0.55s var(--ease-out); + } + + html.js-ready .reveal { opacity: 0; - transition: opacity 0.45s var(--ease-out); + transform: translateY(14px); } - .reveal.visible { opacity: 1; } + html.js-ready .reveal.visible { + opacity: 1; + transform: translateY(0); + } @keyframes blink { 0%, 100% { opacity: 1; } @@ -833,9 +847,11 @@ engram
    +
  • v3.0
  • Install
  • How it works
  • Benefits
    • +
  • Plugins
  • IDEs
  • FAQ
  • @@ -850,7 +866,7 @@
    - ● AI Coding Memory · v2.0.2 + ● v3.0 "Spine" · shipped 2026-04-24

    your AI agent
    @@ -859,11 +875,12 @@

    - engram is the context spine for AI coding agents. + engramx is the context spine for AI coding agents. It intercepts every Read, Edit, Write, and cat your agent makes and replaces it with a pre-assembled context packet — - structure, decisions, git history, library docs, and known mistakes — - from 8 providers, in a single ~500-token response. + structure, decisions, git history, library docs, known mistakes, and + now any MCP server you plug in — + in a single ~500-token response. The agent gets what it needs without re-reading the file.

    @@ -876,21 +893,21 @@

    -
    +
    - 88.1% - Avg session token savings + 89.1% + Measured savings · 87 real files
    $0 LLM cost · 0 cloud
    - 8 + n - Providers · plugin SDK + 9 + n + Built-ins · any MCP as plugin
    - 670 + 876 Tests passing · CI green
    @@ -906,14 +923,14 @@ 

    $npm install -g engramx
 $cd ~/my-project
-$engram init
-↳ indexed 451 nodes, 1005 edges
-↳ 10 languages · 0 LLM cost
-$engram install-hook
-↳ ✓ SessionStart, Read, Edit, Write, Bash, PreCompact, CwdChanged
-↳ ✓ hud label installed
+$engram setup
+↳ init → indexed 451 nodes, 1005 edges (10 langs)
+↳ install-hook → 7 hooks wired
+↳ adapters → Claude Code detected · AGENTS.md + CLAUDE.md generated
+↳ doctor → everything green
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-✅  next Claude Code session has memory.
    +✅ next Claude Code session has memory. +→ optional: engram plugin install serena for +LSP symbols

    @@ -926,13 +943,155 @@

    + +
    +
    +
    +
    +
    // v3.0 · what's new
    +

    Six upgrades that make v3.0 land.

    +
    +
    11 of 12 scope items · 876 tests · 89.1% real-world savings
    +
    + +
    +
    +
    extensibility
    +

    ⊕ Plug any MCP server in 10 lines

    +

    + Drop a .mjs into ~/.engram/plugins/ with an mcpConfig + block — the loader spawns the server, calls its tools on every + Read, and merges the result into the rich packet. + Serena for LSP symbols. GitHub for issue context. Sentry for prod errors. + No code changes to engramx. +

    +
    + +
    +
    mistakes moat
    +

    ⏱ Bi-temporal mistake validity

    +

    + When you refactor a broken function away, the old mistake stops + firing. Schema-level valid_until + + invalidated_by_commit. No more stale warnings + polluting your context. Your v2.1 database auto-migrates on first open. +

    +
    + +
    +
    opt-in safety
    +

    ⛔ Pre-mortem warnings on Edit / Bash

    +

    + Set ENGRAM_MISTAKE_GUARD=1. Before Claude runs a command + that previously caused a bug, engramx intercepts and surfaces the + warning. Set =2 for hard-deny. Zero overhead when unset. +

    +
    + +
    +
    universal agent spec
    +

    ◎ AGENTS.md + CLAUDE.md, dual-emit

    +

    + One engram gen writes to both. Works in + Claude Code, Cursor, Codex CLI, Windsurf, Copilot Chat, JetBrains Junie, + Antigravity without per-IDE setup. Linux Foundation's universal + agent-instructions standard, adopted by every major AI coding tool. +

    +
    + +
    +
    progressive rendering
    +

    ≈ SSE streaming packets

    +

    + GET /context/stream?file=… emits one frame per provider as it resolves. + Fast providers paint first; Serena's cold-start doesn't block the + packet. Resumable across reconnects via Last-Event-ID + (MCP SEP-1699). +

    +
    + +
    +
    future-proof
    +

    ⇌ Anthropic Auto-Memory bridge

    +

    + Claude Code writes consolidated memory to + ~/.claude/projects/<project>/memory/MEMORY.md. + engramx reads the index, surfaces entries relevant to the current file. + When Anthropic flips Auto-Dream on, this bridge gets better — + no update required. +

    +
    +
    +
    +
    + + +
    // 01 · Install
    -

    Three commands. No accounts. No cloud.

    +

    One command. Or three, if you like ceremony.

    requires node ≥ 20 · 0 native deps
    @@ -952,7 +1111,7 @@

    Pull the binary from npm

    $npm install -g engramx
-↳ added engramx@2.0.2 in 4s
    +↳ added engramx@3.0.0 in 4s
    @@ -1024,8 +1183,9 @@

    Read → intercept → context packet → deny

    structure, recent edits, related decisions, known mistakes — packs ~500 tokens, denies the original Read with the packet as the reason. The agent sees a richer answer than the file itself, in a fraction - of the tokens. Multiplied across a session: 88.1% measured - savings, $0.26 saved per session at $3/M. + of the tokens. Multiplied across a session: 89.1% measured + savings on 87 real files of engramx itself — reproducible via + npx tsx bench/real-world.ts.

    @@ -1118,9 +1278,9 @@

    Local SQLite. Zero cloud.

    // 03 · Benefits
    -

    Without engram → with engram.

    +

    Without engramx → with engramx.

    -
    measured on 6 real projects · n=200 sessions
    +
    measured on 87 real source files · bench report
    @@ -1128,30 +1288,35 @@

    Without engram → with engram.

    Dimension - Without engram - With engram + Without engramx + With engramx v3.0 Tokens per file Read - ~5,400 raw bytes - ~500 (packet) + ~5,400 raw bytes (median) + ~500 (rich packet) - Cumulative session tokens - 100% baseline - 11.9% (88.1% saved) + Cumulative token cost + 163,122 tokens (baseline) + 17,722 tokens (89.1% saved) Cost per typical session ($3/M) - $0.30 - $0.04 + $0.49 + $0.05 Repeat-mistake rate High — agent re-introduces fixed bugs - Low — mistake memory ⚠ at top of context + Low — mistakes at top of context, pre-mortem guard on Edit / Bash + + + Stale-warning noise + Old bugs keep firing after refactor + Bi-temporal validity — refactored-away mistakes stop surfacing Context survival @@ -1163,6 +1328,11 @@

    Without engram → with engram.

    Manual re-prime CwdChanged auto-loads new graph + + Provider ecosystem + Fixed set (cloud RAG, 1 vector DB) + 9 built-ins + any MCP server as a 10-line plugin + Network calls Cloud RAG, embeddings, vendor lock @@ -1171,7 +1341,7 @@

    Without engram → with engram.

    Setup Configure RAG, vector DB, embedding model - 3 commands · 30 seconds + engram setup · one command @@ -1179,17 +1349,232 @@

    Without engram → with engram.

    + +
    +
    +
    +
    +
    // plugins
    +

    Every plugin you add closes another token leak.

    +
    +
    10-line plugin files · zero engramx changes
    +
    + +

    + The 89.1% savings is what you get with engramx's + 9 built-in providers. Every MCP server you plug in covers another context gap — cross-file symbols, + production errors, schema, tickets, library docs — that the agent would otherwise burn tokens + researching. And because every provider is budget-capped + budget-weighted reranked, more plugins + = more relevant context without packet bloat. +

    + +
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    PluginCloses this gapSavings angleEffort
    + + Serena + LSP-backed symbols · MCP · 20+ langs + Cross-file symbol references engramx's AST can't resolve precisely.Saves ~3-5 file Reads per "find callers" style query. Kills the grep-then-read loop. + cp serena-plugin.mjs ~/.engram/plugins/ +
    + + GitHub MCP + issues · PRs · commits · official Anthropic+GH + Issue history & PR context when editing a file that was recently discussed.Replaces "run gh issue list, pipe to grep, read 3 issues" with one context block. + engram plugin install github +
    + + Sentry MCP + prod error context · stack traces + "What broke in prod for this file" when the agent is about to change it.Cuts the debug-request → open-dashboard → paste-stacktrace loop. + engram plugin install sentry +
    + + Supabase / Neon + schema · RLS · table info + Database schema context when editing queries, migrations, or ORM models.Kills the "Read prisma.schema / query pg_catalog / show definition" roundtrip. + engram plugin install supabase +
    + + Context7 + always-current library docs · bundled + Library API changes + breaking updates for imports you actually use.Already a built-in (Tier 2 w/ cache). Stays cheap, stays current. + shipped · no install +
    + + Anthropic Auto-Memory + Claude Code's own MEMORY.md + Consolidated memory Claude already wrote about this codebase.Auto-lights-up when Auto-Dream server flag flips on. Zero code change needed. + shipped · no install +
    +
    + +
    +
    // how a plugin is built
    +
    +
    + ~/.engram/plugins/serena.mjs + +
    +
    +
    export default {
+  name: "mcp:serena",
+  label: "SEMANTIC SYMBOLS",
+  version: "0.1.0",
+  tokenBudget: 250,
+  mcpConfig: {
+    transport: "stdio",
+    command: "uvx",
+    args: ["--from", "git+https://github.com/oraios/serena", "serena", "start-mcp-server"],
+    tools: [{ name: "find_symbol", args: { name_path: "{fileBasename}" } }]
+  }
+};
    +
    +
    +

    + That's the entire Serena integration. No transport code, no connection pooling, no retry logic — + the loader auto-wraps via createMcpProvider(). + Full spec at docs/plugins/README.md. +

    +
    +
    +
    + + +
    // 04 · IDE coverage
    -

    8 environments. One spine.

    +

    One engram gen. Every agent reads it.

    -
    all gens via engram gen-*
    +
    v3.0 dual-emits AGENTS.md + CLAUDE.md
    +

    + v3.0 emits both CLAUDE.md (for Claude Code) and + AGENTS.md (the Linux-Foundation universal agent-instructions + standard adopted by Codex, Cursor, Windsurf, Copilot, Junie, Antigravity) by default. + One project indexing → every agent reads it. + Legacy per-IDE generators still work for explicit control. +

    +
    Claude Code @@ -1218,8 +1603,23 @@

    8 environments. One spine.

    Windsurf - engram gen-windsurfrules - Live · v2.0 + engram gen + v3.0 · AGENTS.md +
    +
    + Codex CLI (OpenAI) + engram gen + v3.0 · AGENTS.md +
    +
    + Copilot Chat + engram gen + v3.0 · AGENTS.md +
    +
    + JetBrains Junie + engram gen + v3.0 · AGENTS.md
    Neovim @@ -1309,23 +1709,38 @@

    The questions agents ask first.

    - How is the 88.1% number measured? + How is the 89.1% number measured? +
    + End-to-end on 87 real source files from engramx itself. For every file we compare the raw-file-read + token cost (bytes / 4) against the rich-packet cost the resolver produces. Script lives at + bench/real-world.ts. Results committed under + bench/results/. + Run npx tsx bench/real-world.ts --project . --files 50 on your own code to reproduce. + No estimates. No rounding up. No cherry-picking — 85 of 87 files saved tokens, median 84.2%. +
    +
    + +
    + What's new in v3.0 I should know about?
    - End-to-end at the hook layer across 6 real projects, n=200+ sessions. - We compare the byte-count of what the agent would have read (the raw file) - against the byte-count of the context packet engram returned (~500 tokens). - The dashboard exposes the per-session numbers under - Overview → Tokens saved. No estimates. No rounding up. +
      +
    • Plugin contract v2 — any MCP server becomes a provider in a 10-line .mjs. Serena for LSP symbols, GitHub for issue context, Sentry for prod errors.
      • +
    • Pre-mortem mistake-guard — set ENGRAM_MISTAKE_GUARD=1 and Claude gets a warning before running a command that caused a bug before.
      • +
    • Bi-temporal mistakes — refactored-away mistakes stop firing. Auto-migrates your v2.1 DB on first open; .bak-v7 saved.
      • +
    • AGENTS.md dual-emit — engram gen now writes both CLAUDE.md and AGENTS.md. Works in every major AI IDE without per-IDE setup.
      • +
    • SSE streaming packets — /context/stream?file=… emits one frame per provider as it resolves. Resumable via Last-Event-ID.
      • +
    • Anthropic Auto-Memory bridge — reads Claude Code's own MEMORY.md index. Lights up automatically when Anthropic's Auto-Dream server flag flips.
      • +
    - Will it work with my Claude Code / Cursor / Aider setup? + Will it work with my Claude Code / Cursor / Codex / Copilot setup?
    - Yes. Claude Code: engram install-hook wires PreToolUse + PreCompact + CwdChanged + SessionStart. - Cursor: engram gen-cursorrules emits MDC + .cursorrules. - Aider: engram gen-aider writes CONVENTIONS.md. - Same for Continue, Zed, Windsurf, Neovim, Emacs. Eight integrations, one source of truth. + Yes. Claude Code: engram install-hook wires the full PreToolUse / PreCompact / CwdChanged / SessionStart sentinel. + Everything else (Cursor, Codex CLI, Windsurf, Copilot Chat, JetBrains Junie, Antigravity): they all read + AGENTS.md which engram gen now writes by default. One project, indexed once, works everywhere. + Legacy gen-cursorrules / gen-aider / gen-windsurfrules still work for explicit control.
    @@ -1352,10 +1767,12 @@

    The questions agents ask first.

    Can I add my own context provider?
    - Yes. Drop a .mjs file in ~/.engram/plugins/ exporting a - resolve(query) function that returns a { kind, content, score } packet. - Engram validates the schema before loading. Run engram plugin list to see what's wired. - The provider plugin SDK is documented in docs/PLUGINS.md. + Two paths. Easy path (v3.0): drop a .mjs in ~/.engram/plugins/ + with an mcpConfig block pointing at any MCP server — the loader spawns, connects, and merges + tool calls into the rich packet automatically. Works in ~10 lines. Full control path: write your own + async resolve(filePath, context) + isAvailable() in the same .mjs. + The loader validates the shape before loading; broken plugins are silently skipped and reported in + engram plugin list. Full spec + examples in docs/plugins/README.md.
    @@ -1389,9 +1806,10 @@

    The questions agents ask first.

    Stop paying for context
    you've already paid for.

    -

    - Three commands. The next session has memory. If you don't see at least - 5× fewer tokens in the dashboard, uninstall and we'll buy back the 30 seconds. +

    + One command. The next session has memory. 89.1% fewer tokens + in a reproducible benchmark. Any MCP server becomes a provider in 10 lines. Zero cloud, zero + telemetry, Apache 2.0 — uninstall whenever, it's 30 seconds back either way.

    @@ -1409,7 +1827,7 @@

    engram - v2.0.2 · Apache 2.0 · Built by @NickCirv + v3.0.0 "Spine" · Apache 2.0 · Built by @NickCirv