🐛 engram init --with-skills crashes with JavaScript heap out of memory

[mechtar-ru]

Environment

• Axolotl project size: 2.2 GB, 34,327 files
  • frontend/node_modules: 817 MB
  • frontend/release: 785 MB
  • frontend/jre: 127 MB (bundled JRE)
  • Actual source code: ~2 MB

Steps to Reproduce

NODE_OPTIONS="--max-old-space-size=8192" engram init --with-skills

Root Cause

File: engramx/dist/chunk-IIFAAYDO.js:1203-1244

The extractDirectory function has no maximum depth limit on recursive walk():

function walk(dir) {
  // ...
  if (visitedDirs.has(realDir)) return;
  visitedDirs.add(realDir);
  const entries = readdirSync(dir, { withFileTypes: true });
  for (const entry of entries) {
    if (entry.isDirectory()) {
      // Hardcoded exclusions only — no .engramignore support
      if (entry.name.startsWith(".") || 
          entry.name === "node_modules" || ...) {
        continue;
      }
      walk(fullPath);  // ← RECURSIVE, NO DEPTH LIMIT
    }
  }
}

Problems:

1. NO maximum depth — crashes on deep directory trees
2. realpathSync failure = silent termination of entire walk
3. visitedDirs Set grows unbounded — symlinks to distinct dirs cause memory leak
4. No .engramignore support — users can't exclude node_modules, release, jre

Stack Trace Evidence

InterpreterEntryTrampoline (repeats) ← call stack overflow from unbounded recursion

Suggested Fixes

1. Add MAX_DEPTH constant with limit check in walk()
2. Wrap readdirSync in try-catch
3. Add .engramignore file support

Workaround

Not available — even 8GB heap limit is insufficient due to call stack overflow.

[mechtar-ru]

<--- Last few GCs --->

[4426:0x7f7b99a0a000] 1718296 ms: Scavenge (interleaved) 8106.9 (8147.2) -> 8087.3 (8217.2) MB, pooled: 0 MB, 828.76 / 0.00 ms (average mu = 0.380, current mu = 0.375) allocation failure;
[4426:0x7f7b99a0a000] 2087306 ms: Mark-Compact (reduce) 8087.4 (8217.2) -> 8086.2 (8095.6) MB, pooled: 0 MB, 244093.15 / 0.00 ms (+ 1.9 ms in 1 steps since start of marking, biggest step 1.9 ms, walltime since start of marking 247155 ms) (average mu = 0
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

1: 0x107ccd663 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/local/bin/node]
2: 0x107efeb0d v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/bin/node]
3: 0x10817ac47 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
4: 0x10817e38a v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector, v8::base::TimeTicks) [/usr/local/bin/node]
5: 0x108192306 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_1::operator()() const [/usr/local/bin/node]
6: 0x108191b9f void heap::base::Stack::SetMarkerAndCallbackImpl<v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_1>(heap::base::Stack*, void*, void const*) [/usr/local/bin/node]
7: 0x108c170b8 PushAllRegistersAndIterateStack [/usr/local/bin/node]
8: 0x1081786e9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
9: 0x1080e0e2f v8::internal::StackGuard::HandleInterrupts(v8::internal::StackGuard::InterruptLevel) [/usr/local/bin/node]
10: 0x10866e96f v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
11: 0x108d16236 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
zsh: abort NODE_OPTIONS="--max-old-space-size=8192" engram init

[mechtar-ru]

#6

[NickCirv]

Thanks for the report @mechtar-ru. The stack shows a V8 heap OOM during mark-compact at 8GB — so something in the skills-miner is building an unbounded structure on your particular skills directory, not a uniform scaling bug (engram processes my own 140-skill / 2,690-keyword setup in 27ms).

To pinpoint, could you run these and paste the output?

du -sh ~/.claude/skills/ 2>/dev/null
find ~/.claude/skills -maxdepth 1 -type d | wc -l
find ~/.claude/skills -maxdepth 2 -type f -size +500k 2>/dev/null
find ~/.claude/skills -type l 2>/dev/null

Most likely suspects based on the stack:

1. A single SKILL.md that's much larger than the typical 1-5 KB (triggers catastrophic regex backtracking in extractTriggers).
2. A symlink cycle in ~/.claude/skills/ that discoverSkillFiles is resolving into an unbounded walk.
3. Several thousand skills + a quadratic dedupe pattern compounding.

In parallel, I'm landing defensive limits for v2.1.0 regardless of what your env turns out to be:

• 1 MB cap on SKILL.md size (configurable via --max-skill-size)
• 64 KB cap on the body passed to extractTriggers
• 500 triggers per skill max, 10K skills total max
• Realpath-dedup on symlink resolution to break any cycles

If your output above shows one of the suspects, I can ship a more surgical fix. Either way, you shouldn't need to wait — the --with-skills flag will be bounded in v2.1.0.