Skip to content

fix(cache): compact newest tool results first to preserve prompt cache prefix#1

Open
bcherny wants to merge 3591 commits intomainfrom
cache/context-guard-reverse
Open

fix(cache): compact newest tool results first to preserve prompt cache prefix#1
bcherny wants to merge 3591 commits intomainfrom
cache/context-guard-reverse

Conversation

@bcherny
Copy link
Copy Markdown
Owner

@bcherny bcherny commented Mar 25, 2026

Problem

compactExistingToolResultsInPlace (src/agents/pi-embedded-runner/tool-result-context-guard.ts:111) iterated front-to-back. When context exceeded 75%, it replaced the oldest tool results with [compacted: ...] placeholders first. This rewrote messages[k] for small k, invalidating the provider prompt cache from that point onward on every turn past the threshold.

Fix

Reverse the loop to compact newest-first: for (let i = messages.length - 1; i >= 0; i--). The cached prefix stays byte-identical; only the tail diverges.

Tradeoff

The model loses recent tool output instead of old. Acceptable because:

  • This guard only fires as an emergency measure past 75% context
  • Cache-bust cost (full re-tokenization every turn) outweighs the model-quality hit
  • If compaction isn't enough, the 90% preemptive-overflow cascade still triggers full session compaction

Verification

  • 11/11 tests pass in tool-result-context-guard.test.ts
  • No downstream dependency on compaction order: sole caller enforceToolResultContextBudgetInPlace only reads the return value (a sum) and the in-place mutations
  • Early-exit (reduced >= charsNeeded) is direction-agnostic — the accumulator is a simple sum
  • applyMessageMutationInPlace + cache invalidation are per-message, no ordering assumption

Notes

  • Test assertions updated to reflect newest-first ordering (the "survives repeated large tool results" test previously asserted oldest compacted / newest intact; now inverted)
  • Pre-commit hook currently fails on origin/main due to unrelated getEditorKeybindings import breakage introduced in c6d8318 — this change does not touch those files

🤖 Generated with Claude Code

steipete and others added 30 commits April 3, 2026 18:56
…#60452)

* refactor: move bundled replay policy ownership into plugins

* test: preserve replay fallback until providers adopt hooks

* test: cover response replay branches for ollama and zai

---------

Co-authored-by: Shakker <shakkerdroid@gmail.com>
Takhoffman and others added 29 commits April 3, 2026 17:06
Merged via squash.

Prepared head SHA: 28ae50b
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Reviewed-by: @gumadeiras
…or [AI] (openclaw#59822)

* fix: address issue

* fix: address PR review feedback

* docs: add changelog entry for PR merge

* docs: normalize changelog entry placement

---------

Co-authored-by: Devin Robison <drobison@nvidia.com>
…a exceeds size limit (openclaw#60289)

Merged via squash.

Prepared head SHA: f33dd49
Co-authored-by: efe-arv <259833796+efe-arv@users.noreply.github.com>
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Reviewed-by: @gumadeiras
* fix(sandbox): block home credential binds
* fix(sandbox): harden blocked credential bind checks
Merged via squash.

Prepared head SHA: 64223e5
Co-authored-by: MonkeyLeeT <6754057+MonkeyLeeT@users.noreply.github.com>
Co-authored-by: mcaxtr <7562095+mcaxtr@users.noreply.github.com>
Reviewed-by: @mcaxtr
…e prefix

compactExistingToolResultsInPlace iterated front-to-back, replacing the
oldest tool results with placeholders when context exceeded 75%. This
rewrote messages[k] for small k, invalidating the provider prompt cache
from that point onward on every subsequent turn.

Reverse the loop to compact newest-first. The cached prefix stays intact;
the tradeoff is the model loses recent tool output instead of old, which
is acceptable since this guard only fires as an emergency measure past
the 75% threshold.
@odysseus0 odysseus0 force-pushed the cache/context-guard-reverse branch from adcf7eb to 1358cba Compare April 4, 2026 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.