Further improvements of the orchestrator

[Simon-Free]

First and foremost, thank you a lot for your work. I was able to download it and plug it into my system with minimal efforts. The code was quite easy to iterate on, and I was quickly able to add some simple features.

I spent the last 3 days working specifically on optimizing the agentic orchestrator, and found some - i believe - very meaningful improvements

I would love to try to integrate them into the product, if you are okay with it. But I did a lot of stuff, including some refacto, some minor changes in the way session works, etc. so I need to take the time to do some more atomic pull requests.

The most important optimisations are coming from:

1. Agressively parallelizing tool calls, and defining a "most optimal" 4-steps development framework (find the file to read, read them, plan, write + execute).
2. Have a description for each file, a bit like the description available for each skills (hence the refacto i did to split your code into ~200 lines files), enabling better search and diminishing the number of reading turns.
3. Design a context garbage collector that the model can use to compress/drop information.

On some internal feature development + bugfixes tasks, i was able to cut input token use (majority of the cost) by a factor of x10, and get a x3-5 speedup compared to claude code vanilla with API billing activated.

Here is my fork for reference : https://github.com/Simon-Free/bouzecode. The README explains in more details my reasoning.

[chauncygu]

Hi @Simon-Free, thanks for taking the time to write this up, and for sharing the fork. The numbers you're reporting (≈10× input-token reduction and 3–5× speedup vs Claude Code with API billing on internal tasks) are significant and I want to understand them properly. I went through the bouzecode README, and each of your three ideas lands on a real pain point we've bumped into too.

A few questions that would help me think about how each piece integrates cleanly:

1. Four-step framework + aggressive tool parallelism. We already have concurrent_safe=True on read-only tools and a ThreadPoolExecutor in the agent loop (since v3.05.72), so my question is more about how the four-step discipline is enforced, is it a structured plan node that gates which tool can be called when, system-prompt-level guidance, or a separate planning pass with its own LLM call? Knowing the mechanism makes it much easier to see where the wins are coming from.

2. Per-file descriptions. This feels like the highest-leverage idea of the three. A few practicalities:

• How do descriptions get created — auto-generated on first read, extracted from docstrings, or written as explicit metadata?
• Where are they stored (sidecar file, SQLite index, inline YAML-in-comments)?
• And does the model explicitly query the description registry before calling Read, or is that shielded inside the tool itself?

I can see this fitting next to our existing skill registry pattern, which is already a name + description + content store.

3. Context garbage collector. We have /compact and an auxiliary-model summarizer for side tasks, so I'd like to understand how yours differs in practice. Is the model invoking a GC tool explicitly, or is it triggered heuristically (turn count / token threshold)? And do dropped items leave a placeholder ("had X, compressed on turn N") so the model can re-fetch on demand?

4. Session model changes. You mentioned "some minor changes in the way session works", heads-up that v3.05.74 (just merged to main) moved the web UI onto SQLite (users / chat_sessions / messages / api_credentials tables, see web/db.py + web/models.py). The CLI session store is still session_store.py and is separate. If your session changes touch either, please flag it in the PR so we can decide whether the model changes fit on top.

On integration: yes, atomic PRs would be really helpful. A few things that make the review much faster on my side:

• One concept per PR. If the four-step framework and the per-file description registry are independent, please keep them on separate branches.
• A regression test. Our suite is at tests/ (362 tests total, 21 of them end-to-end HTTP tests for the web UI added in v3.05.74). Even one pytest per PR that exercises the new code path saves me a lot of time.
• Backward compatibility where possible. CheetahClaws is used both as a CLI and as a library, with several pluggable surfaces (tools/__init__.py's _EXTENSION_MODULES, commands under commands/, bridges, skills, plugin manifests). If your refactor shifts public entry points, a note in the PR description about which ones will let us choose between adding an explicit deprecation.

Happy to discuss specific design questions on the fork in the meantime, feel free to drop links to the commits/files you consider the most instructive starting points. Thanks again for putting this much care into the project; looking forward to what you share.

Thanks again for your great contributions to this community.

[Simon-Free]

Hey, thanks for the thoughtful questions — happy to dig in on each of the three.

1. Four-step framework + parallelism

The discipline is entirely system-prompt-level, no planner LLM call and no structured node gating tools. There's a 3-Phase Workflow — MANDATORY section in the system prompt (READ → PLAN → EXECUTE) with very explicit "HARD RULES — Turn Economy" language (ONE LLM ROUND-TRIP IS THE MOST EXPENSIVE THING IN THE SYSTEM, 3 turns MAX, etc.). No enforcement loop, but empirically the model follows it and fan-out goes way up.

A related system-prompt rule that matters a lot in practice: ask forgiveness, not permission. The prompt explicitly tells the model not to re-Read a file before editing it if the file is already in conversation history — attempt the Edit directly, and if the file changed on disk, the Edit result includes a diff warning the model can react to. Same idea for uncertain tool calls in general: If a tool call might fail, EMIT IT ANYWAY. Seeing the error in the same turn is infinitely cheaper than spending a turn to "check first." A failed tool call = free information. An extra LLM turn = wasted money and time. That single shift kills a huge number of defensive "let me just verify first" round-trips.

The real win on top of concurrent_safe=True for reads is that Write/Edit/Bash are also fannable in one turn via depends_on. Each tool call accepts tool_call_alias + depends_on=[...]. Same-file Write/Edit implicitly chain; cross-file Write/Edit run in parallel.

Concrete example — instead of 3 round-trips (write temp script → run it → delete it), it's one turn:

Write(file_path="temp_check.py", content="...",       alias="w1")
Bash (command="python temp_check.py",  depends_on=["w1"], alias="b1")
Bash (command="del   temp_check.py",   depends_on=["b1"])

Same pattern for the Phase 3 execute block — all source edits + all test edits + all GetDiagnostics + one pytest -n auto -v fire in a single DAG:

Edit(file_path="config.py",      alias="e1")
Edit(file_path="main.py",        alias="e2")
Write(file_path="tests/test_x",  alias="t1")
GetDiagnostics(file="config.py", depends_on=["e1"])
GetDiagnostics(file="main.py",   depends_on=["e2"])
Bash("pytest -n auto -v tests/", depends_on=["e1","e2","t1"])

Write parallelism isn't huge in itself — the leverage is collapsing "edit / verify / test" sequences that used to be N turns into one.

2. Per-file descriptions

Two pieces, no sidecar / SQLite / registry — the description lives inline in the file itself as a standard comment on line 1, in a [desc]…[/desc] marker:

# [desc] Manages garbage collection of tool-call results in conversation context to reduce token usage. [/desc]
from __future__ import annotations
...

• Creation: lazy, on demand. There's a GetFolderDescription(folder_path=...) tool. It walks the folder and prints a tree with each file's [desc]. Any code file missing a [desc] triggers generation — a parallel ThreadPoolExecutor(max_workers=20) of Haiku calls, one per missing file, each returning a single 100-char line that gets prepended to the file. Cheap: a 40-file folder with no descriptions yet = 40 small Haiku calls running in parallel, typically a few seconds and a few thousand tokens total, amortized across every future session in that folder. Once a file has a [desc], it's never re-generated on read — only the write hook refreshes it.
• Freshness: a hook on the Write tool spawns a daemon thread after each write. If the file already has a [desc] block, it re-generates it from the new contents via Haiku. So descriptions decay gracefully with edits.
• Size discipline: a <200 lines per file guideline in the system prompt (refactoring skill) keeps the one-liner meaningful. If a file drifts above ~200 lines, the model splits it, and each half gets its own [desc].
• Model usage: the model calls GetFolderDescription explicitly during Phase 1, same batch as its Glob/Grep/Read. No shielding in Read itself — Read stays simple. The value is that GetFolderDescription on a 40-file folder returns a single small tree of "file -- description" lines instead of the full content of 40 files.

Where the descriptions actually live isn't really the important part — in-file, sidecar, SQLite, whatever fits your architecture best. The leverage comes from having any cheap pre-indexed summary so the model doesn't have to Read full files to form a map of the codebase.

3. Context GC

Key difference vs /compact: the model decides, per turn, per tool-call-id, what to drop, not a threshold and not a heuristic. There's a ContextGC tool the system prompt requires to be called in every turn that includes other tool calls, with four fields:

• trash: list of tool_call_ids whose results are no longer needed
• keep_snippets: trim a kept result to just the interesting slice via anchors (keep_after, keep_before, keep_between)
• notes: named working-memory entries ({name, content}) that survive across turns — model's scratchpad for what mattered from trashed stuff
• trash_notes: drop obsolete notes

Dropped tool results aren't replaced by an opaque placeholder — they're collapsed into the same minimal one-liner the user saw in the UI (Read config.py (220 lines), Edit main.py +12 -3, Bash pytest → 47 passed, Grep "AUTH_TOKEN" → 60 hits in 8 files). The model ends up with a compact action log, very low token cost even for dozens of trashed calls, but still enough signal to know what it did and in what order. If it really needs the content back, it just re-Reads.

In the long run, this is probably the most important of the three. The other two (per-file descriptions, parallelism) are genuine wins too, but they lower the amount of steps necessary. The GC is what keeps total context roughly flat across a long task instead of growing linearly with every tool call. Without it, a 30-turn session balloons with every tool call; with it, the transcript stays down to plan + notes + recent work + a scrollable log of prior actions, roughly independent of how many turns have already happened. That's where the "small but relevant context at all times" property comes from, automatically, without the model or the user having to trigger /compact.

Notes are re-injected at the top of the latest user turn each loop, so they stay salient.

Concrete example — fixing a bug in config.py, turn by turn:

• Turn 1 (READ): the model emits Read(config.py) → big file, Read(main.py), Grep("AUTH_TOKEN") → 60 hits, GetFolderDescription(auth/).
• Turn 2 (PLAN): WritePlan(...). In the same batch, ContextGC(trash=["grep_123"], notes=[{name:"auth_token_refs", content:"Used in 3 places: main.py:42, config.py:88, auth/middleware.py:15"}]). The 60-hit grep is gone — 58 lines of noise replaced with one 3-line note the model authored.
• Turn 3 (EXECUTE): Edit(config.py, ...), Bash(pytest ...). Same batch: ContextGC(trash=["read_config", "read_main"], keep_snippets=[{id:"read_middleware", keep_between:["class Auth", "def verify"]}]). The two full Reads disappear — the model already made its edits, it doesn't need the full source in context anymore. Middleware file is trimmed to the 20 lines that mattered.

By turn 4 the context contains the plan, a 3-line note, a trimmed middleware snippet, the Edit, and the pytest result — not the 800 lines of raw Read / Grep output that got us there. That's where most of the 10× reduction comes from on (not even so) long tasks.

4. Session changes

Not load-bearing — just making session saves more robust. I'll rebase onto v3.05.74's SQLite model without pushing that part as its own PR if it turns into churn.

On integration

Fully agreed on atomic PRs. The three concepts are independent — per-file descriptions, context GC tool, and the system-prompt discipline can each ship separately.

[generated by LLM, because at equal time spent I find its prose much clearer than mine, and it crafts minimal but meaningful examples very well, but i carefully read everything to ensure clarity + truthness]