feat: agent core improvements — skills, modes, memory, vision, tasks by Z0mb13V1 · Pull Request #723 · mindcraft-bots/mindcraft

Z0mb13V1 · 2026-03-04T01:05:16Z

Broad improvements across the agent execution loop — 35 files. See branch for full diff.

…n, tasks Key improvements across the agent execution loop: - action_manager.js: improved error handling and action lifecycle management - agent.js: cleaner initialization, connection retry logic - coder.js: SES sandbox hardening, better lint-and-retry flow - skills.js: expanded primitive skills, better error recovery - modes.js: additional autonomous modes (hunting, cowardice, self-preservation) - memory_bank.js: ChromaDB integration for persistent cross-session memory - commands/actions.js: new !commands, improved argument parsing - commands/queries.js: extended world-state query surface - vision/*: llava-based vision pipeline for screenshot analysis - tasks/*: construction, cooking task runners with validation - npc/*: improved NPC goal resolution - utils/usage_tracker.js: per-session token and cost tracking

Copilot

Pull request overview

This PR introduces broad “agent core” improvements across execution, navigation (Baritone), vision robustness, persistence (history/learnings/usage), and command/mode behaviors.

Changes:

Add usage/cost tracking and MindServer usage reporting, plus atomic/safer persistence for histories and new “learnings”.
Switch major navigation/path checks from mineflayer-pathfinder to Baritone (bot.ashfinder) and add new safety/utility skills and modes.
Make vision/browser viewer modules load lazily with graceful fallback when native canvas/WebGL deps aren’t available.

Reviewed changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/utils/usage_tracker.js	New per-agent usage + cost tracking persisted to `bots/<agent>/usage.json`.
src/utils/mcdata.js	Plugin loading tweaks (auto-eat loader) + Baritone plugin load + ashfinder config on spawn.
src/utils/examples.js	Small logging/unused-var cleanup.
src/agent/vision/vision_interpreter.js	Lazy-load Camera module; disable vision gracefully when unavailable/not ready.
src/agent/vision/camera.js	Fix init order for worldView; add async init error handling + destroy cleanup.
src/agent/vision/browser_viewer.js	Lazy-load `prismarine-viewer` with fallback logging.
src/agent/tasks/tasks.js	Formatting + small control-flow cleanup.
src/agent/tasks/cooking_tasks.js	Formatting + minor variable rename.
src/agent/tasks/construction_tasks.js	Safer `hasOwnProperty` usage + formatting/renames.
src/agent/speak.js	Hardening system TTS against injection by switching to `spawn`; queue processing changes.
src/agent/self_prompter.js	Formatting/semicolons + minor logging cleanup.
src/agent/npc/item_goal.js	Formatting + minor unused-var suppression.
src/agent/npc/controller.js	Safer filesystem reads for constructions + `hasOwnProperty` hardening.
src/agent/npc/build_goal.js	Remove unused mcdata import.
src/agent/modes.js	Expanded self-preservation behaviors (drowning/fire/unstuck) + new modes (night_bed/auto_eat/panic_defense).
src/agent/mindserver_proxy.js	Support url-or-port + connection timeout + remote registration + get-usage endpoint.
src/agent/memory_bank.js	Formatting/semicolons.
src/agent/library/world.js	Replace clear-path checks with Baritone path generation; add ashfinder guard.
src/agent/library/skills.js	Large Baritone migration + timeouts + exploration/tool progression + many new helper skills.
src/agent/library/skill_library.js	Improve embedding-model failure logging.
src/agent/library/lockdown.js	Clarify rationale for unsafe eval + minor export formatting.
src/agent/library/index.js	Include `dragon_runner` skills in doc generation.
src/agent/library/full_state.js	Expand full-state to include resume/self-prompter fields.
src/agent/learnings.js	New learnings store with atomic writes + summary/stats.
src/agent/history.js	Atomic writes + memory sanitization + robust load handling.
src/agent/conversation.js	Make conversation ending async and await scheduling/resume operations.
src/agent/connection_handler.js	Minor try/catch cleanup.
src/agent/commands/queries.js	Formatting + fix literal `\\-` to `-`.
src/agent/commands/index.js	Track blocked commands, fix blacklist removal bug, and tighten parsing formatting.
src/agent/commands/actions.js	Add new commands (explore, safeMoveTo, dragon progression, etc.) + teammate-attack block.
src/agent/coder.js	Make templates load synchronously + add code execution timeout + freeze exposed modules.
src/agent/agent.js	Add learnings integration, message validation, prefix targeting, stop/freeze bypass, and more robust shutdown.
src/agent/action_manager.js	Add slow-loop + stuck detection, and cross-invocation collect failure handling/auto-explore.
bots/lintTemplate.js	Adjust lint template imports to avoid unused warnings.
bots/execTemplate.js	Fix trailing braces/semicolon formatting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-04T01:10:01Z

src/agent/conversation.js

+    async endConversation(sender) {
        if (this.convos[sender]) {
            this.convos[sender].end();
            if (this.activeConversation.name === sender) {
                this._stopMonitor();
                this.activeConversation = null;
                if (agent.self_prompter.isPaused() && !this.inConversation()) {
-                    _resumeSelfPrompter();
+                    await _resumeSelfPrompter();
                }
            }
        }
    }

-    endAllConversations() {
+    async endAllConversations() {
        for (const sender in this.convos) {
-            this.endConversation(sender);
+            await this.endConversation(sender); // RC30: await async endConversation
        }
        if (agent.self_prompter.isPaused()) {
-            _resumeSelfPrompter();
+            await _resumeSelfPrompter();
        }


ConversationManager.endConversation() / endAllConversations() are now async, but there are still non-awaited call sites (e.g. in commands or within this module) that will become floating Promises and can cause ordering issues (self-prompter resume happening after the caller continues). After making these APIs async, ensure all invocations either await them or explicitly discard/handle the Promise (void ...catch(...)) to satisfy no-floating-promise and preserve sequencing.

Copilot · 2026-03-04T01:10:01Z

src/utils/usage_tracker.js

+        const isLocal = provider === 'ollama' || provider === 'vllm';
+        const costInfo = isLocal ? COST_TABLE._vllm_default : getCostPerMillion(modelName);
+        if (costInfo) {


Local providers are detected via provider === 'ollama' || provider === 'vllm', but the cost lookup always uses COST_TABLE._vllm_default. This is currently harmless because both defaults are $0, but it’s easy to introduce incorrect accounting if Ollama and vLLM ever diverge. Select the provider-specific default (e.g. _ollama_default vs _vllm_default) or map provider→default key explicitly.

Copilot · 2026-03-04T01:10:01Z

src/agent/library/skills.js

@@ -486,45 +592,176 @@ export async function collectBlock(bot, blockType, num=1, exclude=null) {
            }
            await bot.equip(bucket, 'hand');
        }
-        const itemId = bot.heldItem ? bot.heldItem.type : null
+        const itemId = bot.heldItem ? bot.heldItem.type : null;
        if (!block.canHarvest(itemId)) {
            log(bot, `Don't have right tools to harvest ${blockType}.`);
            return false;


collectBlock() pauses the unstuck mode but does not guarantee it will be unpaused on early returns/exceptions (e.g. missing bucket or missing tool returns before reaching the unpause at the end). This can leave unstuck permanently paused for the rest of the session. Wrap the body after pause() in a try/finally (or use a guard helper) so bot.modes.unpause('unstuck') always runs, including when returning early.

Copilot · 2026-03-04T01:10:02Z

src/agent/library/skills.js

+     **/
+    try {
+        if (!minecart_pos) {
+            const minecarts = bot.entities


enterMinecart() treats bot.entities as an array (bot.entities.filter(...)), but in mineflayer bot.entities is an object keyed by entity id. This will throw at runtime and break the command. Use Object.values(bot.entities) (or Object.values(bot.entities).filter(...)) before sorting/filtering.

Suggested change

const minecarts = bot.entities

const minecarts = Object.values(bot.entities)

Copilot · 2026-03-04T01:10:02Z

src/agent/agent.js

    cleanKill(msg='Killing agent process...', code=1) {
        this.history.add('system', msg);
-        this.bot.chat(code > 1 ? 'Restarting.': 'Exiting.');
+        try { this.bot.chat(code > 1 ? 'Restarting.': 'Exiting.'); } catch {}
        this.history.save();
+        if (this.learnings) {
+            this.learnings.save();
+        }
+        if (this.prompter?.usageTracker) {
+            this.prompter.usageTracker.saveSync();
+            this.prompter.usageTracker.destroy();
+        }
+        try { this.bot.quit(); } catch {}
        process.exit(code);


cleanKill() calls async methods (history.add, history.save, learnings.save) without awaiting or otherwise handling the returned Promises, then immediately process.exit()s. This can lose history/learnings data and will also trip the repo’s no-floating-promise lint rule. Consider making cleanKill async (or splitting into cleanKillAsync), awaiting the saves, and only then exiting; if you need fire-and-forget, explicitly handle the Promise (e.g. void ...catch(...)).

Copilot · 2026-03-04T01:10:02Z

src/agent/speak.js

@@ -23,7 +23,7 @@ export function speak(text, speak_model) {
    }

    speakingQueue.push(item);
-    if (!isSpeaking) processQueue();
+    if (!isSpeaking) await processQueue();
 }


speak() was changed to async and now awaits processQueue(). This makes typical call sites fire a floating Promise (fails no-floating-promise) and also changes semantics from “enqueue and return immediately” to “enqueue and block until the queue drains” when callers do await speak(...). Either keep speak() non-async and internally start the queue with explicit Promise handling, or update all call sites to explicitly handle the returned Promise (await / void ...catch(...)) depending on desired behavior.

Copilot AI review requested due to automatic review settings March 4, 2026 01:05

Copilot started reviewing on behalf of Z0mb13V1 March 4, 2026 01:05 View session

Copilot AI reviewed Mar 4, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: agent core improvements — skills, modes, memory, vision, tasks#723

feat: agent core improvements — skills, modes, memory, vision, tasks#723
Z0mb13V1 wants to merge 1 commit intomindcraft-bots:developfrom
Z0mb13V1:feat/pr4-agent-core

Z0mb13V1 commented Mar 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 4, 2026

Uh oh!

Copilot AI Mar 4, 2026

Uh oh!

Copilot AI Mar 4, 2026

Uh oh!

Copilot AI Mar 4, 2026

Uh oh!

Copilot AI Mar 4, 2026

Uh oh!

Copilot AI Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	const minecarts = bot.entities
	const minecarts = Object.values(bot.entities)

Conversation

Z0mb13V1 commented Mar 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants