🧠mindcraft⛏️

Crafting minds for Minecraft with LLMs and Mineflayer!

Links: FAQ | Discord Support | Video Tutorial | Blog Post | Contributor TODO | Paper Website

Caution

Do not connect this bot to public servers with coding enabled. This project allows an LLM to write/execute code on your computer. The code is sandboxed, but still vulnerable to injection attacks. Code writing is disabled by default, you can enable it by setting allow_insecure_coding to true in settings.js. Ye be warned.

---

This Fork: Hybrid Research Rig

Forked from mindcraft-bots/mindcraft — the original Minecraft AI agent framework by @kolbytn, @MaxRobinsonTheGreat, and the Mindcraft team.

This fork (mindcraft-0.1.3) extends the base Mindcraft framework with a Hybrid Research Rig — two AI bots running simultaneously on AWS EC2, combining cloud ensemble intelligence with local GPU inference.

Live deployment: All 12 services run on AWS EC2 via docker-compose.aws.yml (CloudGrok + Minecraft server + ChromaDB + LiteLLM + Grafana + Prometheus + Tailscale + Discord bot). See the Architecture wiki for full infrastructure diagrams.

Active Bots

Bot	Model	Vision	Status	Role
CloudGrok	4-model ensemble (Gemini + Grok panel)	grok-2-vision-1212	✅ EC2 (always-on)	Persistent survival/research — base maintenance, resource gathering, building
DragonSlayer	sweaterdog/andy-4:q8_0 via Ollama (RTX 3090)	ollama/llava	✅ Local (active)	Autonomous Ender Dragon speedrun — !beatMinecraft with persistent RC29 state
LocalAndy	sweaterdog/andy-4 via Ollama (RTX 3090)	gemini-2.5-flash	⏸ Local (dormant)	Research & exploration — biome exploration, strategy testing

Ensemble Decision Pipeline (CloudGrok)

Phase	Name	Description
1	Heuristic Arbiter	All 4 panel models queried in parallel; proposals scored on length, completeness, and action quality — highest score wins
2	LLM-as-Judge	When top two proposals are within 0.08 margin, Gemini Flash reviews all proposals and picks the winner
3	ChromaDB Memory	Before querying the panel, similar past decisions (similarity > 0.6) are retrieved via 3072-dim Gemini embeddings and injected as [PAST EXPERIENCE] context

Panel Models (CloudGrok Ensemble)

Model	Provider	Role
gemini-2.5-pro	Google	Panel member
gemini-2.5-flash	Google	Panel member + LLM Judge
grok-4-1-fast-non-reasoning	xAI	Panel member
grok-code-fast-1	xAI	Panel member

Infrastructure

Component	Location	Notes
Minecraft server	AWS EC2 (us-east-1)	Paper 1.21.11, non-default external port, ONLINE_MODE=FALSE
CloudGrok (ensemble bot)	AWS EC2 (us-east-1)	Cloud APIs (Gemini + xAI)
DragonSlayer (local bot)	Local Windows PC (RTX 3090)	Connects to EC2 server; Ollama inference on-device
LocalAndy (Ollama bot)	Local Windows PC (RTX 3090)	Available via Tailscale VPN; dormant while DragonSlayer active
ChromaDB vector store	AWS EC2 (us-east-1)	Ensemble memory backend
Discord bot	AWS EC2 (us-east-1)	Discord integration
Ollama (inference)	Local Windows PC (RTX 3090)	sweaterdog/andy-4:q8_0, nomic-embed-text, llava
S3 backup	Daily 3 AM UTC	7-day retention

Running the Hybrid Rig

On EC2 (one-command deploy/restart):

cd /app && bash aws/ec2-go.sh --full    # Pull code + SSM secrets + rebuild + restart
cd /app && bash aws/ec2-go.sh           # Quick restart (code pull only)
cd /app && bash aws/ec2-go.sh --secrets # Refresh API keys from SSM only

ec2-go.sh auto-detects whether it's running on EC2 (local execution) or remotely (SSH wrapper). IMDSv2 supported.

Local bot (DragonSlayer on Windows):

# Option A: One-click launcher (recommended)
.\DragonSlayer.bat              # Double-click or run from terminal

# Option B: Direct PowerShell
.\DragonSlayer-Launcher.ps1     # Or: powershell -ExecutionPolicy Bypass -File .\DragonSlayer-Launcher.ps1

# Option C: Manual
node main.js --profiles ./profiles/dragon-slayer.json   # settings.js host/port must point to your EC2 server

DragonSlayer Launcher (v4.0)

A self-contained one-click launcher for the local DragonSlayer bot. Double-click DragonSlayer.bat and it handles everything:

Feature	Description
Pre-flight checks	Validates Node 20, npm, Ollama, NVIDIA GPU, CUDA, profile, .env
Ollama management	Starts daemon if needed, pulls all 3 models (andy-4 q8_0, nomic-embed-text, llava)
Optional Paper server	Local Minecraft server with EULA prompt
Live colorized logs	7-tier keyword-matched coloring with timestamps
MindServer HUD	Auto-opens browser to http://localhost:8080
!beatMinecraft	Sends the dragon run command via Socket.IO (prompt or auto)
Crash detection	Event loop with graceful shutdown + session duration
GitHub PR workflow	Commit → push to fork → create/update PR to upstream (with feature branch support)
PS2EXE support	Convert to .exe for taskbar pinning

See LAUNCHER_README.md for full configuration, PS2EXE instructions, and troubleshooting.

From Mac (remote deploy):

bash aws/ec2-go.sh                      # SSH into EC2 + deploy (needs .pem)

Key Features

• HUD Overlay — gaming-style dashboard in the MindServer web UI (:8080) with per-bot runtime tracker (MM:SS), current goal / next action display with self-prompter state badges, and a scrollable color-coded command log.
• Live Bot Cameras — first-person prismarine-viewer streams embedded as iframes in the web UI (ports 3000+)
• Vision enabled for both bots — Xvfb + Mesa software rendering in Docker with 2s startup delay for WebGL context init
• Human message priority — requestInterrupt() fires immediately when a human player speaks
• Loop detection — tracks last 12 actions, cancels on 3-action pattern repeats or 5+ of the same action
• Per-profile blocked_actions — LocalAndy blocks !startConversation to prevent hallucinated names
• Graceful vision fallback — if WebGL init fails, bots continue without crashing
• Tailscale VPN — EC2 ↔ local 3090 tunnel for LocalAndy inference
• ec2-go.sh — one-command deploy script with IMDSv2 support, SSM secret refresh, and auto-detection of local vs remote execution
• Progress Reporter (RC30) — 5-minute milestone updates to console and Discord webhook; reports current goal, action, memory, inventory, and agent uptime
• Survival reliability (RC30) — hunger safety net, golden apple priority over regular food, void-edge avoidance, death-location recovery with inventory retrieval
• Auto inventory management (RC30) — overflow detection with automatic chest placement when slots drop below 6 before chunk operations
• Stuck recovery (RC30) — door-break last resort after 8s navigation block; path sanitization against injection; async action chain hardening

Security

This fork includes several security hardening measures:

• Whitelist enforcement — ENFORCE_WHITELIST=TRUE on the Minecraft server. whitelist.json is pre-generated with correct offline-mode UUIDs (OfflinePlayer:<name> MD5 algorithm) and mounted directly into the container. The WHITELIST env var is intentionally omitted — it queries Playerdb (Mojang API) which fails for offline-mode bot names, causing a crash-loop before Minecraft starts.
• Port obscurity — External Minecraft port changed from default 25565 to a non-standard port to reduce automated scanner noise. AWS Security Group restricts access to trusted IPs only.
• Environment variable keys — API keys loaded from .env / env vars (priority over keys.json).
• AWS SSM Parameter Store — secrets stored encrypted at /mindcraft/*, pulled at deploy time via ec2-go.sh --secrets
• Recursive prototype pollution protection — SETTINGS_JSON sanitized at all nesting depths
• Cross-platform path traversal guard — Discord bot profile paths validated with path.sep
• Input validation — message validator with command injection detection, type checks, control char stripping
• Rate limiting with auto-cleanup — prevents abuse and memory leaks from stale entries
• No hardcoded IPs — EC2 public IP, Tailscale IP, and public hostnames loaded from env vars
• ESLint hardening — no-unused-vars, no-unreachable, no-floating-promise enabled as warnings
• Deep audit — 10 priorities resolved across code, config, Docker, and cleanup (e5cf8b7a)

See the Security wiki page for full details.

Documentation

Doc	Description
docs/	Reference docs index — Notebook LLM exports, mod pack research, architecture notes
CLAUDE.md	Architecture overview, commands, configuration notes
LAUNCHER_README.md	DragonSlayer one-click launcher — config, PS2EXE, PR workflow, troubleshooting
Wiki	Full documentation — architecture, bot commands, ensemble pipeline, deployment

---

Getting Started

Requirements

• Minecraft Java Edition (up to v1.21.6, recommend v1.21.6)
• Node.js Installed (Node v18 or v20 LTS recommended. Node v24+ may cause issues with native dependencies)
• At least one API key from a supported API provider. See supported APIs. OpenAI is the default.

Important

If installing node on windows, ensure you check Automatically install the necessary tools

If you encounter npm install errors on macOS, see the FAQ for troubleshooting native module build issues

Install and Run

1. Make sure you have the requirements above.

2. Download the latest release and unzip it, or clone the repository.

3. Set up your API keys (you only need one provider):

   • Recommended: Create a .env file and add your keys (e.g. OPENAI_API_KEY=sk-...). Environment variables take priority.
   • Legacy: Rename keys.example.json to keys.json and fill in your keys. (Less secure — migrate to .env when possible.)

4. In terminal/command prompt, run npm install from the installed directory

5. Start a minecraft world and open it to LAN on localhost port 55916

6. Run node main.js from the installed directory

If you encounter issues, check the FAQ or find support on discord.

Configuration

Model Customization

You can configure project details in settings.js. See file.

You can configure the agent's name, model, and prompts in their profile like andy.json. The model can be specified with the model field, with values like model: "gemini-2.5-pro". You will need the correct API key for the API provider you choose. See all supported APIs below.

Supported APIs

API Name	Config Variable	Docs
openai	OPENAI_API_KEY	docs
google	GEMINI_API_KEY	docs
anthropic	ANTHROPIC_API_KEY	docs
xai	XAI_API_KEY	docs
deepseek	DEEPSEEK_API_KEY	docs
ollama (local)	n/a	docs
qwen	QWEN_API_KEY	Intl./cn
mistral	MISTRAL_API_KEY	docs
replicate	REPLICATE_API_KEY	docs
groq (not grok)	GROQCLOUD_API_KEY	docs
huggingface	HUGGINGFACE_API_KEY	docs
novita	NOVITA_API_KEY	docs
openrouter	OPENROUTER_API_KEY	docs
glhf	GHLF_API_KEY	docs
hyperbolic	HYPERBOLIC_API_KEY	docs
vllm	n/a	n/a
cerebras	CEREBRAS_API_KEY	docs
mercury	MERCURY_API_KEY	docs

For more comprehensive model configuration and syntax, see Model Specifications.

For local models we support ollama and we provide our own finetuned models for you to use. To install our models, install ollama and run the following terminal command:

ollama pull sweaterdog/andy-4:micro-q8_0 && ollama pull embeddinggemma

Online Servers

To connect to online servers your bot will need an official Microsoft/Minecraft account. You can use your own personal one, but will need another account if you want to connect too and play with it. To connect, change these lines in settings.js:

"host": "111.222.333.444",
"port": 55920,
"auth": "microsoft",

// rest is same...

Important

The bot's name in the profile.json must exactly match the Minecraft profile name! Otherwise the bot will spam talk to itself.

To use different accounts, Mindcraft will connect with the account that the Minecraft launcher is currently using. You can switch accounts in the launcher, then run node main.js, then switch to your main account after the bot has connected.

Tasks

Tasks automatically start the bot with a prompt and a goal item to aquire or blueprint to construct. To run a simple task that involves collecting 4 oak_logs run

node main.js --task_path tasks/basic/single_agent.json --task_id gather_oak_logs

Here is an example task json format:

{
    "gather_oak_logs": {
      "goal": "Collect at least four logs",
      "initial_inventory": {
        "0": {
          "wooden_axe": 1
        }
      },
      "agent_count": 1,
      "target": "oak_log",
      "number_of_target": 4,
      "type": "techtree",
      "max_depth": 1,
      "depth": 0,
      "timeout": 300,
      "blocked_actions": {
        "0": [],
        "1": []
      },
      "missing_items": [],
      "requires_ctable": false
    }
}

The initial_inventory is what the bot will have at the start of the episode, target refers to the target item and number_of_target refers to the number of target items the agent needs to collect to successfully complete the task.

For more optimization and automatic launching of the minecraft world, see the Tasks section of the base Mindcraft repo.

Docker Container

If you intend to allow_insecure_coding, it is a good idea to run the app in a docker container to reduce risks of running unknown code. This is strongly recommended before connecting to remote servers, although still does not guarantee complete safety.

docker build -t mindcraft . && docker run --rm --add-host=host.docker.internal:host-gateway -p 8080:8080 -p 3000-3003:3000-3003 -e SETTINGS_JSON='{"auto_open_ui":false,"profiles":["./profiles/gemini.json"],"host":"host.docker.internal"}' --volume ./keys.json:/app/keys.json --name mindcraft mindcraft

or simply

docker-compose up --build

When running in docker, if you want the bot to join your local minecraft server, you have to use a special host address host.docker.internal to call your localhost from inside your docker container. Put this into your settings.js:

"host": "host.docker.internal", // instead of "localhost", to join your local minecraft from inside the docker container

To connect to an unsupported minecraft version, you can try to use viaproxy

Bot Profiles

Bot profiles are json files (such as andy.json) that define:

1. Bot backend LLMs to use for talking, coding, and embedding.
2. Prompts used to influence the bot's behavior.
3. Examples help the bot perform tasks.

Model Specifications

LLM models can be specified simply as "model": "gpt-4o", or more specifically with "{api}/{model}", like "openrouter/google/gemini-2.5-pro". See all supported APIs.

The model field can be a string or an object. A model object must specify an api, and optionally a model, url, and additional params. You can also use different models/providers for chatting, coding, vision, embedding, and voice synthesis. See the example below.

"model": {
  "api": "openai",
  "model": "gpt-4o",
  "url": "https://api.openai.com/v1/",
  "params": {
    "max_tokens": 1000,
    "temperature": 1
  }
},
"code_model": {
  "api": "openai",
  "model": "gpt-4",
  "url": "https://api.openai.com/v1/"
},
"vision_model": {
  "api": "openai",
  "model": "gpt-4o",
  "url": "https://api.openai.com/v1/"
},
"embedding": {
  "api": "openai",
  "url": "https://api.openai.com/v1/",
  "model": "text-embedding-ada-002"
},
"speak_model": "openai/tts-1/echo"

model is used for chat, code_model is used for newAction coding, vision_model is used for image interpretation, embedding is used to embed text for example selection, and speak_model is used for voice synthesis. model will be used by default for all other models if not specified. Not all APIs support embeddings, vision, or voice synthesis.

All apis have default models and urls, so those fields are optional. The params field is optional and can be used to specify additional parameters for the model. It accepts any key-value pairs supported by the api. Is not supported for embedding models.

Embedding Models

Embedding models are used to embed and efficiently select relevant examples for conversation and coding.

Supported Embedding APIs: openai, google, replicate, huggingface, novita

If you try to use an unsupported model, then it will default to a simple word-overlap method. Expect reduced performance. We recommend using supported embedding APIs.

Voice Synthesis Models

Voice synthesis models are used to narrate bot responses and specified with speak_model. This field is parsed differently than other models and only supports strings formatted as "{api}/{model}/{voice}", like "openai/tts-1/echo". We only support openai and google for voice synthesis.

Specifying Profiles via Command Line

By default, the program will use the profiles specified in settings.js. You can specify one or more agent profiles using the --profiles argument: node main.js --profiles ./profiles/andy.json ./profiles/jill.json

Contributing

We welcome contributions to the project! We are generally less responsive to github issues, and more responsive to pull requests. Join the discord for more active support and direction.

While AI generated code is allowed, please vet it carefully. Submitting tons of sloppy code and documentation actively harms development.

Patches

Some of the node modules that we depend on have bugs in them. To add a patch, change your local node module file and run npx patch-package [package-name]

Development Team

Thanks to all who contributed to the project, especially the official development team: @MaxRobinsonTheGreat, @kolbytn, @icwhite, @Sweaterdog, @Ninot1Quyi, @riqvip, @uukelele-scratch, @mrelmida

Citation

This work is published in the paper Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning. Please use this citation if you use this project in your research:

@article{mindcraft2025,
  title = {Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning},
  author = {White*, Isadora and Nottingham*, Kolby and Maniar, Ayush and Robinson, Max and Lillemark, Hansen and Maheshwari, Mehul and Qin, Lianhui and Ammanabrolu, Prithviraj},
  journal = {arXiv preprint arXiv:2504.17950},
  year = {2025},
  url = {https://arxiv.org/abs/2504.17950},
}