feat: expose native MCP server endpoint so grubcrawler can be added as MCP service

[kordless]

Summary

Add a native MCP server endpoint inside grubcrawler so it can be registered directly in Claude Code, Codex, or any MCP-compatible host via .mcp.json — without needing the nemesis8 bridge file.

Motivation

Currently the only way to use grubcrawler as MCP tooling is through MCP/grub-crawler.py in nemesis8, which must be running locally. If grubcrawler itself speaks MCP over HTTP/SSE, any agent can add it as a remote MCP server with a single config entry:

{
  "mcpServers": {
    "grubcrawler": {
      "type": "http",
      "url": "https://grub.nuts.services/mcp"
    }
  }
}

Proposed implementation

• Add /mcp route (or /mcp/sse for SSE transport) to app/main.py
• Use mcp[http] or fastmcp server-side to expose the registered tool surface
• Tools to expose mirror what grub-crawler.py already wraps: crawl_url, crawl_batch, raw_html, download_file, agent_run, ghost_extract, mesh_peers
• Auth via Authorization: Bearer <nuts-token> on the MCP HTTP connection
• Gate behind MCP_ENABLED=true env var (default off, like AGENT_ENABLED)

Registration example (Claude Code)

{
  "mcpServers": {
    "grubcrawler": {
      "type": "http",
      "url": "https://grub.nuts.services/mcp",
      "headers": {
        "Authorization": "Bearer ahp_yourtoken"
      }
    }
  }
}

Notes

• fastmcp is already used in MCP/grub-crawler.py (nemesis8) — same library works server-side
• Should coexist with the existing AHP /{tool_name} catch-all (different protocol, different path)
• The nemesis8 grub-crawler.py bridge stays for local/offline use; this is for remote/cloud