Semantic Remote Control (SRC) — PiKVM MCP Server

Hardware-Level AI Orchestration

Semantic Remote Control (SRC) is an open-source bridge and Model Context Protocol (MCP) server that empowers AI Agents (LLMs) to control completely air-gapped, pre-OS, or structurally offline environments using hardware-level video capture and HID (Keyboard/Mouse) emulation via a PiKVM setup.

While conventional automation requires a functioning OS and network stack (SSH/API), SRC acts at the physical layer. It allows your LLM to:

• Enter and navigate BIOS/UEFI menus visually.
• Install Operating Systems from scratch.
• Administer air-gapped systems utilizing pure screen OCR and USB inputs.

---

🏗 Repository Structure

This repository is structured for two primary audiences: developers deploying the MCP integration and researchers reading the architectural design.

• /pikvm-agent-bridge: A secure, local Node.js HTTP bridge. It exposes a minimal, safe control surface to your AI, preventing you from exposing root PiKVM credentials directly to agents.
• /open-operator-pikvm-mcp: The standard stdio MCP Server. Connect this directly into any MCP-compatible agent (e.g., Claude Desktop, OpenHands) to supply it with physical interaction tools.
• /docs: The full academic research documentation and final year project LaTeX files detailing the AI orchestration framework.

---

🚀 Quick Start for Developers

Integrating SRC into your LLM workflows involves a two-step architecture.

1. Launch the Local Bridge

The bridge protects your raw PiKVM connection. Keep it bound to localhost to ensure the security of your hardware interface.

cd pikvm-agent-bridge

# Configure your variables (see .env.example)
export PIKVM_BASE_URL="https://pikvm.your-network.net"
export PIKVM_USERNAME="admin"
export PIKVM_PASSWORD="your-password"
export BRIDGE_TOKEN="your-secret-local-token"

# Start the bridge (Defaults to port 8787)
node server.mjs

2. Connect the MCP Server to Your Agent

Your intelligent agent connects to the MCP server, which safely communicates with the local bridge. Use the JSON configuration below to attach the MCP server to standard clients like Claude Desktop:

{
  "mcpServers": {
    "semantic-remote-control": {
      "command": "node",
      "args": ["/absolute/path/to/src/open-operator-pikvm-mcp/mcp-server.mjs"],
      "env": {
        "PIKVM_BRIDGE_BASE_URL": "http://127.0.0.1:8787",
        "PIKVM_BRIDGE_TOKEN": "your-secret-local-token"
      }
    }
  }
}

---

🛠 Available MCP Tools

Once attached, your LLM context will dynamically inject the following capabilities:

• pikvm_get_state / pikvm_capture_snapshot: Take physical frame captures.
• pikvm_ocr_screen: Extract localized text coordinates for precise clicking.
• pikvm_type_text / pikvm_send_shortcut / pikvm_send_key: Emit hardware USB key events natively bypassing any firewall or login screen.
• pikvm_mouse_move / pikvm_mouse_click: Emit generic cursor interactions.
• pikvm_power_action: Execute ATX chassis power logic (e.g. hard reboot).

---

🛡 Security First

Do not hand agents raw PiKVM credentials! Use the bridge as an intermediate safety layer. Limit power control toggles (PIKVM_ALLOW_POWER) to false when training or testing agents to prevent accidental hardware corruption. Prefer a secure tunnel (e.g., Tailscale) instead of exposing your PiKVM IP to the public internet.

📸 Assembled Hardware Interface

📚 Components & References

This project is built using the following core hardware components and software tools:

• Pimoroni Clipper HAT - LTE/4G connectivity.
• UPS HAT for Raspberry Pi Zero - Uninterruptible Power Supply context.
• Waveshare HDMI to CSI Adapter - Hardware-level video capture.
• PiKVM v2 Documentation - Core underlying KVM software suite.
• pgrok - Poor man's ngrok for secure tunneling.
• LabXIII Ltd (UK) - Company Affiliation.
• LabXIII LabXIII website

---

Project Affiliation

      

Project Developed by Ali Ganji (Z23599848)
Supervisor: Dr. Sayed Pouria Talebi
Developed at: LabXIII Ltd (UK)