Feature: Virtual Desktop & noVNC Integration for GUI-based "Computer Use" capabilities.

[volkermauel]

Summary

Implements #44 — Virtual Desktop & noVNC Integration for GUI-based "Computer Use" capabilities.

Adds an optional virtual desktop environment (Xvfb + x11vnc + noVNC + openbox) that enables programmatic GUI interaction via screenshot, mouse, and keyboard APIs. Feature-gated behind OPEN_TERMINAL_ENABLE_DESKTOP (default false) so existing deployments are unaffected.

Changes

• open_terminal/utils/desktop.py (new) — DesktopManager class managing Xvfb, x11vnc, noVNC, and openbox lifecycle, with screenshot (scrot), mouse (xdotool), and keyboard input methods
• open_terminal/env.py — Added 5 config vars: ENABLE_DESKTOP, DESKTOP_DISPLAY, DESKTOP_SCREEN_SIZE, DESKTOP_VNC_PORT, DESKTOP_NOVNC_PORT
• open_terminal/main.py — 10 new API endpoints, desktop info in system prompt and /api/config
• Dockerfile — Added xvfb, x11vnc, novnc, openbox, xdotool, scrot, chromium, fonts; exposed port 6080
• entrypoint.sh — DISPLAY setup and stale X11 lock file cleanup when desktop is enabled
• test_desktop.sh (new) — 27 integration tests covering full desktop lifecycle

API Endpoints

Method	Path	Description
GET	/desktop	Desktop status (running, screen size, ports)
POST	/desktop/start	Start virtual desktop
POST	/desktop/stop	Stop virtual desktop
POST	/desktop/screenshot	Capture PNG screenshot (base64 JSON or raw binary)
POST	/desktop/click	Mouse click at (x, y)
POST	/desktop/mouse_move	Move cursor to (x, y)
POST	/desktop/drag	Mouse drag operation
POST	/desktop/type	Type text (with human-like randomized delay by default)
POST	/desktop/key	Press key / key combo (e.g. ctrl+c)
POST	/desktop/scroll	Scroll at position

Usage

docker run -p 8000:8000 -p 6080:6080 \
  -e OPEN_TERMINAL_API_KEY=your-key \
  -e OPEN_TERMINAL_ENABLE_DESKTOP=true \
  open-terminal

• API: http://localhost:8000
• noVNC web client: http://localhost:6080/vnc.html

Test Results

All 27 integration tests pass against a locally built container.

[Arokha]

In your testing did you notice how much memory footprint changed with these enabled? Just curious.

[volkermauel]

i did not to be honest, sorry

[volkermauel]

the grounding for clicking seems to be off, currently investigating with grid overlays and instructing the model as part of the result of desktop_screenshot.

converting to draft as of now, until i feel this is more mature.

[bcw222]

Great work!
It would also be nice to add support in Open WebUI so we can embed noVNC directly into the WebUI. Multi-user support should also be taken into account because it’s already available in the main branch.

For a containerized desktop environment, https://github.com/m1k1o/neko could be a great reference. It uses WebRTC instead of VNC for output, but for a GUI-based AI tool, I think noVNC is enough.

[volkermauel]

Thanks for your feedback. i‘ve opened a respective PR on open-webui to embed the novnc in the frontend. multi-user support does work in theory, but in my approach i‘m using a proxy in between that spawns a per-user pod in k8s and attaches a PVC,so the users are fully seperated in terms of data access and resources

[bcw222]

Thanks for your feedback. i‘ve opened a respective PR on open-webui to embed the novnc in the frontend. multi-user support does work in theory, but in my approach i‘m using a proxy in between that spawns a per-user pod in k8s and attaches a PVC,so the users are fully seperated in terms of data access and resources

k8s? That sounds more like terminals, while I mean in single container, just as the multi user support here does. Or at least we should document this.