feat: Modal.com remote GPU inference backend

[ofekby]

Motivation

QMD relies on three local GGUF models (embedding, query expansion, reranking) via node-llama-cpp. Users without a dedicated GPU — or on machines where llama.cpp doesn't build cleanly — can't use hybrid search at all. This PR adds an optional remote GPU backend so those users get the same search quality without local hardware requirements.

What this does

Adds Modal.com remote GPU inference as a transparent alternative backend. When enabled, all three model operations (embed, expand, rerank) run on a T4 GPU via Modal instead of locally.

• qmd modal deploy / status / test / destroy — full lifecycle CLI
• Transparent routing — when modal.inference=true in config, qmd query and qmd embed use Modal automatically; no other commands change
• Same models, same results — the remote backend runs identical GGUF files through llama-server
• qmd status shows the active inference backend (local vs Modal) so users can tell at a glance which is in use

Architecture

┌─────────────┐    modal npm SDK    ┌──────────────────────────┐
│  qmd (JS)   │ ──────────────────► │  Modal container (T4)    │
│  ModalLLM   │                     │  ┌─ llama-server :8081   │ ← embed
│  implements  │                     │  ├─ llama-server :8082   │ ← expand
│  LLM iface  │                     │  └─ llama-server :8083   │ ← rerank
└─────────────┘                     └──────────────────────────┘

• modal/serve.py — Python Modal app running 3 llama-server subprocesses on separate ports with health checks
• ghcr.io/ofekby/qmd-llama-server:b8179-sm75 — pre-built Docker image (llama.cpp b8179, CUDA, T4/sm_75), built via CI workflow
• src/modal.ts — ModalBackend / ModalSession wrapping the Modal npm SDK with retry logic
• src/llm.ts — ModalLLM implements the LLM interface; getDefaultLLM() routes based on config
• GPU memory snapshots for fast cold starts (~6s vs ~40s without)
• Region auto-detection picks the closest Modal region to minimize latency

Key design decisions

Decision	Rationale
llama-server over llama-cpp-python	llama-cpp-python is abandoned (last release Aug 2025) and doesn't support gemma-embedding
Pre-built Docker image via GHCR	CUDA compilation exceeds Modal's image build timeout
GPU memory snapshots	Without enable_gpu_snapshot, the snapshot phase runs CPU-only
No silent fallback	Hard fail when Modal is unreachable — surprises are worse than clear errors
Region auto-detection	Reduces latency for non-US users without manual config

Test plan

• 65 unit tests covering config, backend, CLI, region detection, and integration
• Smoke test via qmd modal test (embed + generate round-trip)
• Full hybrid query end-to-end: qmd query with expansion → embedding → reranking
• Clean lifecycle test: deploy → test → query → destroy

Config

New keys in ~/.config/qmd/index.yml:

modal:
  inference: true          # enable/disable Modal backend
  gpu: "T4"               # GPU type
  scaledown_window: 15    # idle timeout (seconds)
  region: "us-east"       # auto-detected, or set manually

---

🤖 Generated with Claude Code