"The edit you don't notice is the edit that works."
CutPilot is an AI-powered post-production plugin for DaVinci Resolve and Adobe Premiere Pro, built by Rhematek Solutions. It takes raw multi-camera footage — the kind that piles up after every podcast, YouTube shoot, or live event — and transforms it into a production-ready long-form edit plus a set of platform-optimized viral social clips, all in under 3 minutes. CutPilot lives inside your existing NLE (Non-Linear Editor) as a native panel. There is no separate app to learn, no upload to a third-party website, no loss of editorial control. You drop your footage on the timeline, open the CutPilot panel, pick the modules you want, click Process, and watch the AI construct the edit on your own timeline in real time — cuts, zooms, color grades, sound design, silence removal, and highlight clips, all editable frame-by-frame the moment the pipeline finishes.
Company: Rhematek Solutions Tagline: The edit you don't notice is the edit that works. Supported NLEs: DaVinci Resolve 18+ · Adobe Premiere Pro 2024+ Target processing time: 2–4 minutes for a 30-minute, 3-camera episode (vs. 4–8 hours manually)
| # | File | Description |
|---|---|---|
| — | README.md | Master index (this file) |
| 01 | 01-product-overview.md | Vision, problem, users, modules, competitive analysis |
| 02a | 02-architecture/overview.md | Three-layer architecture, services, ports, env vars |
| 02b | 02-architecture/data-flows.md | Master pipeline, EDL schema, WebSocket protocol, caching |
| 02c | 02-architecture/cloud-apis.md | All cloud API integrations, prompts, costs, error handling |
| 03-F1 | 03-features/F1-multicam-switching.md | Multicam Switching — speaker diarization, LLM camera selection, cut rules |
| 03-F2 | 03-features/F2-smart-zoom.md | Smart Zoom & Animation — keyframe generation, push-in/pull-out, easing |
| 03-F3 | 03-features/F3-silence-removal.md | Silence & Filler Removal — word-level detection, dead air, um/uh removal |
| 03-F4 | 03-features/F4-color-grading.md | Color Grading — LUT-based and AI-driven grade, camera matching |
| 03-F5 | 03-features/F5-sound-design.md | Sound Design — ElevenLabs SFX V2 integration, context-aware placement |
| 03-F6 | 03-features/F6-viral-clips.md | Viral Clip Detector — virality scoring, 30/60/90s exports, aspect ratios |
| 03-F7 | 03-features/F7-audio-cleanup.md | Audio Cleanup — ElevenLabs Voice Isolator, RNNoise offline fallback |
| 03-F8 | 03-features/F8-transitions.md | Transitions — scene-aware cuts, dissolves, whip pans, contextual rules |
| 03-F9 | 03-features/F9-script-context-input.md | Script Context Input — optional script/screenplay ingestion, Scene Intent Map, AI context injection |
| 04 | 04-naturalness-engine.md | Naturalness Engine — validation layer that prevents jarring AI edit decisions |
| 05 | 05-modular-architecture.md | Modular Architecture — how modules are isolated, composed, and sequenced |
| 06a | 06-setup/dev-environment.md | Dev Environment — local setup, dependencies, running the backend and plugins |
| 06b | 06-setup/davinci-resolve-plugin.md | DaVinci Resolve Plugin — install, build, OpenFX + Python scripting setup |
| 06c | 06-setup/premiere-pro-plugin.md | Premiere Pro Plugin — CEP/UXP setup, ExtendScript, WebSocket bridge, packaging |
| 07a | 07-backend/api-reference.md | Backend API Reference — all HTTP and WebSocket endpoints, request/response schemas |
| 07b | 07-backend/database-schema.md | Database Schema — SQLite tables, ORM models, Alembic migrations |
| 08 | 08-security.md | Security — API key handling, local data isolation, threat model |
| 09 | 09-deployment.md | Deployment — Docker, fly.io, AWS Lambda proxy, production checklist |
| 10a | 10-business/pricing-tiers.md | Pricing Tiers — plan structure, Stripe metering, credit system |
| 10b | 10-business/roadmap.md | Roadmap — milestone schedule, feature backlog, release targets |
Note to developers: Start with
01-product-overview.mdto understand what we are building and why. Then read02-architecture/overview.mdto understand the system shape before diving into the detailed flows and API docs.
CutPilot uses a three-layer architecture. Layers 1 and 2 run entirely on the user's local machine. Layer 3 is cloud-based and accessed over HTTPS.
┌─────────────────────────────────────────────────────────────────────────┐
│ USER'S LOCAL MACHINE │
│ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 1 — NLE PLUGIN (local) │ │
│ │ │ │
│ │ ┌─────────────────────┐ ┌──────────────────────────────┐ │ │
│ │ │ DaVinci Resolve │ │ Adobe Premiere Pro │ │ │
│ │ │ Plugin │ │ Extension │ │ │
│ │ │ (C++ OpenFX + │ │ (CEP panel / UXP plugin) │ │ │
│ │ │ Python Scripting) │ │ │ │ │
│ │ └─────────┬───────────┘ └──────────────┬───────────────┘ │ │
│ │ │ │ │ │
│ │ └──────────────┬────────────────┘ │ │
│ │ │ WebSocket ws://localhost:8765 │ │
│ └────────────────────────────┼───────────────────────────────────-┘ │
│ │ │
│ ┌────────────────────────────┼────────────────────────────────────┐ │
│ │ LAYER 2 — ORCHESTRATION SERVICE (local) │ │
│ │ │ │ │
│ │ ┌───────────────▼──────────────────┐ │ │
│ │ │ FastAPI + WebSocket Server │ │ │
│ │ │ localhost:8765 (WS) │ │ │
│ │ │ localhost:8766 (HTTP REST) │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────┐ ┌────────────┐ │ │ │
│ │ │ │ SQLite DB │ │ Redis │ │ │ │
│ │ │ │ (project │ │ Cache │ │ │ │
│ │ │ │ state) │ │ :6379 │ │ │ │
│ │ │ └─────────────┘ └────────────┘ │ │ │
│ │ └───────────────┬──────────────────┘ │ │
│ └────────────────────────────┼───────────────────────────────────┘ │
│ │ │
└───────────────────────────────┼─────────────────────────────────────── ┘
│ HTTPS (outbound only)
│
┌───────────────────────────────▼─────────────────────────────────────────┐
│ LAYER 3 — CLOUD APIs │
│ │
│ ┌───────────────┐ ┌──────────────────┐ ┌─────────────────────────┐ │
│ │ OpenAI Whisper│ │ Claude / GPT-4o │ │ ElevenLabs │ │
│ │ (Transcription│ │ (Scene analysis, │ │ SFX V2 (sound gen) │ │
│ │ $0.006/min) │ │ viral scoring, │ │ Voice Isolator (cleanup) │ │
│ │ │ │ edit decisions) │ │ │ │
│ └───────────────┘ └──────────────────┘ └─────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
| Layer | Where it runs | Owns | Communicates via |
|---|---|---|---|
| 1 — NLE Plugin | Inside DaVinci / Premiere process | Timeline manipulation, UI panel, media I/O | WebSocket to Layer 2 |
| 2 — Orchestration | Background process on local machine | AI pipeline, EDL generation, caching, cost tracking | WebSocket (plugin) + HTTPS (cloud) |
| 3 — Cloud APIs | Third-party servers | Transcription, LLM reasoning, sound generation, voice isolation | HTTPS REST |
-
Product context first. Read
01-product-overview.mdin full before touching any code. It explains the problem, users, modules, and what "success" looks like. -
Architecture before implementation. Read
02-architecture/overview.mdnext. Understand the three-layer shape, where each service runs, and how they talk to each other. -
Data flows are the blueprint.
02-architecture/data-flows.mdcontains the numbered master pipeline that every module follows. Every feature you build must fit this pipeline. -
APIs last.
02-architecture/cloud-apis.mdis a reference document. Read the sections relevant to whichever module you are implementing. -
One module at a time. Each feature module (F1–F8) is designed to be implemented independently. Implement F3 (Silence Removal) before F1 (Multicam) if you want — each module consumes the same EDL format and the same pipeline entry points.
| ID | Module | Priority | One-line description |
|---|---|---|---|
| F1 | Multicam Switching | P0 | AI selects the best camera angle at each moment using speaker-diarization and LLM scene analysis |
| F2 | Smart Zoom & Animation | P0 | Generates dynamic push-in/pull-out keyframe animations to create visual energy on static shots |
| F3 | Silence & Filler Removal | P0 | Detects and removes dead air, "ums", "uhs", and false starts at word-level precision |
| F4 | Color Grading | P1 | Applies LUT-based or AI-driven color grade per camera, matching all sources to a reference look |
| F5 | Sound Design | P1 | Generates context-aware sound effects via ElevenLabs SFX V2 and places them on the timeline |
| F6 | Viral Clip Detector | P0 | Scores transcript moments for virality, exports 30/60/90s clips formatted per platform (9:16, 1:1, 16:9) |
| F7 | Audio Cleanup | P0 | Removes background noise and isolates voice using ElevenLabs Voice Isolator (or local RNNoise offline) |
| F8 | Transitions | P1 | Inserts scene-aware transitions (cuts, dissolves, whip pans) based on contextual rules from the edit plan |
| Layer | Component | Technology |
|---|---|---|
| Plugin — DaVinci | NLE integration | C++ (OpenFX API) + Python (DaVinci Scripting API) |
| Plugin — Premiere | NLE integration | CEP (HTML/JS panel) + UXP (plugin layer) |
| Plugin — UI Panel | In-app UI | React (embedded in CEP/UXP) |
| Backend | Orchestration service | Python 3.11, FastAPI, Uvicorn |
| Backend | Real-time comms | WebSocket (native FastAPI support) |
| Backend | Local database | SQLite via SQLAlchemy (project & session state) |
| Backend | Cache | Redis 7 (transcription results, generated sounds, LLM responses) |
| Backend | Task queue | Python asyncio + optional Celery for long jobs |
| APIs | Transcription | OpenAI Whisper (whisper-1), $0.006/min |
| APIs | LLM reasoning | Claude 3.5 Sonnet (primary) / GPT-4o Mini (fallback) |
| APIs | Sound generation | ElevenLabs SFX V2, ~$0.02–0.06/sound |
| APIs | Voice isolation | ElevenLabs Voice Isolator |
| Payments | Subscription billing | Stripe (subscription + usage-based metering) |
| Frontend | Web dashboard | React 18, Next.js 14 (App Router), Tailwind CSS |
| Frontend | Auth | Clerk or NextAuth.js |
| Deployment | Backend (optional cloud) | Docker, fly.io or AWS Lambda (for API proxy only) |
| Dev tooling | Linting / formatting | Ruff (Python), ESLint + Prettier (JS/TS) |
| Dev tooling | Testing | pytest (backend), Vitest (frontend) |
cutpilot/
├── cutpilot-plugin/ # DaVinci Resolve plugin (C++ OpenFX + Python)
│ ├── src/
│ │ ├── openFX/ # C++ OpenFX plugin effects (zoom, color nodes)
│ │ ├── scripting/ # Python scripts using DaVinci Resolve API
│ │ │ ├── main.py # Plugin entry point
│ │ │ ├── edl_executor.py # Reads EDL and applies to timeline
│ │ │ ├── audio_extract.py# Exports audio tracks as WAV to temp dir
│ │ │ ├── ws_client.py # WebSocket client → FastAPI backend
│ │ │ └── modules/ # Per-module timeline executors
│ │ │ ├── multicam.py
│ │ │ ├── zoom.py
│ │ │ ├── silence.py
│ │ │ ├── color.py
│ │ │ ├── sound.py
│ │ │ ├── clips.py
│ │ │ ├── audio_cleanup.py
│ │ │ └── transitions.py
│ │ └── panel/ # CEP-style panel UI (React, bundled)
│ ├── build/ # Compiled plugin output
│ └── tests/
│
├── cutpilot-premiere/ # Premiere Pro extension (CEP / UXP)
│ ├── src/
│ │ ├── panel/ # UXP plugin panel (React)
│ │ ├── host/ # ExtendScript / UXP host-side code
│ │ │ ├── edl_executor.jsx
│ │ │ ├── audio_extract.jsx
│ │ │ └── ws_client.js
│ │ └── manifest.json # UXP manifest
│ ├── build/
│ └── tests/
│
├── cutpilot-backend/ # FastAPI orchestration service
│ ├── app/
│ │ ├── main.py # FastAPI app + WebSocket endpoint
│ │ ├── config.py # Settings, env var loading
│ │ ├── database.py # SQLite setup via SQLAlchemy
│ │ ├── cache.py # Redis client + cache helpers
│ │ ├── pipeline/ # Core AI pipeline
│ │ │ ├── orchestrator.py # Master pipeline controller
│ │ │ ├── transcription.py# Whisper API integration
│ │ │ ├── vision.py # Keyframe extraction + description
│ │ │ ├── llm.py # Claude / GPT-4o integration
│ │ │ ├── sound.py # ElevenLabs SFX integration
│ │ │ ├── isolator.py # ElevenLabs Voice Isolator
│ │ │ ├── edl_builder.py # Builds final EDL JSON from LLM output
│ │ │ └── naturalness.py # Naturalness engine — validates decisions
│ │ ├── modules/ # Per-module pipeline logic
│ │ │ ├── multicam.py
│ │ │ ├── zoom.py
│ │ │ ├── silence.py
│ │ │ ├── color.py
│ │ │ ├── sound_design.py
│ │ │ ├── viral_clips.py
│ │ │ ├── audio_cleanup.py
│ │ │ └── transitions.py
│ │ ├── models/ # SQLAlchemy ORM models
│ │ │ ├── project.py
│ │ │ ├── session.py
│ │ │ └── cost_ledger.py
│ │ ├── routers/ # FastAPI HTTP routers
│ │ │ ├── jobs.py
│ │ │ ├── projects.py
│ │ │ └── settings.py
│ │ └── schemas/ # Pydantic schemas
│ │ ├── edl.py
│ │ ├── job.py
│ │ └── websocket.py
│ ├── tests/
│ ├── alembic/ # DB migrations
│ ├── requirements.txt
│ ├── .env.example
│ └── Dockerfile
│
├── cutpilot-frontend/ # Web dashboard (React / Next.js)
│ ├── app/ # Next.js App Router
│ │ ├── (auth)/ # Auth routes
│ │ ├── dashboard/ # Main dashboard
│ │ ├── projects/ # Project history
│ │ ├── settings/ # API keys, preferences
│ │ └── billing/ # Stripe billing portal
│ ├── components/
│ ├── lib/
│ ├── public/
│ ├── package.json
│ └── tailwind.config.ts
│
├── cutpilot-site/ # Marketing landing page
│ ├── app/
│ ├── components/
│ └── package.json
│
└── docs/ # This documentation
├── README.md # Master index (this file)
├── 01-product-overview.md
├── 02-architecture/
│ ├── overview.md
│ ├── data-flows.md
│ └── cloud-apis.md
├── 03-features/
│ ├── F1-multicam-switching.md
│ ├── F2-smart-zoom.md
│ ├── F3-silence-removal.md
│ ├── F4-color-grading.md
│ ├── F5-sound-design.md
│ ├── F6-viral-clips.md
│ ├── F7-audio-cleanup.md
│ └── F8-transitions.md
├── 04-naturalness-engine.md
├── 05-modular-architecture.md
├── 06-setup/
│ ├── dev-environment.md
│ ├── davinci-resolve-plugin.md
│ └── premiere-pro-plugin.md # TODO: does not exist yet — see ToC note above
├── 07-backend/
│ ├── api-reference.md
│ └── database-schema.md
├── 08-security.md
├── 09-deployment.md
└── 10-business/
├── pricing-tiers.md
└── roadmap.md
All required environment variables are documented fully in 02-architecture/cloud-apis.md. For a quick local-dev setup:
# Required
OPENAI_API_KEY=sk-...
ELEVENLABS_API_KEY=...
# Optional (Claude is preferred over GPT-4o for editorial decisions)
ANTHROPIC_API_KEY=sk-ant-...
# Infrastructure (defaults shown)
REDIS_URL=redis://localhost:6379
DATABASE_URL=sqlite:///~/.cutpilot/projects.db
CUTPILOT_PORT=8765
CUTPILOT_HTTP_PORT=8766
# Feature flags
CUTPILOT_OFFLINE_MODE=false # Set true to use local Whisper + RNNoise
CUTPILOT_LOG_LEVEL=INFOFull setup instructions will live in each component's own README. This is the 30-second orientation.
# 1. Clone
git clone https://github.com/rhematek/cutpilot.git
cd cutpilot
# 2. Start the backend (requires Python 3.11+, Redis running)
cd cutpilot-backend
cp .env.example .env # Fill in your API keys
pip install -r requirements.txt
uvicorn app.main:app --host localhost --port 8766 --reload
# 3. Install the plugin
# DaVinci Resolve: copy cutpilot-plugin/ to your Fusion Scripts folder
# Premiere Pro: load cutpilot-premiere/ via UXP Developer Mode
# 4. Open your NLE, open the CutPilot panel, start editing.Documentation maintained by Rhematek Solutions. Last updated: 2026-03-19.