Add web UI for video object tracking and inpainting

[abhinaya-gov]

Summary

This PR introduces a complete web application for interactive video object tracking and inpainting using Flask. Users can upload videos, segment objects using SAM (Segment Anything Model), track them across frames using CUTIE, and remove them using ProPainter inpainting.

Key Changes

• Flask Backend (app.py): Implements REST API endpoints for:

  • Video upload and frame extraction
  • Frame selection and SAM-based interactive segmentation via clicks
  • Multi-mask management for tracking multiple objects
  • VOS (Video Object Segmentation) tracking using CUTIE
  • ProPainter-based video inpainting with configurable parameters
  • Session management for handling multiple users

• Frontend (index.html, app.js, style.css):

  • Interactive canvas for clicking on objects to create masks
  • Frame slider controls for selecting template and end frames
  • Toggle between positive/negative clicks for SAM refinement
  • Real-time mask visualization with color-coded tags
  • Collapsible parameters panel for inpainting tuning
  • Loading overlays and status messages for long-running operations
  • Dark theme UI with responsive design

• Dependencies (requirements.txt): Specifies all required packages including PyTorch, OpenCV, Flask, and Segment Anything

Notable Implementation Details

• Session Management: Uses Flask sessions with UUID-based tracking to support multiple concurrent users
• Memory Efficiency: Monitors system memory during frame extraction to prevent OOM crashes
• Base64 Encoding: Frames are transmitted as JPEG base64 strings for efficient web transfer
• Canvas Scaling: Maintains aspect ratio when rendering frames on the canvas
• Multi-mask Support: Allows users to create and track multiple objects simultaneously with distinct color coding
• Configurable Tracking: End frame selection enables partial video processing
• Parameter Tuning: Exposes ProPainter inpainting parameters (resize ratio, dilation, RAFT iterations, etc.) via sliders

Workflow

1. Upload video → Extract frames
2. Select template frame → Click to segment objects with SAM
3. Refine masks with positive/negative clicks → Add to multi-mask set
4. Run CUTIE tracking across frames
5. Run ProPainter inpainting with tuned parameters
6. Download result video

https://claude.ai/code/session_01HFrWikq7HDQfQGXhVYepqW