ESP32 PWM motor control + Python host system with Flask UI, vision tracking, ML perception, and CI/CD#5
Merged
Sa1koro merged 14 commits intofeat/py-cv-dashboardfrom Mar 16, 2026
Conversation
…rol panel, vision tracking, ML perception, CI/CD - Step 1: Upgrade sylvie_main.ino from digitalWrite to ledcAttach/ledcWrite PWM (0-255) - Step 2: Extend routeMotor1/routeMotor2 to accept dir + speed OSC parameters - Step 3: Add test_osc_motor.py minimal test script - Step 4: Weighted multi-face tracking with multi-camera support - Step 5: Flask control panel with video, sliders, 2D XY pad, Override, Tag & Save - Step 6: MediaPipe/DeepFace perception module with lazy loading - Step 7: GitHub Actions CI/CD workflow for Python tests Co-authored-by: Sa1koro <13943286+Sa1koro@users.noreply.github.com>
- Replace deprecated substr() with substring() in HTML template - Replace alert() with inline toast notification for better UX - Remove duplicate cv2 imports in perception.py - Store target info separately instead of accessing private _address/_port - Use proper FaceTracker re-instantiation instead of __init__ call - Add permissions: contents: read to CI workflow Co-authored-by: Sa1koro <13943286+Sa1koro@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Extend face tracking to include Deepface and Mediapipe integration
ESP32 PWM motor control + Python host system with Flask UI, vision tracking, ML perception, and CI/CD
Mar 4, 2026
…iguration - Move ESP32 firmware files to new refactored directory structure - Update Flask server port from 5000 to 15000 in main.py - Rename Chinese component label from "舵机" to English "Servo" in hardware guide - Add Kait node section to hardware documentation - Create pyproject.toml with project metadata and pyserial dependency - Generate uv.lock file with package dependencies - Maintain all Python test scripts and control utilities in new structure
- Implement mDNS and gateway-based device discovery mechanisms - Add device registry with node type inference and labeling - Create device selection and management API endpoints - Integrate threading support for concurrent device operations - Add raw OSC console with history logging capabilities - Implement dynamic UI controls based on detected node types - Replace static target configuration with dynamic device management - Add UTF-8 encoding for training sample file operations - Create device registry JSON configuration file - Update frontend UI with device scanning and selection controls
- Introduced threading-based camera lifecycle state management. - Added API endpoints for camera control: start, stop, switch, and state. - Implemented camera index handling and switching logic. - Updated video feed and face detection routes to check camera state. - Enhanced `/api/faces` endpoint to include camera running state.
- Introduced complete project files for Kait Node v2 including firmware, Python debug tools, and comprehensive documentation. - Added `requirements-kait.txt` listing project dependencies. - Created KAIT_INDEX.md as a centralized reference for all files and usage instructions. - Included KAIT_V2_DELIVERY_REPORT.md as a detailed summary of project deliverables, features, and improvements. - Developed quick reference and troubleshooting guide in QUICK_REFERENCE.md. - Implemented `kait_motion_visualization.py` for motion mode visualization and parameter exploration. - Added Python scripts for OSC debugging, serial debugging, and motion visualization tools.
- Created `API_REFERENCE.md`, detailing OSC commands, motion modes, sequences, and firmware APIs. - Added `DELIVERY_CHECKLIST.md`, providing a thorough package validation and content summary. - Completed English translation for all documentation and scripts. - Updated `kait_osc_debug_en.py` with sequence functions and an interactive mode.
- Implemented motion sequence recorder with label-based organization. - Added `/api/sequences` endpoints for saving, loading, and listing sequences. - Extended Flask app tests to cover sequence API. - Updated frontend with sequence recording, playback features, and status tracking. - Enhanced UI accessibility with `aria-label` attributes. - Adjusted camera selection route to use specific backend for macOS compatibility. - Refined Kait node UI with motion modes and motor speed controls.
…munication support - Implemented `CoordinatePublisher` for face-tracking data via OSC or USB serial. - Added `SerialCoordinateSender` with pyserial integration and serial port handling. - Introduced `/api/tracking/config` and `/api/serial/ports` endpoints for coordinate tracking and serial communication. - Updated UI with tracking transport switch and serial connection status. - Added Arduino-based ESP32 firmware for servo-based face tracking via OSC or serial.
- Introduced `/api/discovery/*` compatibility routes for mDNS, gateway, and auto discovery modes. - Refactored device scan logic into a reusable `_scan_and_register_devices` function. - Added UI support for manual device addition via IP and port. - Implemented fallback CSS for offline usage when Tailwind CDN is unavailable. - Updated tests to cover new discovery APIs and scenarios.
…UI controls - Implemented lazy loading for pyserial modules to optimize app startup time. - Added `/api/serial/raw` endpoint for sending raw serial commands and handling errors. - Updated `/api/serial/ports` endpoint to support optional scanning with `scan` query parameter. - Enhanced UI with serial debug command field and improved manual motor and drive pad controls. - Updated tests to cover new API endpoints and serial behaviors.
…d Sylvie motor functions - Simplified camera list population logic and improved error handling. - Removed deprecated Sylvie motor and drive pad functionality. - Introduced helper functions for refreshing devices and tracking configuration. - Enhanced LED, OSC, and device control operations for clarity and consistency. - Added fallback logic to improve device list handling during scans.
- Added deadband and minimum effective threshold logic for motor inputs to improve precision. - Introduced crosshair and mono font styles to the drive pad for better usability and feedback. - Implemented snapping and clamping for motor values to ensure consistent control behavior. - Added new manual and stop button behaviors with improved state handling. - Included preset 3 button for extended functionality.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades the ESP32 firmware from binary
digitalWritemotor control to PWM vialedcAttach/ledcWrite, and builds out the Python host system: weighted face tracking, Flask control panel, optional ML perception (MediaPipe/DeepFace), and GitHub Actions CI.Firmware (
sylvie_main.ino)setMotor(motor, dir, speed)usingledcWritewith 8-bit resolution (0–255), replacingdigitalWriteon/off/motor1and/motor2now accept[dir, speed](speed defaults to 255 for backward compat)Python Host (
python_host/)vision/face_tracker.py— Weighted multi-face selection:weight = area × 1/(1 + dist/max_dist). Thread-safe JPEG streaming. Multi-camera hot-switching (iPhone Continuity Camera support).vision/perception.py— Lazy-loaded MediaPipe face mesh/pose + DeepFace emotion. Graceful degradation when ML deps absent. Separaterequirements-ml.txtto avoid dependency hell.network/osc_sender.py— Thread-safe OSC sender with override flag (manual UI blocks auto-tracking commands).stop_all()bypasses override. TFT eye animation stub reserved.ui/app.py+index.html— Flask control panel: live video feed, motor dir+speed sliders, 2D XY pad (speed/jitter), LED color picker, presets, Override toggle, Tag & Save (JSONL data labeling for future training).CI/CD (
.github/workflows/python-ci.yml)💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.