Skip to content

ESP32 PWM motor control + Python host system with Flask UI, vision tracking, ML perception, and CI/CD#5

Merged
Sa1koro merged 14 commits intofeat/py-cv-dashboardfrom
copilot/extend-face-tracking-functionality
Mar 16, 2026
Merged

ESP32 PWM motor control + Python host system with Flask UI, vision tracking, ML perception, and CI/CD#5
Sa1koro merged 14 commits intofeat/py-cv-dashboardfrom
copilot/extend-face-tracking-functionality

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 4, 2026

Upgrades the ESP32 firmware from binary digitalWrite motor control to PWM via ledcAttach/ledcWrite, and builds out the Python host system: weighted face tracking, Flask control panel, optional ML perception (MediaPipe/DeepFace), and GitHub Actions CI.

Firmware (sylvie_main.ino)

  • PWM motor control: setMotor(motor, dir, speed) using ledcWrite with 8-bit resolution (0–255), replacing digitalWrite on/off
  • OSC speed parameter: /motor1 and /motor2 now accept [dir, speed] (speed defaults to 255 for backward compat)
// Before: full speed or stop only
void setMotor(int motor, int direction) {
  digitalWrite(pinA, direction > 0 ? HIGH : LOW);
}

// After: PWM speed control via L298N
void setMotor(int motor, int direction, int speed = 255) {
  if (direction > 0) { ledcWrite(pinA, speed); ledcWrite(pinB, 0); }
}

Python Host (python_host/)

  • vision/face_tracker.py — Weighted multi-face selection: weight = area × 1/(1 + dist/max_dist). Thread-safe JPEG streaming. Multi-camera hot-switching (iPhone Continuity Camera support).
  • vision/perception.py — Lazy-loaded MediaPipe face mesh/pose + DeepFace emotion. Graceful degradation when ML deps absent. Separate requirements-ml.txt to avoid dependency hell.
  • network/osc_sender.py — Thread-safe OSC sender with override flag (manual UI blocks auto-tracking commands). stop_all() bypasses override. TFT eye animation stub reserved.
  • ui/app.py + index.html — Flask control panel: live video feed, motor dir+speed sliders, 2D XY pad (speed/jitter), LED color picker, presets, Override toggle, Tag & Save (JSONL data labeling for future training).

CI/CD (.github/workflows/python-ci.yml)

  • Core tests on Python 3.10/3.11/3.12 (no ML deps)
  • ML integration tests on 3.11
  • flake8 lint
  • 27 tests covering weight algorithm, OSC sender override logic, Flask API endpoints, perception graceful degradation

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 2 commits March 4, 2026 12:39
…rol panel, vision tracking, ML perception, CI/CD

- Step 1: Upgrade sylvie_main.ino from digitalWrite to ledcAttach/ledcWrite PWM (0-255)
- Step 2: Extend routeMotor1/routeMotor2 to accept dir + speed OSC parameters
- Step 3: Add test_osc_motor.py minimal test script
- Step 4: Weighted multi-face tracking with multi-camera support
- Step 5: Flask control panel with video, sliders, 2D XY pad, Override, Tag & Save
- Step 6: MediaPipe/DeepFace perception module with lazy loading
- Step 7: GitHub Actions CI/CD workflow for Python tests

Co-authored-by: Sa1koro <13943286+Sa1koro@users.noreply.github.com>
- Replace deprecated substr() with substring() in HTML template
- Replace alert() with inline toast notification for better UX
- Remove duplicate cv2 imports in perception.py
- Store target info separately instead of accessing private _address/_port
- Use proper FaceTracker re-instantiation instead of __init__ call
- Add permissions: contents: read to CI workflow

Co-authored-by: Sa1koro <13943286+Sa1koro@users.noreply.github.com>
Copilot AI changed the title [WIP] Extend face tracking to include Deepface and Mediapipe integration ESP32 PWM motor control + Python host system with Flask UI, vision tracking, ML perception, and CI/CD Mar 4, 2026
Sa1koro added 11 commits March 4, 2026 12:51
…iguration

- Move ESP32 firmware files to new refactored directory structure
- Update Flask server port from 5000 to 15000 in main.py
- Rename Chinese component label from "舵机" to English "Servo" in hardware guide
- Add Kait node section to hardware documentation
- Create pyproject.toml with project metadata and pyserial dependency
- Generate uv.lock file with package dependencies
- Maintain all Python test scripts and control utilities in new structure
- Implement mDNS and gateway-based device discovery mechanisms
- Add device registry with node type inference and labeling
- Create device selection and management API endpoints
- Integrate threading support for concurrent device operations
- Add raw OSC console with history logging capabilities
- Implement dynamic UI controls based on detected node types
- Replace static target configuration with dynamic device management
- Add UTF-8 encoding for training sample file operations
- Create device registry JSON configuration file
- Update frontend UI with device scanning and selection controls
- Introduced threading-based camera lifecycle state management.
- Added API endpoints for camera control: start, stop, switch, and state.
- Implemented camera index handling and switching logic.
- Updated video feed and face detection routes to check camera state.
- Enhanced `/api/faces` endpoint to include camera running state.
- Introduced complete project files for Kait Node v2 including firmware, Python debug tools, and comprehensive documentation.
- Added `requirements-kait.txt` listing project dependencies.
- Created KAIT_INDEX.md as a centralized reference for all files and usage instructions.
- Included KAIT_V2_DELIVERY_REPORT.md as a detailed summary of project deliverables, features, and improvements.
- Developed quick reference and troubleshooting guide in QUICK_REFERENCE.md.
- Implemented `kait_motion_visualization.py` for motion mode visualization and parameter exploration.
- Added Python scripts for OSC debugging, serial debugging, and motion visualization tools.
- Created `API_REFERENCE.md`, detailing OSC commands, motion modes, sequences, and firmware APIs.
- Added `DELIVERY_CHECKLIST.md`, providing a thorough package validation and content summary.
- Completed English translation for all documentation and scripts.
- Updated `kait_osc_debug_en.py` with sequence functions and an interactive mode.
- Implemented motion sequence recorder with label-based organization.
- Added `/api/sequences` endpoints for saving, loading, and listing sequences.
- Extended Flask app tests to cover sequence API.
- Updated frontend with sequence recording, playback features, and status tracking.
- Enhanced UI accessibility with `aria-label` attributes.
- Adjusted camera selection route to use specific backend for macOS compatibility.
- Refined Kait node UI with motion modes and motor speed controls.
…munication support

- Implemented `CoordinatePublisher` for face-tracking data via OSC or USB serial.
- Added `SerialCoordinateSender` with pyserial integration and serial port handling.
- Introduced `/api/tracking/config` and `/api/serial/ports` endpoints for coordinate tracking and serial communication.
- Updated UI with tracking transport switch and serial connection status.
- Added Arduino-based ESP32 firmware for servo-based face tracking via OSC or serial.
- Introduced `/api/discovery/*` compatibility routes for mDNS, gateway, and auto discovery modes.
- Refactored device scan logic into a reusable `_scan_and_register_devices` function.
- Added UI support for manual device addition via IP and port.
- Implemented fallback CSS for offline usage when Tailwind CDN is unavailable.
- Updated tests to cover new discovery APIs and scenarios.
…UI controls

- Implemented lazy loading for pyserial modules to optimize app startup time.
- Added `/api/serial/raw` endpoint for sending raw serial commands and handling errors.
- Updated `/api/serial/ports` endpoint to support optional scanning with `scan` query parameter.
- Enhanced UI with serial debug command field and improved manual motor and drive pad controls.
- Updated tests to cover new API endpoints and serial behaviors.
…d Sylvie motor functions

- Simplified camera list population logic and improved error handling.
- Removed deprecated Sylvie motor and drive pad functionality.
- Introduced helper functions for refreshing devices and tracking configuration.
- Enhanced LED, OSC, and device control operations for clarity and consistency.
- Added fallback logic to improve device list handling during scans.
- Added deadband and minimum effective threshold logic for motor inputs to improve precision.
- Introduced crosshair and mono font styles to the drive pad for better usability and feedback.
- Implemented snapping and clamping for motor values to ensure consistent control behavior.
- Added new manual and stop button behaviors with improved state handling.
- Included preset 3 button for extended functionality.
@Sa1koro Sa1koro merged commit 1141d89 into feat/py-cv-dashboard Mar 16, 2026
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants