diff --git a/README.md b/README.md new file mode 100644 index 0000000..e88a249 --- /dev/null +++ b/README.md @@ -0,0 +1,511 @@ +# The Garden's Response - Interactive AI Flower Installation +## 机械花朵交互式 AI 艺术装置 + +[English Version](./docs/PORTFOLIO_EN.md) + +![ESP32](https://img.shields.io/badge/ESP32-Embedded%20Systems-111111?style=flat-square) +![Physical Computing](https://img.shields.io/badge/Physical%20Computing-Kinetic%20Installation-2f855a?style=flat-square) +![Computer Vision](https://img.shields.io/badge/Computer%20Vision-OpenCV%20%2B%20Face%20Tracking-2563eb?style=flat-square) +![Emotion AI](https://img.shields.io/badge/Emotion%20AI-DeepFace%20%2B%20ViT-7c3aed?style=flat-square) +![Robotics](https://img.shields.io/badge/Robotics-Servo%20%2B%20DC%20Motors-d97706?style=flat-square) +![OSC](https://img.shields.io/badge/OSC-WiFi%20Control-0f766e?style=flat-square) +![UI/UX](https://img.shields.io/badge/UI%2FUX-Hardware%20Dashboard-db2777?style=flat-square) + +## 项目概述 + +**The Garden's Response** 是 York University Digital Media Program **DATT 3700: Collaborative Project Development** 课程中的互动机械花装置。项目被选中参加 **YES!! - Year End Showcase!!**,在 InterAccess / Digital Media Year 2026 展出:[Digital Media Gallery - YES!! Year End Showcase](https://dmgallery.apps01.yorku.ca/yes-year-end-showcase/)。 + +这个作品将 Python 感知控制端、表情识别、OSC 网络通信与多个 ESP32 物理节点结合起来。主机通过摄像头追踪人脸并分析情绪,再把数字感知结果转译为机械花的伺服、电机、屏幕动画和预设动作,让花朵以一种具身的方式回应观众。 + +> Showcase 官方描述:The Garden's Response is an interactive kinetic flower installation that integrates computer vision, emotion AI, and distributed hardware nodes. The system utilizes a central Python-based host to track human faces and analyze emotional states, translating these digital perceptions into physical movements across a network of ESP32-powered flower nodes. + +### 项目链接 + +- Showcase 页面:[YES!! Year End Showcase - The Garden's Response](https://dmgallery.apps01.yorku.ca/yes-year-end-showcase/) +- 主仓库:[dattodigm/DATT3700](https://github.com/dattodigm/DATT3700) +- 调试控制面板在线演示:https://esp32garden.saikoro.me/ + - 分支源码:[v0/kaminodice-06183338](https://github.com/dattodigm/DATT3700/tree/v0/kaminodice-06183338) +- GC9D01/TFT_eSPI 显示驱动仓库:[Arduino_GC9D01_Driver_TFT_eSPI_Lvgl](https://github.com/dattodigm/Arduino_GC9D01_Driver_TFT_eSPI_Lvgl) +- RGB565 眼睛动画预览/转换工具:[EyeSeed](https://github.com/dattodigm/EyeSeed) + +## 系统架构 + +```text +Camera / iPhone Continuity Camera + | + v +Python Host (Flask + OpenCV + optional ML) + | + |-- FaceTracker: Haar 人脸检测,多人人脸面积加权中心 + |-- PerceptionModule: DeepFace、MediaPipe 预留、ViT 表情识别 + |-- EmotionReactor: 有损积分、状态机、OSC 路由 + |-- Web Dashboard: 实时预览、控制面板、动作录制、OSC Console + | + v +OSC over WiFi / USB serial fallback + | + v +ESP32 Flower Nodes + | + |-- Sylvie: AP/gateway、DC 电机、RGB LED、预设动作 + |-- Kait: DC 电机速度与运动模式 + |-- Sue: 舵机花朵开合、GC9A01 眼睛、眨眼/呼吸/追踪 + |-- Face Tracking Node: 8 舵机、4 组 pan/tilt 花朵追踪 + |-- Eye Anime Node: GC9D01/TFT_eSPI 眼睛动画与追踪 +``` + +### Early Interaction Concept + +下图是项目早期的交互概念草图,用来表达“人-感知层-机器代理-物理反馈”的循环关系。最终实现从这个概念收敛为摄像头人脸追踪、表情识别、Python Host 和 ESP32 花朵节点网络。 + +```mermaid +sequenceDiagram + participant User as 🧘 User (Embodied) + participant Sensors as 📡 Input Layer (Phone/Sensors) + participant AI as 🧠 Machine Agent (The "Soul") + participant Output as 🌸 Physical/Visual Body + + Note over User, Output: Theme: Interpersonal Connection in Third Spaces + + User->>Sensors: Enters Space / Touches Wall / Moves Phone + Sensors->>AI: Transmits Raw Data (Gyro, Distance, Touch) + + rect rgb(240, 248, 255) + Note right of AI: Analysis & Interpretation + AI->>AI: Interpret "Intention" (Wekinator)
(e.g., Aggressive vs. Gentle) + AI->>AI: Determine "Emotional State"
(e.g., Bloom, Withdraw, Pulse) + end + + AI->>Output: Send Control Commands (OSC/Serial) + + par Parallel Feedback + Output->>User: 🌺 3D Printer Component(Physical) + Output->>User: 💡 LED Color Shift (Atmosphere) + Output->>User: 📽️ Visual Projections (Digital) + end + + User->>Sensors: Reacts to Feedback (Loop Continues) +``` + +## 核心实现 + +### Python 控制端与 Web Dashboard + +主机端代码在 `python_host/`。基础运行方式: + +```bash +python -m pip install -r python_host/requirements.txt +python -m python_host.main --port 15000 +``` + +`python_host/ui/app.py` 中的 Flask 应用提供: + +- 摄像头生命周期管理:启动、停止、切换摄像头,以及镜像预览对应的 X 方向反转。 +- mDNS 设备扫描与 Gateway fallback 扫描。 +- 通过 `python_host/ui/device_registry.json` 做节点类型识别与已知设备注册。 +- 根据节点类型渲染不同控制面板:`sylvie`、`kait`、`sue`、`face_track`、`eye_anime`。 +- Raw OSC Console、发送历史和自动刷新。 +- 动作序列录制与回放,保存到 `python_host/data/sequences/