SixthSense - First Place at HackPrinceton
Over 253 million people worldwide live with visual impairments. SixthSense redefines assistive technology by introducing a groundbreaking dual-feedback system that allows users to "feel" their environment through touch without overloading their hearing.
By combining Meta glasses, YOLOv8 object detection, and custom haptic feedback through a wearable glove and gauntlet, SixthSense lets users perceive both what an object is and where it is, providing a tactile language that becomes intuitive over time.
- Visual Recognition: Captures real-world scenes using Meta glasses
- Object Detection: Identifies objects using YOLOv8
- Tactile Signatures: Converts object embeddings into unique vibration patterns using inverse Fourier transforms
- Directional Guidance: Uses servo motors to physically guide users' hands toward object locations
- Simultaneous Sensory Composition: Merges multiple objects’ tactile signals into a harmonic, non-overwhelming experience
- Brain-Inspired Architecture: Separates object identity ("what") from location ("where"), mirroring human visual processing
- Embedding-to-Vibration Mapping: Translates semantic object vectors into unique, recognizable tactile patterns
- Harmonic Touch Language: Similar objects produce related sensations, allowing users to intuitively understand complex scenes
- Minimal Auditory Interference: Preserves hearing by relying solely on touch for feedback
-
YOLO Detection Stream
Python-based MJPEG streaming server with WebSocket support to detect and serve object data in real-time. -
Next.js Dashboard
Web interface for developers and testers to visualize detections and track statistics. -
Haptic Feedback System
Arduino-controlled glove and gauntlet delivering dual feedback: vibration patterns (identity) and servo-based guidance (location).
Python,YOLOv8,OpenCV,Arduino,Next.js,Flask,WebSockets,Fourier Transforms
- Devpost Submission: [https://devpost.com/software/sixthsense-xuw41r]
- Demo Video: [https://www.youtube.com/watch?v=vNlNHXVv8kw]