While discussing the joy instrumental music brings to people, our team realized that learning and playing instruments isn't equally accessible to everyone. Traditional lessons and instruments can be expensive, and many digital music tools assume the user can physically interact with a keyboard, mouse, or instrument in conventional ways.
The word "Ayre" originates from the Renaissance era and refers to a genre of solo performed songs, which perfectly encompasses the user's experience with our product.
For QuackHacks 2026, we specifically built this project for the Accessibility track: traditional music creation is locked behind expensive gear and years of practice. Ayre removes both barriers. It runs in a browser or as a desktop app, requires no installation beyond a webcam, and includes features specifically designed for users with motor impairments.
- Hand 1 controls the arpeggio — raise your hand to raise the pitch, pinch your thumb and index finger to control volume. Make a fist to cycle synth presets.
- Hand 2 controls the instrument sounds — raise individual fingers to toggle kick, snare, hi-hat, clap patterns, etc.
Both hands are tracked in real time via your webcam. No touching required.
- Tremor smoothing — an adjustable stabilization slider reduces jitter caused by hand tremors or conditions like Parkinson's or muscle weakness.
- One-hand mode — redesigned control scheme so instrument sounds and volume adjustment are controllable from a single hand, for users with limb differences or impairments.
- Voice-only mode — trigger instrument sounds and change synth presets via voice commands using the Web Speech API, for users who cannot use hand gestures at all.
- Customizable finger assignment — reassign which finger controls which instrument. Accommodates non-standard hand anatomy or personal preference.
- No specialized hardware required — runs on any modern laptop or desktop with a webcam and microphone.
The backend handles AI synthesizer preset generation.
cd ayre
npm install
# Create a .env file and add your GEMINI_API_KEY
npm run serverThe server will run at http://localhost:3000.
In the browser:
# From the root directory
python -m http.server 8000
Navigate to http://localhost:8000.
As a desktop app (Electron):
npm start
To build a distributable .app / .exe:
npm install electron-builder --save-dev
npx electron-builder
- MediaPipe — hand tracking and landmark detection
- Tone.js — synthesizer, sequencer, and audio effects
- Three.js — real-time WebGL waveform visualizer
- Electron — desktop app wrapper
- Gemini API — automates synthesizer presets based on user input
- Hosting — Vercel (frontend) and Railway (backend)
- Web Speech API — voice command support
- Modern browser with WebGL support (Chrome recommended)
- Webcam and microphone access
MIT
Inspired by Fun with Computer Vision, Ayre extends computer vision capabilities with an accessibility-focused UI, tremor smoothing controls, customizable finger assignment, and one-hand or voice-only modes.
- Three.js — https://threejs.org/
- MediaPipe — https://mediapipe.dev/
- Tone.js — https://tonejs.github.io/
- ESM.sh — Used for modern package management and imports.
- Vaibhav Achuthananda
- Shasha Alvares
- Ravisara Cheakdkaipejchara
Two Hand Demo 
One Hand Demo 
Voice Command Control Panel 