+ :5000`. You should be able to see "Hello World" on the webpage.
+
+
+
+---
-### Storyboard
+## Part 1-A. Storyboard - *"Fairy Mate" Concepts*
-Storyboard and/or use a Verplank diagram to design a speech-enabled device. (Stuck? Make a device that talks for dogs. If that is too stupid, find an application that is better than that.)
+**Design Tool:** Verplank Diagram
-\*\***Post your storyboard and diagram here.**\*\*
+Serving Pages
+ + In Lab 1, we served a webpage with flask. In this lab, you may find it useful to serve a webpage for the controller on a remote device. Here is a simple example of a webserver. + + ``` + pi@ixe00:~/Interactive-Lab-Hub/Lab 3 $ python server.py + * Serving Flask app "server" (lazy loading) + * Environment: production + WARNING: This is a development server. Do not use it in a production deployment. + Use a production WSGI server instead. + * Debug mode: on + * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) + * Restarting with stat + * Debugger is active! + * Debugger PIN: 162-573-883 + ``` + From a remote browser on the same network, check to make sure your webserver is working by going to `http://
-Write out what you imagine the dialogue to be. Use cards, post-its, or whatever method helps you develop alternatives or group responses.
-\*\***Please describe and document your process.**\*\*
+### 💡 Idea
+Modern life often leaves people emotionally disconnected, even when surrounded by others.
+**Fairy Mate** is a friendly, speech-enabled companion that helps users express and reflect on their feelings.
-### Acting out the dialogue
+### 🌈 Scenario
+1. Fairy Mate greets the user: *“How was your day?”*
+2. The user shares their thoughts and emotions.
+3. Fairy Mate offers empathetic responses or practical advice.
+4. The device automatically creates a **journal entry** summarizing the conversation.
-Find a partner, and *without sharing the script with your partner* try out the dialogue you've designed, where you (as the device designer) act as the device you are designing. Please record this interaction (for example, using Zoom's record feature).
+### ⚙️ Key Functions
+- **Emotion Detection:** Analyzes environmental and biological signals to gauge mood.
+- **Emotional Support:** Provides comfort and encouragement.
+- **Personal Journaling:** Logs conversations and emotions for long-term reflection.
-\*\***Describe if the dialogue seemed different than what you imagined when it was acted out, and how.**\*\*
+---
-### Wizarding with the Pi (optional)
-In the [demo directory](./demo), you will find an example Wizard of Oz project. In that project, you can see how audio and sensor data is streamed from the Pi to a wizard controller that runs in the browser. You may use this demo code as a template. By running the `app.py` script, you can see how audio and sensor data (Adafruit MPU-6050 6-DoF Accel and Gyro Sensor) is streamed from the Pi to a wizard controller that runs in the browser `http://
-
+> 💬 What are concrete things that could use improvement in the design of your device? For example: wording, timing, anticipation of misunderstandings
+
+| Aspect | Observation | Suggested Improvement |
+|--------|--------------|-----------------------|
+| **Wording** | Fairy Mate often gives support too quickly. | Confirm the user’s emotional tone first. |
+| **Timing** | Responses can feel rushed or abrupt. | Add short pauses and detect conversation endings gracefully. |
+
+
+### 🖐️ 2. Beyond Speech — Expanding Interaction Modes
+> 💬 What are other modes of interaction _beyond speech_ that you might also use to clarify how to interact?
+
+To enhance intuitiveness, we added **touch** and **visual feedback** alongside voice
+
+#### 📝 Mode Overview
+- 🩵 **Emotional Support Mode** — Comforting, empathetic responses.
+- 💚 **Solution Support Mode** — Practical suggestions or motivational help.
+
+#### 🧲 Touch Controls
+Using an **Adafruit MPR121 capacitive touch sensor**:
+
+| Pad Range | Mode | Example Behavior |
+|------------|------|------------------|
+| 1–5 | Emotional Support | “I hear you. That sounds really tough. Would you like to take a deep breath together?” |
+| 6–10 | Solution Support | “Let’s think of one small thing you could do right now to feel better.” |
-## Test the system
-Try to get at least two people to interact with your system. (Ideally, you would inform them that there is a wizard _after_ the interaction, but we recognize that can be hard.)
+#### 💡 Visual Feedback
+- 🔵 **Blue Light** → Emotional Support Mode
+- 🟢 **Green Light** → Solution Support Mode
-Answer the following:
+These color cues make mode-switching clear and intuitive.
-### What worked well about the system and what didn't?
-\*\**your answer here*\*\*
+### 🧭 3. Updated Storyboard / Diagram
-### What worked well about the controller and what didn't?
+New storyboards and scripts reflect improved user flow and feedback mechanisms.
-\*\**your answer here*\*\*
+Submission Cleanup Reminder (Click to Expand)
- - **Before submitting your README.md:** - - This readme.md file has a lot of extra text for guidance. - - Remove all instructional text and example prompts from this file. - - You may either delete these sections or use the toggle/hide feature in VS Code to collapse them for a cleaner look. - - Your final submission should be neat, focused on your own work, and easy to read for grading. - - This helps ensure your README.md is clear professional and uniquely yours! -
-### What lessons can you take away from the WoZ interactions for designing a more autonomous version of the system?
+---
+
+## 🎬 Part 2-B. Demo
+
+https://github.com/user-attachments/assets/e1faef80-e93d-4fc6-898c-d23e209f9f9c
+
+
+---
+
+## 🧪 Part 2-C. System Testing & Reflections
+
+### ✅ What Worked Well
-\*\**your answer here*\*\*
+> 💬 What worked well about the system?
+1. The color-coded modes (blue for Emotional Support and green for Solution Support) made it easy to recognize the device’s state.
+2. The device could provide more personalized feedback based on tone or emotional cues, rather than relying only on preset responses.
+3. Adding clearer visual or sound cues for when the device is “listening” versus “processing” would also make the interaction feel more intuitive.
-### How could you use your system to create a dataset of interaction? What other sensing modalities would make sense to capture?
-\*\**your answer here*\*\*
+### ⚠️ What Could Improve (advice from other test users)
+> 💬 What didn't work well about the system?
+1. The touch sensor controller worked well in providing a simple and intuitive way for users to interact with Fairy Mate.
+2. The use of color feedback, blue for Emotional Support and green for Solution Support, clearly indicated which mode was active, helping users feel confident that their input was recognized.
+3. The sensitivity of the touch sensor sometimes caused accidental activations or missed touches, especially if the user’s finger didn’t make full contact with the pad.
+4. Because the pads were numbered rather than labeled with words or icons, some users had to remember which numbers corresponded to each mode.
+### 🧩 Lessons from WoZ Interactions
+
+> 💬 What lessons can you take away from the WoZ interactions for designing a more autonomous version of the system?
+
+To make Fairy Mate more autonomous:
+- Collect **anonymized interaction data** (voice, touch, response timing, user sentiment).
+- Label data by **emotional state** and **support type**.
+- Train the system to adapt its tone and timing dynamically.
+
+### 📉 Building an Interaction Dataset
+
+> 💬 How could you use your system to create a dataset of interaction? What other sensing modalities would make sense to capture?
+
+| Sensor | Purpose |
+|--------|----------|
+| 🎤 **Microphone** | Detect tone, stress, or hesitation. |
+| 📷 **Facial Recognition** | Identify expressions (smile, frown, eye contact). |
+| 🩺 **Motion Sensors (IMU)** | Track restlessness or relaxation. |
+| 🌡️ **Environmental Sensors** | Adjust feedback based on context (e.g., quiet/dark room → bedtime suggestion). |
+| ✋ **Touch Pressure** | Infer emotional intensity from contact force or duration. |
+
+---
+### 🌻 Summary
+Fairy Mate blends **speech, emotion sensing, and multi-modal feedback** to create a compassionate, responsive digital companion.
+Through iterative design, testing, and user reflection, we refined it from a concept into a system that **listens with empathy and responds with understanding**.
diff --git a/Lab 3/assets/FairyMate.png b/Lab 3/assets/FairyMate.png
new file mode 100644
index 0000000000..67add7cf7d
Binary files /dev/null and b/Lab 3/assets/FairyMate.png differ
diff --git a/Lab 3/assets/FairyMate2.png b/Lab 3/assets/FairyMate2.png
new file mode 100644
index 0000000000..72e0b7ad12
Binary files /dev/null and b/Lab 3/assets/FairyMate2.png differ
diff --git a/Lab 3/fairymate.py b/Lab 3/fairymate.py
new file mode 100644
index 0000000000..5185b32f85
--- /dev/null
+++ b/Lab 3/fairymate.py
@@ -0,0 +1,153 @@
+import time
+import board
+import busio
+import adafruit_mpr121
+import speech_recognition as sr
+import subprocess
+
+# --- Setup hardware ---
+i2c = busio.I2C(board.SCL, board.SDA)
+mpr121 = adafruit_mpr121.MPR121(i2c)
+
+import digitalio
+from PIL import Image, ImageDraw, ImageFont
+import adafruit_rgb_display.st7789 as st7789
+
+# --- ST7789 setup (adjust pins if needed) ---
+cs_pin = digitalio.DigitalInOut(board.D5)
+dc_pin = digitalio.DigitalInOut(board.D25)
+reset_pin = None
+
+BAUDRATE = 64000000
+
+spi = board.SPI()
+
+DISPLAY_WIDTH = 135
+DISPLAY_HEIGHT = 240
+disp = st7789.ST7789(
+ spi,
+ cs=cs_pin,
+ dc=dc_pin,
+ rst=reset_pin,
+ baudrate=BAUDRATE,
+ width=DISPLAY_WIDTH,
+ height=DISPLAY_HEIGHT,
+ x_offset=53,
+ y_offset=40, # adjust depending on your display orientation
+)
+
+# --- Create image buffer ---
+image = Image.new("RGB", (disp.width, disp.height))
+draw = ImageDraw.Draw(image)
+font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 20)
+
+def update_mode_display(mode):
+ # Clear the screen
+ draw.rectangle((0, 0, disp.width, disp.height), outline=0, fill=(0, 0, 0))
+
+ # Choose text and color
+ if mode == "emotional":
+ text = "Emotional\nSupport <3"
+ color = (255, 100, 150)
+ else:
+ text = "Solution\nSupport :)"
+ color = (100, 180, 255)
+
+ # Measure text size (Pillow ≥10 uses textbbox)
+ try:
+ bbox = draw.textbbox((0, 0), text, font=font)
+ w, h = bbox[2] - bbox[0], bbox[3] - bbox[1]
+ except AttributeError:
+ w, h = font.getsize(text)
+
+ # Draw centered text
+ draw.text(
+ ((disp.width - w) // 2, (disp.height - h) // 2),
+ text,
+ font=font,
+ fill=color,
+ align="center"
+ )
+
+ disp.image(image)
+
+
+# --- Speech recognition ---
+recognizer = sr.Recognizer()
+
+# --- System state ---
+mode = "solution" # default mode
+print("System ready! Default mode: Solution Support.")
+
+# --- Speak using espeak ---
+def speak(text):
+ print(f"Pi says: {text}")
+ subprocess.run(["espeak", "-s", "165", text])
+
+# --- Listen for voice input ---
+def listen():
+ with sr.Microphone() as source:
+ recognizer.adjust_for_ambient_noise(source, duration=0.5)
+ print("🎤 Listening...")
+ audio = recognizer.listen(source)
+ try:
+ text = recognizer.recognize_google(audio, language="en-US")
+ print(f"User said: {text}")
+ return text
+ except Exception:
+ print("Could not understand speech.")
+ return ""
+
+# --- Simple keyword-based mood detection ---
+def analyze_mood(user_text):
+ if not user_text:
+ return "neutral"
+ text = user_text.lower()
+ if any(w in text for w in ["good", "great", "happy", "awesome", "amazing"]):
+ return "positive"
+ elif any(w in text for w in ["bad", "sad", "tired", "angry", "terrible"]):
+ return "negative"
+ else:
+ return "neutral"
+
+def emotional_response(mood):
+ responses = {
+ "positive": "That’s wonderful to hear! I’m so glad you’re feeling good today.",
+ "neutral": "I see. It sounds like a calm day. I’m here if you’d like to share more.",
+ "negative": "I’m really sorry it’s been a tough day. Remember, it’s okay to rest and take things slow."
+ }
+ return responses[mood]
+
+def solution_response(mood):
+ responses = {
+ "positive": "Awesome! Maybe keep that momentum going with something you enjoy.",
+ "neutral": "Alright. Maybe you could set a small goal to make the day more productive.",
+ "negative": "Sounds like you’ve had a rough day. Maybe try writing down one thing you can solve step by step."
+ }
+ return responses[mood]
+
+# --- Main loop ---
+while True:
+ for i in range(12):
+ if mpr121[i].value:
+ if i < 6:
+ mode = "emotional"
+ speak("Switched to emotional support mode.")
+ else:
+ mode = "solution"
+ speak("Switched to solution support mode.")
+
+ update_mode_display(mode)
+ time.sleep(1)
+ speak("How’s your day?")
+ user_text = listen()
+ mood = analyze_mood(user_text)
+
+ if mode == "emotional":
+ speak(emotional_response(mood))
+ else:
+ speak(solution_response(mood))
+
+ print(f"Mode: {mode}, Mood: {mood}\n")
+ time.sleep(3) # debounce
+ time.sleep(0.1)
diff --git a/Lab 3/speech-scripts/ask_number.sh b/Lab 3/speech-scripts/ask_number.sh
new file mode 100755
index 0000000000..6795ce0873
--- /dev/null
+++ b/Lab 3/speech-scripts/ask_number.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+# ask_number.sh - Verbal prompt and record spoken answer
+
+OUTPUT_FILE="response.wav"
+
+# Step 1: Speak the question
+espeak "Please say your zip code."
+
+# Step 2: Record audio for 5 seconds
+echo "Recording... please speak now."
+arecord -f cd -t wav -d 5 -r 16000 -c 1 $OUTPUT_FILE
+echo "Recording saved to $OUTPUT_FILE"
+
+# Step 3: Confirm
+espeak "Thank you. Your response has been recorded."
diff --git a/Lab 3/speech-scripts/faster_whisper_try.py b/Lab 3/speech-scripts/faster_whisper_try.py
old mode 100644
new mode 100755
diff --git a/Lab 3/speech-scripts/greet.sh b/Lab 3/speech-scripts/greet.sh
new file mode 100755
index 0000000000..8e7d62b26d
--- /dev/null
+++ b/Lab 3/speech-scripts/greet.sh
@@ -0,0 +1,5 @@
+#!/bin/bash
+# greet.sh - Simple TTS greeting script using espeak
+
+NAME="Jessica Hsiao"
+espeak "Hello $NAME, welcome back to your Raspberry Pi!"
diff --git a/Lab 3/speech-scripts/response.wav b/Lab 3/speech-scripts/response.wav
new file mode 100644
index 0000000000..29add1fda7
Binary files /dev/null and b/Lab 3/speech-scripts/response.wav differ
diff --git a/Lab 3/speech-scripts/test.txt b/Lab 3/speech-scripts/test.txt
new file mode 100644
index 0000000000..a385b8f2bc
--- /dev/null
+++ b/Lab 3/speech-scripts/test.txt
@@ -0,0 +1 @@
+zero one two three four
diff --git a/Lab 3/speech-scripts/voice_chat.py b/Lab 3/speech-scripts/voice_chat.py
new file mode 100644
index 0000000000..8a8945a5ff
--- /dev/null
+++ b/Lab 3/speech-scripts/voice_chat.py
@@ -0,0 +1,50 @@
+import subprocess
+import vosk
+import sys
+import sounddevice as sd
+import ollama
+import pyttsx3
+import queue
+import json
+
+# --------------------
+# 1. Speech recognition setup
+# --------------------
+model = vosk.Model("model") # path to your vosk model (downloaded separately)
+samplerate = 16000
+q = queue.Queue()
+
+def callback(indata, frames, time, status):
+ q.put(bytes(indata))
+
+# --------------------
+# 2. Start mic stream
+# --------------------
+rec = vosk.KaldiRecognizer(model, samplerate)
+
+print("🎤 Say something... (Ctrl+C to stop)")
+with sd.RawInputStream(samplerate=samplerate, blocksize=8000, dtype='int16',
+ channels=1, callback=callback):
+ while True:
+ data = q.get()
+ if rec.AcceptWaveform(data):
+ result = rec.Result()
+ text = json.loads(result).get("text", "")
+ if text:
+ print(f"You said: {text}")
+
+ # --------------------
+ # 3. Send to Ollama
+ # --------------------
+ response = ollama.chat(model="llama3", messages=[
+ {"role": "user", "content": text}
+ ])
+ reply = response['message']['content']
+ print(f"Ollama: {reply}")
+
+ # --------------------
+ # 4. Text-to-speech
+ # --------------------
+ engine = pyttsx3.init()
+ engine.say(reply)
+ engine.runAndWait()
diff --git a/Lab 3/speech-scripts/whisper_try.py b/Lab 3/speech-scripts/whisper_try.py
old mode 100644
new mode 100755
diff --git a/Lab 4/README.md b/Lab 4/README.md
index afbb46ed98..bb28a062be 100644
--- a/Lab 4/README.md
+++ b/Lab 4/README.md
@@ -13,9 +13,9 @@
This helps ensure your README.md is clear, professional, and uniquely yours!
----
-
-## Lab 4 Deliverables
+
+
-## Part 1 Lab Preparation
+Lab 4 Deliverable (Click to Expand) +
### Part 1 (Week 1) **Submit the following for Part 1:** @@ -52,15 +52,22 @@ - Written summary: what it looks like, works like, acts like - Reflection on what you learned and next steps ---- +The deliverables for this lab are, writings, sketches, photos, and videos that show what your prototype: -## Lab Overview -**NAMES OF COLLABORATORS HERE** +"Looks like": shows how the device should look, feel, sit, weigh, etc. +"Works like": shows what the device can do. +"Acts like": shows how a person would interact with the device. +For submission, the readme.md page for this lab should be edited to include the work you have done: +Upload any materials that explain what you did, into your lab 4 repository, and link them in your lab 4 readme.md. +Link your Lab 4 readme.md in your main Interactive-Lab-Hub readme.md. +Labs are due on Mondays, make sure to submit your Lab 4 readme.md to Canvas. -For lab this week, we focus both on sensing, to bring in new modes of input into your devices, as well as prototyping the physical look and feel of the device. You will think about the physical form the device needs to perform the sensing as well as present the display or feedback about what was sensed. +
+
-The deliverables for this lab are, writings, sketches, photos, and videos that show what your prototype:
-* "Looks like": shows how the device should look, feel, sit, weigh, etc.
-* "Works like": shows what the device can do.
-* "Acts like": shows how a person would interact with the device.
+---
-For submission, the readme.md page for this lab should be edited to include the work you have done:
-* Upload any materials that explain what you did, into your lab 4 repository, and link them in your lab 4 readme.md.
-* Link your Lab 4 readme.md in your main Interactive-Lab-Hub readme.md.
-* Labs are due on Mondays, make sure to submit your Lab 4 readme.md to Canvas.
+## Lab Overview
+Collaborators: Jessica Hsiao (dh779), Irene Wu (yw2785)
+For lab this week, we focus both on sensing, to bring in new modes of input into your devices, as well as prototyping the physical look and feel of the device. You will think about the physical form the device needs to perform the sensing as well as present the display or feedback about what was sensed.
-## Lab Overview
A) [Capacitive Sensing](#part-a)
@@ -250,22 +252,121 @@ You can go to the [SparkFun GitHub Page](https://github.com/sparkfun/Qwiic_Proxi
### Physical considerations for sensing
-Usually, sensors need to be positioned in specific locations or orientations to make them useful for their application. Now that you've tried a bunch of the sensors, pick one that you would like to use, and an application where you use the output of that sensor for an interaction. For example, you can use a distance sensor to measure someone's height if you position it overhead and get them to stand under it.
+Usually, sensors need to be positioned in specific locations or orientations to make them useful for their application. Now that you've tried a bunch of the sensors, pick one that you would like to use, and an application where you use the output of that sensor for an interaction. For example, you can use a distance sensor to measure someone's height if you position it overhead and get them to stand under it.
+
+We chose **Light/Proximity/Gesture sensor (APDS-9960)** as our sensor.
+
+**Application #1: Plant Health Monitor**
+
+- Sensor information: light
+- The device is capable of detecting plant color variations by RGB. When the color shifts from bright and vibrant tones to brown, the system alerts users that the plants are under stress or nearing death. Users can then respond by providing additional water or nutrients to restore plant health.
+
+**Application #2: Natural Lighting Room**
+
+- Sensor information: light
+- Mounted near a window or ceiling fixture, the sensor continuously measures ambient light intensity. When sufficient natural light is detected, the system automatically dims or turns off artificial lighting to save energy.
+
+**Application #3: Exhibition Visitor Counter**
+
+- Sensor information: gesture
+- Placed on a side or above an exhibit entryway, the APDS-9960 detects directional hand or body movements. When a visitor passes by, the gesture data (e.g., left or right swipe) increments a counter, helping curators estimate visitor flow and engagement.
+
+**Application #4: Eye Wellness Assistant**
+
+- Sensor information: proximity
+- Integrated near a computer monitor or laptop bezel, the proximity sensor tracks how close a user’s face is to the screen. If the user sits too close for prolonged periods, the system gently alerts them to maintain a healthier viewing distance.
+
+**Application #5: Movie Controller**
+
+- Sensor information: gesture
+- Installed near a TV or projector, the sensor enables touch-free gesture controls. Users can swipe left to rewind, swipe right to skip, or perform an upward gesture to pause/play the movie.
+
**\*\*\*Draw 5 sketches of different ways you might use your sensor, and how the larger device needs to be shaped in order to make the sensor useful.\*\*\***
+Application #1: Plant Health Monitor
+
+Lab 4 Part 1. Preparation
+ ## Part 1 Lab Preparation ### Get the latest content: As always, pull updates from the class Interactive-Lab-Hub to both your Pi and your own GitHub repo. As we discussed in the class, there are 2 ways you can do so: @@ -100,20 +107,15 @@ Option 3: (preferred) use the Github.com interface to update the changes. (We do offer shared cutting board, cutting tools, and markers on the class cart during the lab, so do not worry if you don't have them!) -## Deliverables \& Submission for Lab 4 +
+
+Application #2: Natural Lighting Room
+
+
+
+Application #3: Exhibition Visitor Counter
+
+
+
+Application #4: Eye Wellness Assistant
+
+
+
+Application #5: Movie Controller
+
+
+
+
**\*\*\*What are some things these sketches raise as questions? What do you need to physically prototype to understand how to anwer those questions?\*\*\***
+**Application #1: Plant Health Monitor**
+- Questions raised:
+ 1. Connection and communication: The sketch doesn’t specify how the sensor and the display communicate — whether they are connected by wire, use wireless transmission, or operate through an intermediary microcontroller such as a Raspberry Pi.
+ 2. Sensor placement: It’s unclear how the sensor should be positioned or angled relative to the plant’s leaves to ensure consistent and accurate color readings.
+ 3. Environmental lighting effects: Different ambient lighting conditions (sunny vs. cloudy, indoor vs. outdoor) could alter the perceived color of the plant, even when its actual health remains the same.
+
+- To address these questions, we can first build a prototype that tests various sensor positions and angles to determine which setup yields the most stable readings. Next, running lighting variation tests — capturing RGB data under multiple environmental conditions — would help calibrate thresholds or develop compensation algorithms. Finally, prototyping the communication pathway between the sensor and display (wired vs. wireless) will clarify how to make the system responsive and portable for real-world use.
+
+**Application #2: Natural Lighting Room**
+
+- Questions raised:
+ 1. Light differentiation: The sketch doesn’t explain how the sensor distinguishes between natural and artificial light — will it rely solely on overall brightness, or also consider color temperature and spectral balance?
+ 2. Sensitivity and stability: It’s unclear how sensitive the system should be before adjusting lighting intensity. Minor fluctuations in sunlight (e.g., when clouds pass) could cause unstable dimming or flickering.
+ 3. Sensor orientation: The sensor’s placement and angle might significantly affect readings — pointing it directly at a window versus facing the interior ceiling could produce very different results.
+ 4. Communication and control: The sketch doesn’t specify how the sensor interacts with the lighting system — through direct wiring, a microcontroller, or a wireless connection.
+
+- To address these questions, we can start by testing the sensor at different positions and orientations within a room to measure how light readings vary across the day. Next, a calibration experiment can help determine appropriate brightness thresholds and smoothing algorithms to avoid frequent or erratic lighting changes. Finally, building a prototype connection between the sensor and a light controller (e.g., Raspberry Pi + LED dimmer) will allow us to evaluate real-time response and communication reliability.
+
+**Application #3: Exhibition Visitor Counter**
+
+- Questions raised:
+ 1. Detection accuracy: The sketch doesn’t specify how the sensor distinguishes between individual visitors — for example, how it avoids counting the same person twice or missing people walking closely together.
+ 2. Range and directionality: It’s unclear how far the APDS-9960 can reliably detect gestures or movement and whether it can sense entry versus exit directions.
+ 3. Environmental interference: Exhibition lighting, reflections, or nearby displays might interfere with gesture detection or falsely trigger the counter.
+ 4. Physical placement: The optimal height and orientation of the sensor relative to the doorway are uncertain — it may need to be tested at various positions to ensure consistent detection.
+
+- By testing different sensor heights and distances near an actual doorway, we can measure detection accuracy for varying visitor speeds and group sizes. Additionally, running environmental tests under different lighting conditions will help determine if shielding or calibration is needed to reduce false triggers. Finally, prototyping a count display system will help evaluate real-time feedback and timing consistency during high-traffic scenarios.
+
+**Application #4: Eye Wellness Assistant**
+
+- Questions raised:
+ 1. Unclear sense of distance: The sketch doesn’t convey how far the user is from the screen, making it difficult to judge whether the sensor can accurately measure a comfortable viewing range.
+ 2. Screen tilt impact: The monitor’s tilt angle could affect proximity readings, since the sensor’s detection depends heavily on its facing direction.
+ 3. Sensor form and placement: The drawing doesn’t indicate the sensor’s actual size or visibility, leaving uncertainty about whether it would distract the user or blend seamlessly into the screen design.
+
+- To answer the first question, we need to design a way to display the distance information to the user, for instance, putting a tiny monitor next to the screen to show the number. For the second and the third questions, they can be approached by conducting user studies with the physical prototype to experiment in a real-world settings. We can run user trials with different sensor placements and screen angles to evaluate accuracy, comfort, and intrusiveness in daily use.
+
+**Application #5: Movie Controller**
+
+- Questions raised:
+ 1. Gesture recognition accuracy: The sketch doesn’t clarify how reliably the sensor can differentiate between gestures such as left, right, or up swipes, especially when users vary in hand speed or distance.
+ 2. Detection range and angle: It’s unclear how far the user can sit from the screen while still being detected, or how wide the sensor’s field of view needs to be to capture gestures effectively.
+ 3. Interference and usability: Ambient lighting, reflections, or nearby movement (like someone walking past) could accidentally trigger commands. The sketch doesn’t show how the system might prevent or handle such false positives.
+
+- To address these questions, we can build a functional prototype that connects the gesture sensor to a simple media controller. This would allow testing of gesture detection accuracy across multiple users, distances, and lighting conditions. We also need to experiment with sensor placement and viewing angles — for example, mounting it on top of or below the TV — to find the most reliable setup. Lastly, collecting real interaction data can help define gesture thresholds and filtering strategies to minimize unintended activations.
+
+
**\*\*\*Pick one of these designs to prototype.\*\*\***
+We picked the forth one to prototype: Eye Wellness Assistant
+
### Part D
### Physical considerations for displaying information and housing parts
-
-
-Here is a Pi with a paper faceplate on it to turn it into a display interface:
+
+ 
@@ -298,201 +399,332 @@ Fold the first flap of the strip so that it sits flush against the back of the f
Here is an example:
+
+
+
Think about how you want to present the information about what your sensor is sensing! Design a paper display for your project that communicates the state of the Pi and a sensor. Ideally you should design it so that you can slide the Pi out to work on the circuit or programming, and then slide it back in and reattach a few wires to be back in operation.
**\*\*\*Sketch 5 designs for how you would physically position your display and any buttons or knobs needed to interact with it.\*\*\***
+**Design #1**: The physical prototype includes an adjustable sensor holder extending from the computer, with a small board placed next to it for sensor integration. When the user is too close to the monitor, the board will display a red light and flash continuously to alert them. Otherwise, the board will remain black without providing any additional notifications.
+
+Examples (Click to expand) +
+ Here is a Pi with a paper faceplate on it to turn it into a display interface:
@@ -298,201 +399,332 @@ Fold the first flap of the strip so that it sits flush against the back of the f
Here is an example:
+
+
+**Design #2**: The physical prototype includes an adjustable curved phone stand extending from the computer, with a small light placed beside it to indicate the proximity detection status. It uses a color light (green = good, yellow = a bit close, red = too close) to present the status.
+
+
+
+**Design #3**: The physical prototype includes an adjustable curved phone stand extending from the computer, with a small board placed next to it for sensor integration. The actual distance between the screen and the user’s face will be displayed on the board.
+
+
+
+**Design #4**: Place the sensor on top of the computer with the sensor facing downward. When the user gets too close to the computer (meaning their head is below the sensor), the sensor detects the proximity and uses a beeping sound to alert the users.
+
+
+
+**Design #5**: Place the sensor on top of the computer with the sensor facing downward. When the user gets too close to the computer (meaning their head is below the sensor), a voice message will be played to remind the user of the distance, such as “you are too close to your screen, please keep away from it”
+
+
+
+
**\*\*\*What are some things these sketches raise as questions? What do you need to physically prototype to understand how to anwer those questions?\*\*\***
+#### Design #1
+
+- Questions raised:
+ 1. How sensitive should the proximity threshold be to accurately detect when the user is “too close” without triggering false alerts?
+ 2. Will the flashing red light be noticeable yet comfortable for the user, or could it become distracting during long use?
+ 3. How much does the adjustable sensor angle affect detection accuracy across different screen tilt positions and user heights?
+- What to prototype:
+ 1. Test various distance thresholds to determine the optimal trigger range for comfort and accuracy.
+ 2. Experiment with different light intensities, colors, and flashing rates to find a balance between visibility and user comfort.
+ 3. Build and evaluate the adjustable holder to measure how sensor angle and placement influence proximity detection reliability.
+
+
+#### Design #2
+
+- Questions raised:
+ 1. What are the optimal distance thresholds for each color indicator (green, yellow, red) to provide meaningful and intuitive feedback to users?
+ 2. Will users easily notice and interpret the color changes, or would additional feedback (e.g., brightness or flashing) improve clarity?
+ 3. How does the curved stand’s angle and height affect the accuracy of proximity detection for users of different sitting positions?
+- What to prototype:
+ 1. Calibrate and test various distance ranges to define clear and consistent thresholds for each light color.
+ 2. Conduct short user tests to evaluate whether color feedback alone is sufficient for awareness and comfort.
+ 3. Build the curved stand prototype and measure detection performance at multiple sensor angles and user heights to ensure reliability.
+
+
+#### Design #3
+
+- Questions raised:
+ 1. How accurate and responsive will the displayed distance values be when the user moves slightly or changes posture?
+ 2. What is the most effective way to display the distance information so that it’s noticeable without being distracting?
+ 3. How does the sensor’s placement or tilt angle influence measurement consistency across different screen setups and user positions?
+- What to prototype:
+ 1. Test the sensor’s real-time distance measurement accuracy and response speed under typical usage movements.
+ 2. Experiment with different display formats (numerical, graphical, or color-coded) to assess readability and user comfort.
+ 3. Build the adjustable stand prototype and measure how various angles and distances affect data stability and reliability.
+
+#### Design #4:
+
+- Questions raised:
+ 1. How accurately can the downward-facing sensor detect when the user’s head crosses the threshold distance without being affected by lighting or hair color?
+ 2. What should the distance threshold be to ensure the alert triggers at a comfortable and safe viewing distance?
+ 3. Will the beeping alert remain effective over time, or might it become annoying or easy to ignore during long computer use?
+- What to prototype:
+ 1. Test detection reliability across users with different heights, hairstyles, and seating positions.
+ 2. Experiment with several threshold distances to determine an optimal range that balances comfort and alert sensitivity.
+ 3. Prototype different audio feedback patterns (e.g., single beep vs. continuous tone) to evaluate which is most noticeable yet least disruptive.
+
+
+#### Design #5:
+- Questions raised:
+ 1. How should the timing and frequency of the voice alert be set so it effectively reminds users without feeling intrusive?
+ 2. Will the downward-facing sensor maintain consistent accuracy across users with different heights and seating distances?
+ 3. Do users perceive the voice message as helpful feedback or as a distraction during focused work?
+- What to prototype:
+ 1. Experiment with various voice alert intervals, tones, and volumes to find a balance between clarity and comfort.
+ 2. Test the sensor’s performance across diverse user positions and lighting environments to ensure reliable detection.
+ 3. Conduct short user evaluations comparing voice feedback with other alert methods (e.g., light or sound) to determine the most preferred design.
+
+
+
**\*\*\*Pick one of these display designs to integrate into your prototype.\*\*\***
+We picked the forth one to integrate into our prototype.
+
**\*\*\*Explain the rationale for the design.\*\*\*** (e.g. Does it need to be a certain size or form or need to be able to be seen from a certain distance?)
-Build a cardboard prototype of your design.
+#### Why the device is elevated
+- The sensor needs to face downward to accurately detect the user’s head position. However, because the sensor strip is relatively short, we had to raise the Raspberry Pi to achieve the proper downward angle. Elevating the device also ensures that the user’s head does not accidentally block or strike the sensor during normal use.
+- During testing, we found that the sensor can reliably detect objects within a range of about 15 cm. Beyond this distance, detection becomes unreliable. Therefore, raising the device ensures that the user’s head remains within this effective detection range for consistent performance.
+
+#### Why the device has several holes
+- The holes serve multiple functional purposes. One is for the Bluetooth module, which broadcasts audio alerts to notify the user when they are sitting too close to the screen. Another set of holes supports display visibility and ventilation. We integrated a small screen on the Raspberry Pi to show system details and user feedback. To make this display clear and user-friendly, we positioned it near the radio broadcaster and ensured there are openings for both visibility and heat dissipation.
+#### Why handles were added to the device
+- We added two handles on the right side of the device to make maintenance easier. These handles allow users or developers to conveniently open the casing for adjustments, sensor calibration, or component replacement without damaging the housing.
+
+
+Build a cardboard prototype of your design.
**\*\*\*Document your rough prototype.\*\*\***
+
# LAB PART 2
-### Part 2
-
Following exploration and reflection from Part 1, complete the "looks like," "works like" and "acts like" prototypes for your design, reiterated below.
+---
-### Part E
+### Part C: Chaining Devices and Exploring Interaction Effects
-#### Chaining Devices and Exploring Interaction Effects
+For Part 2, We designed and built a fun interactive prototype using multiple inputs and outputs. We use two inputs and two outputs to give users hints about the distance between user's head and their computers.
-For Part 2, you will design and build a fun interactive prototype using multiple inputs and outputs. This means chaining Qwiic and STEMMA QT devices (e.g., buttons, encoders, sensors, servos, displays) and/or combining with traditional breadboard prototyping (e.g., LEDs, buzzers, etc.).
+#### Detail Function:
-**Your prototype should:**
-- Combine at least two different types of input and output devices, inspired by your physical considerations from Part 1.
-- Be playful, creative, and demonstrate multi-input/multi-output interaction.
+- Proximity Detection
+ - The sensor continuously measures the distance between the user’s head and the computer area.
+ - When the user gets too close, the system switches from the “normal” state to the “warning” state.
-**Document your system with:**
-- Code for your multi-device demo
-- Photos and/or video of the working prototype in action
-- A simple interaction diagram or sketch showing how inputs and outputs are connected and interact
-- Written reflection: What did you learn about multi-input/multi-output interaction? What was fun, surprising, or challenging?
-**Questions to consider:**
-- What new types of interaction become possible when you combine two or more sensors or actuators?
-- How does the physical arrangement of devices (e.g., where the encoder or sensor is placed) change the user experience?
-- What happens if you use one device to control or modulate another (e.g., encoder sets a threshold, sensor triggers an action)?
-- How does the system feel if you swap which device is "primary" and which is "secondary"?
+- Visual Feedback (Screen)
+ - In the normal state, the screen displays a motivational or positive message to encourage healthy posture.
+ - In the warning state, the screen turns red and shows a clear alert message (“You are too close to the screen”) to draw attention.
-Try chaining different combinations and document what you discover!
-See encoder_accel_servo_dashboard.py in the Lab 4 folder for an example of chaining together three devices.
+- Audio Feedback (Speaker)
+ - When the warning is triggered, the speaker plays a sound alert.
+ - This adds an additional layer of feedback that is harder to ignore than visuals alone.
-**`Lab 4/encoder_accel_servo_dashboard.py`**
+- User Control (Buttons)
+ - One button allows the user to start or activate the monitoring system.
+ - The second button lets the user silence the audio alert or stop the notification, giving the user agency.
-#### Using Multiple Qwiic Buttons: Changing I2C Address (Physically & Digitally)
+- Interaction Flow
+ - The sensor detects → screen and speaker react → user acknowledges with a button press → system resets.
+ - This creates a loop of detection, feedback, and user response.
-If you want to use more than one Qwiic Button in your project, you must give each button a unique I2C address. There are two ways to do this:
-##### 1. Physically: Soldering Address Jumpers
+##### Input:
-On the back of the Qwiic Button, you'll find four solder jumpers labeled A0, A1, A2, and A3. By bridging these with solder, you change the I2C address. Only one button on the chain can use the default address (0x6F).
+1. Proximity Sensor: The sensor is placed next to the computer (typically around 40 cm away). When it detects an object below it—usually the user’s head—it triggers the application to remind the user to maintain a proper distance from the screen.
-**Address Table:**
+2. Two Led Button: Users can press one button to start the application, while the second button is used to stop notifications. For example, if the application plays a sound to warn the user that they are too close to the screen, pressing the second button will silence the alert.
-| A3 | A2 | A1 | A0 | Address (hex) |
-|----|----|----|----|---------------|
-| 0 | 0 | 0 | 0 | 0x6F |
-| 0 | 0 | 0 | 1 | 0x6E |
-| 0 | 0 | 1 | 0 | 0x6D |
-| 0 | 0 | 1 | 1 | 0x6C |
-| 0 | 1 | 0 | 0 | 0x6B |
-| 0 | 1 | 0 | 1 | 0x6A |
-| 0 | 1 | 1 | 0 | 0x69 |
-| 0 | 1 | 1 | 1 | 0x68 |
-| 1 | 0 | 0 | 0 | 0x67 |
-| ...| ...| ...| ... | ... |
+##### Output:
+1. Screen: When the application is in a normal state (no warning is needed), the screen displays a motivational message to encourage the user to maintain good posture and continue working. When a warning is triggered, the screen switches to a red background and displays the message “You are too close to the screen” to alert the user.
+2. Speaker: When a warning is triggered, which indicates that the user is too close to the computer, the speaker plays an alert sound.
-For example, if you solder A0 closed (leave A1, A2, A3 open), the address becomes 0x6E.
-**Soldering Tips:**
-- Use a small amount of solder to bridge the pads for the jumper you want to close.
-- Only one jumper needs to be closed for each address change (see table above).
-- Power cycle the button after changing the jumper.
+**Hardware Block Diagram**
-##### 2. Digitally: Using Software to Change Address
+```scss
+ ┌─────────────────────┐
+ │ Proximity Sensor │
+ └─────────┬───────────┘
+ │ (distance data)
+ ▼
+ ┌─────────────────────┐
+ │ Raspberry Pi │
+ └───────┬───────┬─────┘
+ │ │ │
+ (visual) (audio) (user input)
+ ▼ ▼ ▼
+ ┌──────┐ ┌───────┐ ┌───────┐
+ │Screen│ │Speaker│ │Buttons│
+ └──────┘ └───────┘ └───────┘
-You can also change the address in software (temporarily or permanently) using the example script `qwiic_button_ex6_changeI2CAddress.py` in the Lab 4 folder. This is useful if you want to reassign addresses without soldering.
-Run the script and follow the prompts:
-```bash
-python qwiic_button_ex6_changeI2CAddress.py
```
-Enter the new address (e.g., 5B for 0x5B) when prompted. Power cycle the button after changing the address.
-**Note:** The software method is less foolproof and you need to make sure to keep track of which button has which address!
+**Interaction Flow Diagram**
+
+```pgsql
+ ┌────────────────────────┐
+ │ Sensor checks distance │
+ └─────────────┬──────────┘
+ │
+ ┌────────────────┴─────────────────┐
+ │ │
+ ▼ ▼
+ (User at safe distance) (User too close)
+ │ │
+ ▼ ▼
+Screen shows motivation Screen turns red + alert text
+ │ │
+ ▼ ▼
+ No sound Speaker plays warning
+ │
+ ▼
+ User presses button to silence alert
+ │
+ ▼
+ System returns to normal state
+```
-##### Using Multiple Buttons in Code
+**System Flowchart**
+
+```pgsql
+ ┌──────────────────────┐
+ │ System Started │
+ │ (User presses start) │
+ └───────────┬──────────┘
+ │
+ ▼
+ ┌──────────────────────┐
+ │ Sensor reads distance│
+ └───────────┬──────────┘
+ │
+ ┌─────────────┴─────────────┐
+ │ │
+ ▼ ▼
+ ┌───────────────────┐ ┌─────────────────────┐
+ │ User far enough │ NO │ User too close │ YES
+ │ (safe distance) │────── │ (below threshold) │
+ └─────────┬─────────┘ └─────────┬───────────┘
+ │ │
+ ▼ ▼
+ ┌────────────────────┐ ┌────────────────────────┐
+ │ Show motivational │ │ Turn screen red │
+ │ text on screen │ │ Display warning text │
+ └─────────┬──────────┘ └─────────┬──────────────┘
+ │ │
+ ▼ ▼
+ ┌────────────────────┐ ┌────────────────────────┐
+ │ No sound played │ │ Speaker plays sound │
+ └─────────┬──────────┘ └─────────┬──────────────┘
+ │ │
+ │ ┌────────────┴────────────┐
+ │ │ User presses stop btn │
+ │ └────────────┬────────────┘
+ │ ▼
+ │ ┌──────────────────────┐
+ └───────────────▶│ Sound is silenced │
+ │ System goes back to │
+ │ normal state │
+ └───────────┬──────────┘
+ │
+ ▼
+ (Loop continues)
-After setting unique addresses, you can use multiple buttons in your script. See these example scripts in the Lab 4 folder:
+```
-- **`qwiic_1_button.py`**: Basic example for reading a single Qwiic Button (default address 0x6F). Run with:
- ```bash
- python qwiic_1_button.py
- ```
+#### System Architecture
-- **`qwiic_button_led_demo.py`**: Demonstrates using two Qwiic Buttons at different addresses (e.g., 0x6F and 0x6E) and controlling their LEDs. Button 1 toggles its own LED; Button 2 toggles both LEDs. Run with:
- ```bash
- python qwiic_button_led_demo.py
- ```
+| Layer | Components | Role |
+| ----------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| **Sensors (Input)** | APDS9960 proximity sensor, Qwiic Button #1 & #2, GPIO buttons | Detect user proximity and button interaction |
+| **Compute & Logic** | Raspberry Pi running Python main loop | Coordinates all logic and state machine (NORMAL / WARNING mode) |
+| **Communication Buses** | I2C (APDS9960 + QWIIC buttons), SPI (TFT display), GPIO (backlight + onboard buttons) | Hardware communication backbone |
+| **Output** | ST7789 TFT screen, backlight, text-to-speech (espeak) | Displays user feedback and audible warning |
-Here is a minimal code example for two buttons:
-```python
-import qwiic_button
-# Default button (0x6F)
-button1 = qwiic_button.QwiicButton()
-# Button with A0 soldered (0x6E)
-button2 = qwiic_button.QwiicButton(0x6E)
-button1.begin()
-button2.begin()
+#### Demo (Photos and/or video of the working prototype in action)
-while True:
- if button1.is_button_pressed():
- print("Button 1 pressed!")
- if button2.is_button_pressed():
- print("Button 2 pressed!")
-```
+Coding: look at [demo.py](https://github.com/JessicaDJ0807/Interactive-Lab-Hub/blob/Fall2025/Lab%204/demo.py)
+
+* "Looks like": shows how the device should look, feel, sit, weigh, etc.
+
+
+
+* "Works like": shows what the device can do
+
+
+
+* "Acts like": shows how a person would interact with the device
-For more details, see the [Qwiic Button Hookup Guide](https://learn.sparkfun.com/tutorials/qwiic-button-hookup-guide/all#i2c-address).
+https://github.com/user-attachments/assets/16318d59-851d-438b-872a-084e3ad348d3
+
+
+#### Users Feedback
+* At first the sound surprised the user, but it was helpful because they sometimes don’t notice visual warnings when they're focused on the screen.
+* They like that they can press a button to stop the sound. It feels like they still have control, instead of being stuck with an alarm.
+* When the sensor was placed in front of them, it triggered too often and felt a bit annoying. But once you moved it to the side, it felt much smoother and more natural.
---
-### PCF8574 GPIO Expander: Add More Pins Over I²C
+### Part D. Written Reflection
-Sometimes your Pi’s header GPIO pins are already full (e.g., with a display or HAT). That’s where an I²C GPIO expander comes in handy.
+- Learning about multi-input/multi-output interaction
-We use the Adafruit PCF8574 I²C GPIO Expander, which gives you 8 extra digital pins over I²C. It’s a great way to prototype with LEDs, buttons, or other components on the breadboard without worrying about pin conflicts—similar to how Arduino users often expand their pinouts when prototyping physical interactions.
+ - We learned that combining multiple sensors and actuators enables much richer and more meaningful interaction than using a single component. A proximity sensor alone can only detect that the user is too close, but it cannot communicate anything back. Once we added the screen, speaker, and buttons, the system could respond in multiple ways, including displaying a visual warning, playing a sound, or showing encouragement when the user maintained a healthy distance. This made the feedback intuitive and noticeable, transforming the system from a hidden measurement tool into an interactive assistant.
-**Why is this useful?**
-- You only need two wires (I²C: SDA + SCL) to unlock 8 extra GPIOs.
-- It integrates smoothly with CircuitPython and Blinka.
-- It allows a clean prototyping workflow when the Pi’s 40-pin header is already occupied by displays, HATs, or sensors.
-- Makes breadboard setups feel more like an Arduino-style prototyping environment where it’s easy to wire up interaction elements.
+- New types of interaction from combining components
-**Demo Script:** `Lab 4/gpio_expander.py`
+ - A proximity sensor alone only measures distance, but when paired with a screen and speaker, it becomes a behavioral feedback tool.
-
- 
-
- 
-
+
## Overview
Building upon the paper-airplane metaphor (we're understanding the material of machine learning for design), here are the four sections of the lab activity:
@@ -37,9 +40,11 @@ C) [Flight test](#part-c)
D) [Reflect](#part-d)
---
+## Part A: Play with different sense-making algorithms.
-### Part A
-### Play with different sense-making algorithms.
+Prep (Click to Expand)
1. Install VNC on your laptop if you have not yet done so. This lab will actually require you to run script on your Pi through VNC so that you can see the video stream. Please refer to the [prep for Lab 2](https://github.com/FAR-Lab/Interactive-Lab-Hub/blob/-/Lab%202/prep.md#using-vnc-to-see-your-pi-desktop). 2. Install the dependencies as described in the [prep document](prep.md). @@ -24,6 +26,7 @@ This lab will help you think through the design of observant systems, particular 1. Show pictures, videos of the "sense-making" algorithms you tried. 1. Show a video of how you embed one of these algorithms into your observant system. 1. Test, characterize your interactive device. Show faults in the detection and how the system handled it. +
+
+
+
+
+## Part B: Construct a simple interaction.
+
+Part A content (Click to expand)
#### Pytorch for object recognition @@ -102,8 +107,6 @@ Consider how you might use this position based approach to create an interaction (You might also consider how this notion of percentage control with hand tracking might be used in some of the physical UI you may have experimented with in the last lab, for instance in controlling a servo or rotary encoder.) - - #### Moondream Vision-Language Model [Moondream](https://www.ollama.com/library/moondream) is a lightweight vision-language model that can understand and answer questions about images. Unlike the classification models above, Moondream can describe images in natural language and answer specific questions about what it sees. @@ -125,7 +128,7 @@ This will capture an image from your webcam and let you ask questions about it i #### Teachable Machines Google's [TeachableMachines](https://teachablemachine.withgoogle.com/train) is very useful for prototyping with the capabilities of machine learning. We are using [a python package](https://github.com/MeqdadDev/teachable-machine-lite) with tensorflow lite to simplify the deployment process. - + To get started, install dependencies into a virtual environment for this exercise as described in [prep.md](prep.md): @@ -152,34 +155,158 @@ Teachable machines provides an audio classifier too. If you want to use audio cl In an earlier version of this class students experimented with foundational computer vision techniques such as face and flow detection. Techniques like these can be sufficient, more performant, and allow non discrete classification. Find the material here: [CV_optional/cv.md](CV_optional/cv.md). -### Part B -### Construct a simple interaction. +
+
+
+
+### Simple Interaction Prototype: Smile Together
+
+For this prototype, we selected the BlazeFace (short-range) face detection model and extended it to support multi-person smiling detection. The goal was to explore how computer vision can shape playful social interaction in a shared space.
+
+#### Interaction Concept
+
+The system acts as a “smile check” camera, designed for situations like taking group photos. When multiple people are in the frame:
+
+- If everyone is smiling, the camera remains calm (ready to take the picture).
+
+- If any person is not smiling, the system triggers feedback—such as playing a sound or flashing an LED—to gently nudge the group to smile before the photo is taken.
+
+This creates a playful, cooperative interaction: instead of one person asking “okay, is everyone smiling?”, the device automatically enforces the moment.
+
+#### Input
+
+- Live webcam feed
+- BlazeFace detects multiple faces simultaneously
+- A simple heuristic (prototype stage) estimates a "smile score" for each face (in future versions this can be replaced by a trained smile classifier)
+
+#### Output / Feedback
+The system experiments with different feedback modes when not all detected faces are smiling:
+
+- **Audio signal**: Soft “ding” tone as a reminder
+- **LED blink**: Simulated LED flashes on screen (or physical LED if connected)
+- **Screen overlay**: Red border + text: “Someone isn’t smiling yet!”
+- **Visual smile sync bar**: Progress bar showing overall “smile status” before approval
+
+#### Interaction Logic
+
+- Track each detected face independently
+- Assign a filtered “smile probability” to each face
+- Average group smile score
+- Trigger feedback when at least one person falls below the smile threshold
-**\*\*\*Describe and detail the interaction, as well as your experimentation here.\*\*\***
+#### Prototype rule:
-### Part C
-### Test the interaction prototype
+If any face has smile score < 0.65 for more than 1 second → trigger feedback
+
+### Part C: Test the interaction prototype
+
+Instruction (Click to Expand)
* Pick one of the models you have tried, and experiment with prototyping an interaction. * This can be as simple as the boat detector shown in lecture. * Try out different interaction outputs and inputs. +
+
+
+#### When does it do what it’s supposed to do?
+
+- When users are facing the camera and they are within detection range, the system will detect their faces and figure out whether all users are smiling or not.
+
+- When all users are smiling, it will trigger the systems to display fireworks and play music.
+
+- When some people are smiling and others aren't, the screen will display a message about “cheer your friend up.” Also, it would play a piece of music to cheer people up.
+
+#### When does it fail?
+
+- The systems occasionally fail when:
+
+ - The light in the place is uneven or dim, leading some people’s faces to be partially shadowed. It may lead to wrong detections.
+ - Some users turning their head or partially exiting the frame may cause the system to fail because it can not fully detect people’s faces.
+ - Some users may have subtle smiles (e.g., small smirks), which are misclassified as neutral.
+ - Glasses or facial hair obscure key landmarks, causing errors in emotion detection.
+
+#### Why does it fail?
+
+- BlazeFace and MediaPipe’s face mesh models are optimized for frontal, well-lit faces; off-axis angles reduce confidence.
+The smile detection logic likely relies on mouth corner distance or AU12 activation thresholds, which can be sensitive to lighting and camera resolution.
+
+- Frame rate fluctuations can cause inconsistent detections, especially if webcam performance drops.
+
+- Large groups or background faces triggering false detections.
+
+- People wearing masks or using virtual backgrounds.
+
+- Different emotional expressions (e.g., laughter or talking) being mistaken for smiles.
+
+- Network latency or computation delay causing asynchronous reactions (music/fireworks triggering late).
+
+***Think about someone using the system. Describe how you think this will work.***
+
+1. Are they aware of the uncertainties in the system?
+2. How bad would they be impacted by a miss classification?
+3. How could change your interactive system to address this?
+4. Are there optimizations you can try to do on your sense-making algorithms?
-### Part D
-### Characterize your own Observant system
+**User Experience: How It Might Work**
+
+- When a group gathers for a photo, the Smile-Check Camera becomes a “smart reminder” rather than a passive device. The participants position themselves, look at the screen, and notice live visual feedback. If everyone smiles, the system stays quiet, which is implicitly approving the moment.
+
+- If someone’s expression is neutral or missing, the audio “ding” or red border gently nudges them to adjust. This playful loop continues until all detected smiles reach the threshold, prompting laughter and coordination. The system subtly shifts social responsibility from a human photographer to an automated, impartial “smile referee,” making the act of getting ready part of the fun.
+
+**Awareness of System Uncertainties**
+
+- Users are partially aware of the system’s uncertainties, though not technically. They may notice that the camera sometimes fails to recognize subtle smiles or misreads a talking face as neutral. In these moments, people tend to attribute quirks to “the system being picky” rather than technical limitations.
+
+- This ambiguity actually contributes to playfulness. It invites the group to exaggerate their smiles or “test” the system. However, frequent false detections could frustrate users if they are trying to capture a real photo quickly.
+
+
+#### Impact of Misclassification
+
+Misclassifications have low practical impact but moderate experiential impact:
+
+- **Low stakes**: The worst outcome is a short delay or a false “not smiling yet” alert.
+
+- **Playful context**: Small errors often make people laugh or over-perform expressions, reinforcing the cooperative aspect.
+
+- **Potential downside**: If errors persist (e.g., one person never registers as smiling), it can create mild annoyance or exclude that participant from the “success” moment.
+
+#### Design Improvements
+
+To make the system more robust and user-friendly:
+
+- **Calibration Step**: Let users briefly record their neutral and smiling faces so the threshold (0.65) adjusts to individual facial features.
+
+- **Confidence Averaging**: Smooth the smile probability over several frames (e.g., exponential moving average) to reduce flickering classifications.
+
+- **Visual Feedback Clarity**: Instead of a red “error” state, show encouraging messages like “Almost there!” or a filling progress bar to sustain engagement.
+
+- **Lighting Adaptation**: Integrate brightness normalization or display a “too dark” hint if the camera confidence drops.
+
+### Part D: Characterize your own Observant system
+
+Instruction (Click to Expand)
Now flight test your interactive prototype and **note down your observations**: For example: 1. When does it what it is supposed to do? -1. When does it fail? -1. When it fails, why does it fail? -1. Based on the behavior you have seen, what other scenarios could cause problems? +2. When does it fail? +3. When it fails, why does it fail? +4. Based on the behavior you have seen, what other scenarios could cause problems? **\*\*\*Think about someone using the system. Describe how you think this will work.\*\*\*** 1. Are they aware of the uncertainties in the system? -1. How bad would they be impacted by a miss classification? -1. How could change your interactive system to address this? -1. Are there optimizations you can try to do on your sense-making algorithm. +2. How bad would they be impacted by a miss classification? +3. How could change your interactive system to address this? +4. Are there optimizations you can try to do on your sense-making algorithm. + +
+
+
+**What can you use Smile Together for?**
+
+- **Group picture**: One possible circumstance to use this application is when taking a group picture. It is difficult to check if everyone is smiling or looking at the camera at the same time, and with Smile Together, it automatically monitors all faces in the frame and notifies the group if someone is not smiling yet. This makes taking group photos smoother, faster, and more fun.
+
+- **Shared social moments**: Another circumstance is during shared social moments, such as events or celebrations, where people want to maintain a positive group expression. The system can be used as a playful prompt to encourage group engagement, cooperation, and synchronized expressions.
+
+- **Team-building**: It can also function as a team-building activity. For example, a workplace or social club could use the system as a “smile synchronizing challenge,” where participants must coordinate their expressions to unlock an animation or audio reward.
+
+- **Photo booth**: It could be used in self-service photo booths at events or public spaces. Instead of needing a photographer to announce “say cheese,” the system guides participants automatically and facilitates a joyful interactive experience.
+
+#### What is a good/bad environment for Smile Together?
+
+The accuracy of the detection largely depends on the face recognition performance. To ensure reliable results, the environment should be well-lit, have minimal visual clutter, and place users at an appropriate distance from the camera. On the other hand, if the environment is messy, busy, or does not have sufficient light, it might lead to poor detection results.
+
+#### When will Smile Together break and how will it break?
+
+Smile Together currently uses a heuristic-based smile detection method on cropped face regions instead of a fully trained smile-classification model. Because of this, the system can struggle in situations where facial features are harder to interpret. It may fail in low-light environments, when users are too far from the camera, partially turned away, or moving quickly. In these cases, the system may miss real smiles or incorrectly flag someone as “not smiling.” It can also break when people have subtle or closed-mouth smiles, are talking, or cover their mouths, since the heuristic may not treat those as valid smile cues. Overall, these breakdowns usually appear as false negatives (failing to detect a real smile) or inconsistent triggering when the scene is noisy or ambiguous.
+
+**Common breakdown cases**
+
+1. Poor lighting or occlusion
+ - Failure: Faces aren't reliably detected or mouth shapes are misread.
+ - Cause: Low contrast, shadows, or users turning away.
+ - Result: Missed smiles or false “not smiling” feedback.
+
+2. Subtle or non-standard smiles
+ - Failure: Genuine smiles aren't recognized.
+ - Cause: Expression differences (e.g., closed-mouth smiles, cultural variation).
+ - Result: Inconsistent smile detection across users.
+
+3. Camera angle or distance issues
+ - Failure: Detection works unevenly across the frame.
+ - Cause: Smaller or angled faces reduce confidence.
+ - Result: Distant or partially angled users may not “count” even if smiling.
+
+#### How does Smile Together feel?
+
+Smile Together feels playful and enjoyable for users. Unlike a regular camera, it encourages everyone in the frame to be more aware of each other, turning the photo-taking moment into a fun group interaction. People often laugh, make eye contact, and try to figure out who isn’t smiling enough yet, which creates a light, cooperative atmosphere and turns photo-taking into a more joyful experience.
### Part 2.
Following exploration and reflection from Part 1, finish building your interactive system, and demonstrate it in use with a video.
**\*\*\*Include a short video demonstrating the finished result.\*\*\***
+
+https://github.com/user-attachments/assets/abb44086-3acd-4b8a-a799-374121a67d83
+
+#### Reflections from Users of Smile Together
+
+Good:
+- Most users describe Smile Together as a delightful and surprising experience.
+- They enjoy the moment when the system recognizes a shared smile and responds. It feels playful, human, and emotionally rewarding. Many say it creates a sense of connection even in a short interaction.
+- It promotes eye contact, timing, and cooperation, turning technology into a social connector.
+
+Bad:
+- Users often don’t realize how sensitive the system is to lighting, angles, or timing until they see it fail.
+- They sometimes think the system is “judging” them for not smiling “correctly,” which can make them self-conscious.
diff --git a/Lab 5/smile_check_camera.py b/Lab 5/smile_check_camera.py
new file mode 100644
index 0000000000..51a355bc6f
--- /dev/null
+++ b/Lab 5/smile_check_camera.py
@@ -0,0 +1,839 @@
+"""
+Multi-Person Smile-Check Camera
+A simple interaction prototype that detects multiple faces and checks if everyone is smiling.
+Uses OpenCV Haar Cascade for face detection and image analysis for smile estimation.
+
+Interaction Concept:
+- Detects multiple faces simultaneously
+- Estimates smile score for each face using mouth region analysis
+- Triggers feedback when not everyone is smiling (smile score < 0.65 for >1 second)
+- Multiple feedback modes: audio, LED simulation, screen overlay, progress bar
+"""
+
+import cv2
+import numpy as np
+import time
+import math
+import os
+import subprocess
+import threading
+import struct
+import tempfile
+
+class SmileCheckCamera:
+ def __init__(self):
+ # Initialize OpenCV Haar Cascade Face Detection
+ # This is more reliable on Raspberry Pi and doesn't require MediaPipe
+ cascade_path = cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'
+
+ # Check if cascade file exists, if not try alternative paths
+ if not os.path.exists(cascade_path):
+ # Try common alternative paths on Raspberry Pi
+ alt_paths = [
+ '/usr/share/opencv4/haarcascades/haarcascade_frontalface_default.xml',
+ '/usr/local/share/opencv4/haarcascades/haarcascade_frontalface_default.xml',
+ 'haarcascade_frontalface_default.xml'
+ ]
+ for alt_path in alt_paths:
+ if os.path.exists(alt_path):
+ cascade_path = alt_path
+ break
+
+ try:
+ self.face_cascade = cv2.CascadeClassifier(cascade_path)
+ if self.face_cascade.empty():
+ raise FileNotFoundError(f"Haar cascade file not found: {cascade_path}")
+ print(f"Loaded face detector from: {cascade_path}")
+ except Exception as e:
+ print(f"Error loading face cascade: {e}")
+ print("Falling back to default OpenCV cascade path...")
+ self.face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
+
+ # Face tracking: store (face_id, smile_score, last_update_time, below_threshold_since, smoothed_score)
+ # below_threshold_since: timestamp when face first dropped below threshold (None if above threshold)
+ # smoothed_score: temporally smoothed smile score to reduce jitter
+ self.face_tracking = {}
+ self.face_id_counter = 0
+ self.smoothing_factor = 0.7 # Higher = more smoothing (0.7 means 70% old, 30% new)
+
+ # Feedback state
+ self.feedback_active = False
+ self.feedback_start_time = None
+ self.led_state = False
+ self.last_led_toggle = time.time()
+ self.led_blink_rate = 0.5 # seconds
+
+ # Audio feedback state
+ self.last_audio_play = 0
+ self.audio_interval = 2.0 # Play sound every 2 seconds max when someone isn't smiling
+
+ # Smile detection parameters
+ self.smile_threshold = 0.48 # Higher threshold - more accurate, reduce false positives
+ self.feedback_delay = 1.0 # seconds before triggering feedback
+
+ # Audio feedback (using pygame, aplay, or system beep)
+ self.use_pygame = False
+ self.use_aplay = False
+
+ # Try pygame first (best quality)
+ try:
+ import pygame
+ pygame.mixer.init()
+ self.use_pygame = True
+ print("Audio: Using PyGame for sound notifications")
+ except ImportError:
+ # Try aplay (available on Raspberry Pi)
+ try:
+ # Test if aplay is available
+ result = subprocess.run(['which', 'aplay'],
+ capture_output=True,
+ timeout=1)
+ if result.returncode == 0:
+ self.use_aplay = True
+ print("Audio: Using aplay for sound notifications")
+ else:
+ print("Audio: Using system beep (install pygame for better sound)")
+ except:
+ print("Audio: Using system beep (install pygame for better sound)")
+
+ def calculate_smile_score(self, face_bbox, image):
+ """
+ Calculate smile score based on mouth curvature and shape analysis.
+ Focuses on detecting actual smiles (upward-curving mouth) rather than just mouth opening.
+ """
+ h, w = image.shape[:2]
+
+ # Extract face bounding box coordinates
+ x_min, y_min, face_width, face_height = face_bbox
+ x_max = x_min + face_width
+ y_max = y_min + face_height
+
+ # Ensure coordinates are within image bounds
+ x_min = max(0, x_min)
+ y_min = max(0, y_min)
+ x_max = min(w, x_max)
+ y_max = min(h, y_max)
+
+ if x_max <= x_min or y_max <= y_min or face_width < 30 or face_height < 30:
+ return 0.2 # Default to low score if face too small
+
+ # Extract face region
+ face_region = image[y_min:y_max, x_min:x_max].copy()
+
+ if face_region.size == 0:
+ return 0.2
+
+ # Estimate mouth region (typically in lower 1/3 of face, centered horizontally)
+ face_height_px = y_max - y_min
+ face_width_px = x_max - x_min
+
+ # Mouth region: lower portion of face (narrower region focused on mouth)
+ mouth_y_start = int(face_height_px * 0.60)
+ mouth_y_end = int(face_height_px * 0.85)
+ mouth_x_start = int(face_width_px * 0.25)
+ mouth_x_end = int(face_width_px * 0.75)
+
+ if mouth_y_end <= mouth_y_start or mouth_x_end <= mouth_x_start:
+ return 0.2
+
+ mouth_region = face_region[mouth_y_start:mouth_y_end, mouth_x_start:mouth_x_end]
+
+ if mouth_region.size == 0:
+ return 0.2
+
+ # Convert to grayscale
+ if len(mouth_region.shape) == 3:
+ mouth_gray = cv2.cvtColor(mouth_region, cv2.COLOR_BGR2GRAY)
+ else:
+ mouth_gray = mouth_region
+
+ # Apply Gaussian blur to reduce noise
+ mouth_gray = cv2.GaussianBlur(mouth_gray, (5, 5), 0)
+
+ mouth_h, mouth_w = mouth_gray.shape
+ if mouth_h < 10 or mouth_w < 20:
+ return 0.2
+
+ # Feature 1: Detect upward curvature using edge analysis
+ # Smiles have upward-curving edges, neutral mouths are more horizontal
+ # Use horizontal gradient to detect the smile curve
+
+ # Apply horizontal gradient (Sobel X) to detect vertical edges
+ sobel_x = cv2.Sobel(mouth_gray, cv2.CV_64F, 1, 0, ksize=3)
+ sobel_x = np.abs(sobel_x)
+
+ # Apply vertical gradient (Sobel Y) to detect horizontal edges (smile curve)
+ sobel_y = cv2.Sobel(mouth_gray, cv2.CV_64F, 0, 1, ksize=3)
+ sobel_y = np.abs(sobel_y)
+
+ # For a smile, we expect:
+ # - Strong horizontal gradients in the middle/upper portion (smile curve)
+ # - The curve should be upward (higher gradient in upper half)
+
+ # Divide mouth into thirds: left, center, right
+ left_third = sobel_y[:, :mouth_w//3]
+ center_third = sobel_y[:, mouth_w//3:2*mouth_w//3]
+ right_third = sobel_y[:, 2*mouth_w//3:]
+
+ # Upper half (where smile curve is most visible)
+ upper_half = mouth_gray[:mouth_h//2, :]
+ lower_half = mouth_gray[mouth_h//2:, :]
+
+ # Calculate curvature score based on gradient distribution
+ # In a smile, horizontal gradients (sobel_y) should be stronger in the upper portion
+ upper_sobel_y = sobel_y[:mouth_h//2, :] if mouth_h > 4 else sobel_y
+ lower_sobel_y = sobel_y[mouth_h//2:, :] if mouth_h > 4 else sobel_y
+
+ upper_gradient = np.mean(upper_sobel_y) if upper_sobel_y.size > 0 else 0
+ lower_gradient = np.mean(lower_sobel_y) if lower_sobel_y.size > 0 else 0
+
+ # Smile has stronger gradients in upper portion (upward curve)
+ # Balanced - require noticeable curvature but not too strict
+ curvature_score = 0.0
+ if upper_gradient > 0:
+ gradient_ratio = upper_gradient / (upper_gradient + lower_gradient + 1e-6)
+ # Balanced threshold - not too strict, not too lenient
+ curvature_score = np.clip((gradient_ratio - 0.36) * 2.8, 0.0, 1.0) # Balanced
+ # Give minimum score if gradient difference is noticeable
+ if abs(upper_gradient - lower_gradient) > 8: # Balanced threshold
+ curvature_score = max(curvature_score, 0.30) # Moderate minimum score
+
+ # Feature 2: Analyze mouth shape using contour detection
+ # Smiling mouths have a distinct upward-curving contour
+ edges = cv2.Canny(mouth_gray, 50, 150)
+ contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+
+ shape_score = 0.0
+ if len(contours) > 0:
+ # Find the largest contour (likely the mouth outline)
+ largest_contour = max(contours, key=cv2.contourArea)
+ # Lower threshold for contour area to be more lenient
+ if cv2.contourArea(largest_contour) > mouth_w * mouth_h * 0.05: # Reduced from 0.1
+ # Fit a curve/ellipse to the contour
+ if len(largest_contour) >= 5:
+ try:
+ ellipse = cv2.fitEllipse(largest_contour)
+ # Get ellipse center and axes
+ center, axes, angle = ellipse
+ major_axis, minor_axis = max(axes), min(axes)
+
+ # For a smile, the ellipse should be wider (major axis horizontal)
+ # and angled upward (negative angle indicates upward curve)
+ if major_axis > 0:
+ axis_ratio = minor_axis / major_axis
+ # Wider ellipses (smaller ratio) suggest a smile - more lenient
+ shape_score = np.clip((0.7 - axis_ratio) * 2.5, 0.0, 0.6) # More lenient, higher max
+ except:
+ # If ellipse fitting fails, still give some score for having a contour
+ shape_score = 0.2
+ else:
+ # Even if contour is small, if there are multiple contours, it might indicate a smile
+ if len(contours) >= 2:
+ shape_score = 0.15
+
+ # Feature 3: Corner elevation analysis
+ # Smiling mouths have corners that are elevated relative to center
+ # Analyze the vertical profile of the mouth
+
+ # Get horizontal line profiles at different vertical positions
+ top_row = mouth_gray[0, :]
+ middle_row = mouth_gray[mouth_h//2, :]
+ bottom_row = mouth_gray[-1, :]
+
+ # Calculate the "curvature" by comparing corner positions
+ # In a smile, corners should be higher than the center
+ left_corner_top = np.mean(mouth_gray[:mouth_h//3, :mouth_w//4])
+ left_corner_bottom = np.mean(mouth_gray[2*mouth_h//3:, :mouth_w//4])
+ right_corner_top = np.mean(mouth_gray[:mouth_h//3, 3*mouth_w//4:])
+ right_corner_bottom = np.mean(mouth_gray[2*mouth_h//3:, 3*mouth_w//4:])
+ center_top = np.mean(mouth_gray[:mouth_h//3, mouth_w//3:2*mouth_w//3])
+ center_bottom = np.mean(mouth_gray[2*mouth_h//3:, mouth_w//3:2*mouth_w//3])
+
+ # In a smile, corners are elevated (brighter/different) and center may dip
+ corner_elevation = 0.0
+ if center_top > 0 and center_bottom > 0:
+ left_elevation = abs(left_corner_top - left_corner_bottom) / 255.0
+ right_elevation = abs(right_corner_top - right_corner_bottom) / 255.0
+ avg_elevation = (left_elevation + right_elevation) / 2.0
+ # More sensitive to elevation differences
+ corner_elevation = np.clip(avg_elevation * 4.0, 0.0, 0.7) # Increased sensitivity and max
+
+ # Also give score if corners are simply different from center (even if not elevated)
+ if center_top > 0:
+ corner_diff_left = abs(left_corner_top - center_top) / 255.0
+ corner_diff_right = abs(right_corner_top - center_top) / 255.0
+ avg_corner_diff = (corner_diff_left + corner_diff_right) / 2.0
+ corner_elevation = max(corner_elevation, np.clip(avg_corner_diff * 2.0, 0.0, 0.4))
+
+ # Feature 4: Width increase (but NOT due to opening)
+ # Smiles widen the mouth without necessarily opening it much
+ mouth_width = mouth_x_end - mouth_x_start
+ mouth_height = mouth_y_end - mouth_y_start
+
+ # Calculate aspect ratio, but penalize if mouth is too open (not a smile, just opening)
+ width_ratio = mouth_width / mouth_height if mouth_height > 0 else 1.0
+
+ # Smile should have good width ratio but not be too open
+ # Balanced - accept reasonable range but penalize excessive opening
+ width_score = 0.0
+ if 1.9 <= width_ratio <= 3.9: # Moderate ideal range
+ # Good range for smiles
+ width_score = np.clip((width_ratio - 1.8) / 2.1, 0.0, 0.45) # Moderate max
+ elif 3.9 < width_ratio <= 4.8:
+ # Borderline - might be opening mouth or big smile
+ width_score = np.clip((4.8 - width_ratio) / 0.9, 0.0, 0.22) # Lower score
+ elif width_ratio > 4.8:
+ # Too open - likely just opening mouth, but give tiny score
+ width_score = 0.03 # Very small score instead of zero
+ else:
+ # Too narrow - give moderate score
+ width_score = np.clip(width_ratio / 3.0, 0.0, 0.35) # Moderate max for narrow
+
+ # Combine features with emphasis on curvature and shape (not opening)
+ # Balanced - require good indicators but not overly strict
+ smile_score = (
+ curvature_score * 0.40 + # Curvature is most important (40%)
+ shape_score * 0.29 + # Shape analysis (29%)
+ corner_elevation * 0.19 + # Corner elevation (19%)
+ width_score * 0.12 # Width - moderate weight (12%)
+ )
+
+ # Balanced bonus - reward multiple indicators
+ active_features = sum([1 for score in [curvature_score, shape_score, corner_elevation, width_score] if score > 0.28]) # Moderate threshold
+ if active_features >= 3: # Require 3 indicators
+ smile_score += 0.10 # Moderate bonus
+ elif active_features >= 2:
+ smile_score += 0.06 # Small bonus
+
+ # Moderate boost - balanced
+ smile_score = smile_score * 1.33 # Moderate multiplier
+ smile_score = np.clip(smile_score, 0.0, 1.0)
+
+ # Moderate penalty for low/ambiguous scores
+ if smile_score < 0.34:
+ smile_score = smile_score * 0.74 # Moderate penalty
+
+ # Check: if width is high but curvature is low, might be opening mouth
+ # Balanced threshold
+ if width_score > 0.35 and curvature_score < 0.25:
+ smile_score *= 0.62 # Moderate penalty - likely mouth opening
+
+ # Require reasonable minimum curvature to be considered a smile
+ if curvature_score < 0.18:
+ smile_score *= 0.78 # Light penalty if curvature is very weak
+
+ return smile_score
+
+ def play_audio_feedback(self, play_now=False):
+ """
+ Play audio feedback (notification sound) when someone isn't smiling.
+
+ Args:
+ play_now: If True, play immediately. If False, respect rate limiting.
+ """
+ current_time = time.time()
+
+ # Rate limiting - don't play too frequently
+ if not play_now and (current_time - self.last_audio_play) < self.audio_interval:
+ return
+
+ self.last_audio_play = current_time
+
+ if self.use_pygame:
+ try:
+ import pygame
+ # Generate a notification beep (lower frequency, slightly longer for attention)
+ sample_rate = 44100
+ duration = 0.3 # Longer duration for better noticeability
+ frequency = 350 # Lower frequency (more noticeable)
+
+ frames = int(duration * sample_rate)
+ arr = np.zeros((frames, 2), dtype=np.float32)
+
+ # Create a beep with slight fade-in/fade-out for smoother sound
+ for i in range(frames):
+ # Generate sine wave with envelope for smoother sound
+ t = i / sample_rate
+ envelope = 1.0
+ if i < frames * 0.1: # Fade in
+ envelope = i / (frames * 0.1)
+ elif i > frames * 0.9: # Fade out
+ envelope = (frames - i) / (frames * 0.1)
+
+ wave = np.sin(2 * np.pi * frequency * t) * envelope
+ arr[i][0] = wave
+ arr[i][1] = wave # Stereo
+
+ # Create and play sound
+ sound = pygame.sndarray.make_sound((arr * 32767).astype(np.int16))
+ sound.play()
+
+ except Exception as e:
+ print(f"Audio error: {e}")
+ elif self.use_aplay:
+ # Use aplay to generate a beep on Raspberry Pi
+ try:
+ # Generate a simple beep tone
+ sample_rate = 44100
+ duration = 0.3
+ frequency = 350
+
+ # Create WAV file data
+ num_samples = int(sample_rate * duration)
+ samples = []
+
+ for i in range(num_samples):
+ t = i / sample_rate
+ envelope = 1.0
+ if i < num_samples * 0.1:
+ envelope = i / (num_samples * 0.1)
+ elif i > num_samples * 0.9:
+ envelope = (num_samples - i) / (num_samples * 0.1)
+
+ sample = int(32767 * 0.3 * np.sin(2 * np.pi * frequency * t) * envelope)
+ samples.append(struct.pack('Instruction (Click to Expand)
Now that you have experimented with one or more of these sense-making systems **characterize their behavior**. During the lecture, we mentioned questions to help characterize a material: @@ -191,10 +318,63 @@ During the lecture, we mentioned questions to help characterize a material: * What are other properties/behaviors of X? * How does X feel? -**\*\*\*Include a short video demonstrating the answers to these questions.\*\*\*** + +
+
+
+Prep (Click to Expand)
1. Pull the new changes 2. Read: [The Presence Table](https://dl.acm.org/doi/10.1145/1935701.1935800) ([video](https://vimeo.com/15932020)) -## Overview +
+
+
---
## Part A: MQTT Messaging
+Overview (Click to Expand)
Build interactive systems where **multiple devices communicate over a network** using MQTT messaging. Work in teams of 3+ with Raspberry Pis. @@ -20,10 +24,15 @@ Build interactive systems where **multiple devices communicate over a network** - B: Try collaborative pixel grid demo - C: Build your own distributed system +
+
**💡 Brainstorm 5 ideas for messaging between devices**
+1. **Storyteller game**: Start randomly from a person’s pi, using a word chain structure. Participants collaboratively weave a narrative by linking words, where each subsequent word must begin with the last letter of the previous one.
+2. An online forum, such as Poll Everywhere, where everyone can **participate in a real-time discussion**. A central moderator publishes the questions or topics, and all the other users can express their opinion through pi.
+3. **School announcements**: whenever the school sends out an announcement, it’s delivered directly to each student’s device. Students can also use the device to share useful information with each other.
+4. **Personal Data Sharing**: sync all personal device data, such as notes, health data, and plans, across all personal devices without being limited to a single brand.
+5. When 2+ Pis come within Wi-Fi range, they automatically open a chat window which could **exchange personal symbols** (emojis, sound or text that represents its user’s mood of the day).
+
+
---
## Part B: Collaborative Pixel Grid
+Part A Spec (Click to Expand)
+ MQTT = lightweight messaging for IoT. Publish/subscribe model with central broker. **Concepts:** @@ -56,13 +65,24 @@ mosquitto_pub -h farlab.infosci.cornell.edu -p 1883 -t 'IDD/test/yourname' -m 'H **🔧 Debug Tool:** View all MQTT messages in real-time at `http://farlab.infosci.cornell.edu:5001`  +
+
**📸 Include: Screenshot of grid + photo of your Pi setup**
+
+
---
## Part C: Make Your Own
+### Color Scavenger Hunt
+
+## 1\. Project Description
+
+### What does it do?
+
+Our project is a real-time, competitive "Color Scavenger Hunt" game. A central "Game Master" (a laptop) publishes a target color (e.g., "RED") over MQTT. All players (using Raspberry Pis with APDS-9960 sensors) must then race to find a real-world object of that color and scan it.
+
+### Why is it interesting?
-**Requirements:**
-- 3+ people, 3+ Pis
-- Each Pi contributes sensor input via MQTT
-- Meaningful or fun interaction
+This project creates a fun, high-energy interaction that bridges the digital and physical worlds. It uses a distributed system (MQTT) not just for passive data display, but as the core mechanic for a fast-paced game. The system has to instantly manage game state, identify a single winner from multiple simultaneous inputs, and reset for the next round, all using lightweight messages.
-**Ideas:**
+### What is the user experience?
+
+A player's Pi screen displays, "Waiting for game to start...". Suddenly, it updates: "**Find: [ BLUE ]**". The player frantically looks around, grabs a blue water bottle, and points the sensor at it. Their screen flashes: "**You found it\!**". A moment later, all player screens update with the winner: "**Player 2 wins\!**" After a 3-second pause, a new round begins: "**Find: [ GREEN ]**".
+
+## 2\. Architecture Diagram
+
+Our system is built on a publish/subscribe model using one central server (Game Master) and multiple Pi clients (Players).
+
+```
+ +-------------------+
+ | MQTT Broker |
+ | (farlab.infosci...) |
+ +-------------------+
+ ^ |
+ | |
+(Pub/Sub: All Topics) . . . . . . . | | . . . . . . . (Pub/Sub: All Topics)
+ | v
++--------------------------+ +--------------------------+ +--------------------------+
+| Pi 1 (Player) | | Server (Game Master) | | Pi 2 (Player) |
+| [Input] APDS-9960 Sensor | | [Input] MQTT Sub | | [Input] APDS-9960 Sensor |
+| [Compute] Check Color Match | | [Compute] Game Logic | | [Compute] Check Color Match |
+| [Output] MQTT Pub, Screen | | [Output] MQTT Pub | | [Output] MQTT Pub, Screen |
++--------------------------+ +--------------------------+ +--------------------------+
+```
+
+ * **Inputs:**
+ * **Pi:** Reads (R, G, B) values from the APDS-9960 sensor.
+ * **Pi:** Subscribes to `IDD/game/hunt/master` and `IDD/game/hunt/winner` topics.
+ * **Server:** Subscribes to the `IDD/game/hunt/player_found` topic.
+ * **Computation:**
+ * **Pi:** Continuously compares its sensor's (R, G, B) value against the current target color.
+ * **Server:** Runs the main game loop: selects a random color, publishes it, listens for the *first* player to respond, validates the winner, and publishes the result.
+ * **Outputs:**
+ * **Server:** Publishes the target color to `IDD/game/hunt/master`.
+ * **Server:** Publishes the round's winner to `IDD/game/hunt/winner`.
+ * **Pi:** Publishes its unique name (e.g., `Pi_1_Ariel`) to `IDD/game/hunt/player_found` when it detects a match.
+ * **Pi:** Displays the current game state on its screen.
+
+## 3\. Build Documentation
+
+Part B Spec (Click to Expand)
+ Each Pi = one pixel, controlled by RGB sensor, displayed in real-time grid. **Architecture:** `Pi (sensor) → MQTT → Server → Web Browser` @@ -120,66 +140,269 @@ python pixel_grid_publisher.py Hold colored objects near sensor to change your pixel!  +
+
-## Debugging Tools
+Code Files (Click to Expand)
**Server files:** - `app.py` - Pixel grid server (Flask + WebSocket + MQTT) @@ -196,9 +419,10 @@ Replace this README with your documentation: - `templates/controller.html` - Color picker - `templates/mqtt_viewer.html` - Message viewer ---- +
+
----
-## Troubleshooting
+Debugging Tools (Click to Expand)
**MQTT Message Viewer:** `http://farlab.infosci.cornell.edu:5001` - See all MQTT messages in real-time @@ -210,10 +434,11 @@ Replace this README with your documentation: # See all IDD messages mosquitto_sub -h farlab.infosci.cornell.edu -p 1883 -t "IDD/#" -u idd -P "device@theFarm" ``` +
+
-
----
-
-## Submission Checklist
+Troubleshooting (Click to Expand)
**MQTT:** Broker `farlab.infosci.cornell.edu:1883`, user `idd`, pass `device@theFarm` @@ -222,11 +447,10 @@ mosquitto_sub -h farlab.infosci.cornell.edu -p 1883 -t "IDD/#" -u idd -P "device **Grid:** Verify server running, check MQTT in console, test with web controller **Pi venv:** Make sure to activate: `source .venv/bin/activate` +
+
----
+Submission Checklist (Click to Expand)
Before submitting: - [ ] Delete prep/instructions above @@ -237,7 +461,11 @@ Before submitting: - [ ] List team names at top **Your README = story of what YOU built!** +
+
diff --git a/Lab 6/assets/grid.png b/Lab 6/assets/grid.png
new file mode 100644
index 0000000000..f24980de77
Binary files /dev/null and b/Lab 6/assets/grid.png differ
diff --git a/Lab 6/assets/pi_setup.jpg b/Lab 6/assets/pi_setup.jpg
new file mode 100644
index 0000000000..13c6c03c08
Binary files /dev/null and b/Lab 6/assets/pi_setup.jpg differ
diff --git a/Lab 6/game_master.py b/Lab 6/game_master.py
new file mode 100644
index 0000000000..2e099b7b7c
--- /dev/null
+++ b/Lab 6/game_master.py
@@ -0,0 +1,83 @@
+import paho.mqtt.client as mqtt
+import time
+import random
+
+# --- MQTT 設定 ---
+BROKER = "farlab.infosci.cornell.edu"
+PORT = 1883
+USER = "idd"
+PASS = "device@theFarm"
+CLIENT_ID = "game_master_server"
+
+# --- 遊戲設定 ---
+TARGET_COLORS = ["RED", "GREEN", "BLUE"]
+PLAYER_TOPIC = "IDD/game/hunt/player_found"
+MASTER_TOPIC = "IDD/game/hunt/master"
+WINNER_TOPIC = "IDD/game/hunt/winner"
+
+# --- 遊戲狀態變數 ---
+current_target_color = None
+round_winner = None
+
+# --- MQTT 回調 (Callback) ---
+def on_connect(client, userdata, flags, rc):
+ print(f"Connected to broker with result code {rc}")
+ # 訂閱玩家的「搶答」頻道
+ client.subscribe(PLAYER_TOPIC)
+
+def on_message(client, userdata, msg):
+ global round_winner
+
+ # 檢查是否已經有勝利者 (避免多人同時獲勝)
+ if round_winner is None:
+ player_name = msg.payload.decode()
+ print(f"WINNER DETECTED: {player_name}")
+
+ # 設置勝利者
+ round_winner = player_name
+
+ # 廣播勝利者
+ winner_message = f"{player_name} wins this round!"
+ client.publish(WINNER_TOPIC, winner_message)
+
+# --- 主程式 ---
+client = mqtt.Client(CLIENT_ID)
+client.username_pw_set(USER, PASS)
+client.on_connect = on_connect
+client.on_message = on_message
+
+client.connect(BROKER, PORT, 60)
+client.loop_start() # 在背景線程中處理 MQTT
+
+print("Game Master is running...")
+
+try:
+ while True:
+ # 1. 重置回合狀態
+ round_winner = None
+
+ # 2. 選擇並發布新顏色
+ current_target_color = random.choice(TARGET_COLORS)
+ print(f"\n--- NEW ROUND ---")
+ print(f"Telling players to find: {current_target_color}")
+ client.publish(MASTER_TOPIC, current_target_color)
+
+ # 3. 等待勝利者出現 (on_message 會處理)
+ # 這裡我們給玩家 10 秒鐘時間
+ start_time = time.time()
+ while round_winner is None and (time.time() - start_time) < 10:
+ time.sleep(0.1)
+
+ # 4. 如果 10 秒後沒人獲勝
+ if round_winner is None:
+ print("Time's up! No winner this round.")
+ client.publish(WINNER_TOPIC, "Time's up! No winner.")
+
+ # 5. 等待 3 秒進入下一輪
+ print("Next round in 3 seconds...")
+ time.sleep(3)
+
+except KeyboardInterrupt:
+ print("Game shutting down...")
+ client.loop_stop()
+ client.disconnect()
diff --git a/Lab 6/player.py b/Lab 6/player.py
new file mode 100644
index 0000000000..7da375bbf0
--- /dev/null
+++ b/Lab 6/player.py
@@ -0,0 +1,107 @@
+import paho.mqtt.client as mqtt
+import time
+from adafruit_apds9960.apds9960 import APDS9960
+import board
+
+# --- 感測器設定 ---
+i2c = board.I2C()
+apds = APDS9960(i2c)
+apds.enable_color = True
+
+# --- MQTT 設定 ---
+BROKER = "farlab.infosci.cornell.edu"
+PORT = 1883
+USER = "idd"
+PASS = "device@theFarm"
+PLAYER_NAME = "Pi_1_Ariel" # !! 每個 Pi 都要改成自己的名字 !!
+CLIENT_ID = f"player_{PLAYER_NAME}"
+
+# --- 訂閱的 Topics ---
+MASTER_TOPIC = "IDD/game/hunt/master"
+WINNER_TOPIC = "IDD/game/hunt/winner"
+# --- 發布的 Topic ---
+PLAYER_TOPIC = "IDD/game/hunt/player_found"
+
+# --- 遊戲狀態變數 ---
+current_target_color = None
+i_have_won_this_round = False
+
+# --- 顏色判斷邏輯 ---
+def check_color_match(r, g, b, target_color):
+ if target_color is None:
+ return False
+
+ MIN_CLEAR = 60
+ c = r + g + b
+ if c < MIN_CLEAR:
+ return False
+
+ RATIO = 1.45
+ if target_color == "RED":
+ return (r > g * RATIO) and (r > b * RATIO)
+ if target_color == "GREEN":
+ return (g > r * RATIO) and (g > b * RATIO)
+ if target_color == "BLUE":
+ return (b > r * RATIO) and (b > g * RATIO)
+ return False
+
+# --- MQTT 回調 (Callback) ---
+def on_connect(client, userdata, flags, rc):
+ print(f"Connected as {PLAYER_NAME} with code {rc}")
+ # 訂閱遊戲主持人的指令
+ client.subscribe(MASTER_TOPIC)
+ client.subscribe(WINNER_TOPIC)
+
+def on_message(client, userdata, msg):
+ global current_target_color, i_have_won_this_round
+
+ payload = msg.payload.decode()
+
+ if msg.topic == MASTER_TOPIC:
+ # 收到新回合的目標
+ current_target_color = payload
+ i_have_won_this_round = False # 重置狀態
+ print(f"*** New Target: Find [{current_target_color}]! ***")
+ # 可以在這裡更新 Pi 的螢幕
+
+ elif msg.topic == WINNER_TOPIC:
+ # 收到勝利者廣播
+ print(f"*** Game Status: {payload} ***")
+ current_target_color = None # 停止感測，直到下一輪
+
+# --- 主程式 ---
+client = mqtt.Client(CLIENT_ID)
+client.username_pw_set(USER, PASS)
+client.on_connect = on_connect
+client.on_message = on_message
+
+client.connect(BROKER, PORT, 60)
+client.loop_start()
+
+print(f"Player {PLAYER_NAME} is ready...")
+
+try:
+ while True:
+ # 只有在有目標顏色，且我還沒贏的時候，才進行感測
+ if current_target_color is not None and not i_have_won_this_round:
+
+ # 讀取顏色
+ r, g, b, c = apds.color_data
+ print(f"Sensing: R={r}, G={g}, B={b}") # Debug 用
+
+ # 檢查是否匹配
+ if check_color_match(r, g, b, current_target_color):
+ print(f"MATCH FOUND! I see {current_target_color}!")
+
+ # 標記我已經贏了，避免重複發送
+ i_have_won_this_round = True
+
+ # 發送搶答訊息！
+ client.publish(PLAYER_TOPIC, PLAYER_NAME)
+
+ time.sleep(0.1) # 避免過於頻繁
+
+except KeyboardInterrupt:
+ print("Player shutting down...")
+ client.loop_stop()
+ client.disconnect()
diff --git a/Project/LumiTune-webpage/README.md b/Project/LumiTune-webpage/README.md
new file mode 100644
index 0000000000..4543febd0b
--- /dev/null
+++ b/Project/LumiTune-webpage/README.md
@@ -0,0 +1,11 @@
+
+ # LumiTune Controller Page
+
+ This is a code bundle for LumiTune Controller Page. The original project is available at https://www.figma.com/design/xxHdRY90vdWUS7nBasUbDt/Responsive-Time-Machine-Controller-Page.
+
+ ## Running the code
+
+ Run `npm i` to install the dependencies.
+
+ Run `npm run dev` to start the development server.
+
\ No newline at end of file
diff --git a/Project/LumiTune-webpage/index.html b/Project/LumiTune-webpage/index.html
new file mode 100644
index 0000000000..1c5afcc187
--- /dev/null
+++ b/Project/LumiTune-webpage/index.html
@@ -0,0 +1,16 @@
+
+
+
+
+
+
+ Resources (Click to Expand)
Resources: [MQTT Guide](https://www.hivemq.com/mqtt-essentials/) | [Paho Python](https://www.eclipse.org/paho/index.php?page=clients/python/docs/index.php) | [Flask-SocketIO](https://flask-socketio.readthedocs.io/) + +
+ {/* Main content */}
+
+ );
+}
diff --git a/Project/LumiTune-webpage/src/Attributions.md b/Project/LumiTune-webpage/src/Attributions.md
new file mode 100644
index 0000000000..9b7cd4e134
--- /dev/null
+++ b/Project/LumiTune-webpage/src/Attributions.md
@@ -0,0 +1,3 @@
+This Figma Make file includes components from [shadcn/ui](https://ui.shadcn.com/) used under [MIT license](https://github.com/shadcn-ui/ui/blob/main/LICENSE.md).
+
+This Figma Make file includes photos from [Unsplash](https://unsplash.com) used under [license](https://unsplash.com/license).
\ No newline at end of file
diff --git a/Project/LumiTune-webpage/src/components/figma/ImageWithFallback.tsx b/Project/LumiTune-webpage/src/components/figma/ImageWithFallback.tsx
new file mode 100644
index 0000000000..0e26139bea
--- /dev/null
+++ b/Project/LumiTune-webpage/src/components/figma/ImageWithFallback.tsx
@@ -0,0 +1,27 @@
+import React, { useState } from 'react'
+
+const ERROR_IMG_SRC =
+ 'data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iODgiIGhlaWdodD0iODgiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgc3Ryb2tlPSIjMDAwIiBzdHJva2UtbGluZWpvaW49InJvdW5kIiBvcGFjaXR5PSIuMyIgZmlsbD0ibm9uZSIgc3Ryb2tlLXdpZHRoPSIzLjciPjxyZWN0IHg9IjE2IiB5PSIxNiIgd2lkdGg9IjU2IiBoZWlnaHQ9IjU2IiByeD0iNiIvPjxwYXRoIGQ9Im0xNiA1OCAxNi0xOCAzMiAzMiIvPjxjaXJjbGUgY3g9IjUzIiBjeT0iMzUiIHI9IjciLz48L3N2Zz4KCg=='
+
+export function ImageWithFallback(props: React.ImgHTMLAttributes
+ {/* Hero Section */}
+
+
+ {/* Main title */}
+
+
+ {/* Central Panel */}
+ + LumiTune Controller +
+ + {/* Subtitle */} ++ Choose a decade, let the light pick the mood, and control music with gesture. +
+ + {/* Small MQTT status */} +
+ MQTT: {isConnected ? 'Connected ✅' : 'Disconnected ❌'}
+ {mqttError && ({mqttError})}
+
+
+
+
+
+ {/* Footer hint */}
+
+ Designed with care ✨
+
+
+
+ ) : (
+
+ 
+
+ {children}
+
+ {children}
+
+
+
+
+ );
+}
+
+function CarouselItem({ className, ...props }: React.ComponentProps<"div">) {
+ const { orientation } = useCarousel();
+
+ return (
+
+ );
+}
+
+function CarouselPrevious({
+ className,
+ variant = "outline",
+ size = "icon",
+ ...props
+}: React.ComponentProps
+
+ {children}
+
+
+