An AI-powered web application that detects emotions from facial expressions in real-time and recommends personalized music playlists based on the detected mood.
- Features
- Architecture
- Tech Stack
- Prerequisites
- Quick Start with Docker
- Manual Installation
- Dataset Setup
- Training the Model
- API Documentation
- Project Structure
- Configuration
- Deployment
- Troubleshooting
- Contributing
- License
- π― Real-time Emotion Detection - Detects 7 different emotions from facial expressions
- π΅ Music Recommendations - Suggests Spotify playlists based on detected emotions
- πΈ Live Camera Feed - Capture photos directly from webcam
- π€ Deep Learning CNN - Custom-trained Convolutional Neural Network
- π¨ Modern React UI - Responsive, mobile-friendly interface with Tailwind CSS
- π³ Docker Ready - One-command deployment with Docker Compose
- π RESTful API - Clean Flask backend with JSON responses
- π High Accuracy - ~70-80% accuracy on emotion detection
- π Production Ready - Nginx reverse proxy with optimized settings
| Emotion | Icon | Music Genre |
|---|---|---|
| π Happy | π | Pop, Party, Summer |
| π’ Sad | π’ | Chill, Acoustic, Rainy-day |
| π Angry | π | Rock, Metal, Power |
| π Neutral | π | Study, Ambient, Focus |
| π² Surprise | π² | Dance, Pop, Exciting |
| π¨ Fear | π¨ | Calm, Peaceful, Acoustic |
| π€’ Disgust | π€’ | Indie, Alternative, Fresh |
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Docker Compose β
β β
β ββββββββββββββββββββββββββ ββββββββββββββββββββββββ β
β β emotion-frontend β β emotion-backend β β
β β (nginx:alpine) βββββΊβ (python:3.10-slim) β β
β β β β β β
β β - React App (build) β β - Flask API β β
β β - Nginx Server β β - TensorFlow Model β β
β β - Reverse Proxy β β - OpenCV β β
β β Port: 80 β Host β β - Spotify API β β
β β β β Port: 8000 (internal)β β
β ββββββββββββββββββββββββββ ββββββββββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
User Browser (http://localhost)
β
Nginx (Port 80) - emotion-frontend container
β
βββ /api/* β Proxy to backend:8000/api/*
βββ /health β Proxy to backend:8000/health
βββ /* β Serve React static files
β
Flask API (Port 8000) - emotion-backend container
β
βββ Face Detection (OpenCV)
βββ Emotion Prediction (TensorFlow/Keras)
βββ Music Recommendations (Spotify API)
- Python 3.10 - Core programming language
- TensorFlow/Keras - Deep learning framework
- Flask - Web framework
- Flask-CORS - Cross-Origin Resource Sharing
- OpenCV - Computer vision & face detection
- Spotipy - Spotify Web API wrapper
- NumPy - Numerical computing
- Pillow - Image processing
- React 18 - UI framework
- Tailwind CSS - Utility-first CSS framework
- Axios - HTTP client
- React Webcam - Camera integration (if used)
- Nginx - Production web server
- Docker - Containerization
- Docker Compose - Multi-container orchestration
- Nginx - Reverse proxy, static file serving, gzip compression
- Multi-stage builds - Optimized Docker images
For Docker Deployment (Recommended):
- Docker 20.10+
- Docker Compose 2.0+
For Manual Installation:
- Python 3.10+
- Node.js 18+
- npm or yarn
- RAM: Minimum 8GB (16GB recommended for training)
- Storage: ~5GB free space (dataset + model + containers)
- OS: Windows 10/11, macOS, or Linux
- Webcam: For live capture (optional)
- Internet: For Spotify API and npm packages
- Go to Spotify Developer Dashboard
- Log in and create a new app
- Set redirect URI:
http://localhost:5000/callback - Copy Client ID and Client Secret
git clone https://github.com/yourusername/emotion-music-recommender.git
cd emotion-music-recommenderCreate .env in the project root:
# .env
SPOTIFY_CLIENT_ID=your_spotify_client_id_here
SPOTIFY_CLIENT_SECRET=your_spotify_client_secret_hereIMPORTANT: The model must be trained before Docker containers can run!
# Navigate to backend
cd backend
# Create virtual environment
python -m venv venv
# Activate virtual environment
# Windows:
venv\Scripts\activate
# macOS/Linux:
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Verify dataset structure
python train_model.py
# If dataset is ready, train the model (takes 1-3 hours)
python train_model.py
# Expected output:
# β Model saved to models/emotion_model.keras# Go back to project root
cd ..
# Build and start all containers
docker-compose up --build
# Or run in detached mode:
docker-compose up -d --buildOpen your browser and navigate to:
http://localhost
Or if you're deploying on a server:
http://your_server_ip
# Stop containers
docker-compose down
# Stop and remove volumes
docker-compose down -vIf you prefer to run without Docker:
cd backend
# Create virtual environment
python -m venv venv
# Activate virtual environment
# Windows:
venv\Scripts\activate
# macOS/Linux:
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Train model (if not done already)
python train_model.py
# Run Flask server
python app.py
# Server will run on http://localhost:8000cd frontend
# Install dependencies
npm install
# Create .env file for frontend
echo "REACT_APP_BACKEND_URL=http://localhost:8000" > .env
# Run development server
npm start
# Frontend will run on http://localhost:3000
# Or build for production
npm run build
# Build files will be in frontend/build/Your dataset should be organized as follows:
backend/data/1/
βββ train/
β βββ angry/
β β βββ img001.jpg
β β βββ img002.jpg
β β βββ ...
β βββ disgust/
β βββ fear/
β βββ happy/
β βββ neutral/
β βββ sad/
β βββ surprise/
βββ test/
βββ angry/
βββ disgust/
βββ fear/
βββ happy/
βββ neutral/
βββ sad/
βββ surprise/
Option 1: FER2013 from Kaggle
- Visit Kaggle FER2013 Dataset
- Download and extract to
backend/data/1/ - Ensure folder structure matches above
Option 2: Custom Dataset
Organize your own images following the structure above. Requirements:
- Images should be faces (will be auto-cropped)
- Minimum 1000 images per emotion recommended
- Supported formats:
.jpg,.jpeg,.png - Minimum resolution: 48x48 pixels
cd backend
python -c "
import os
data_path = 'data/1'
for split in ['train', 'test']:
print(f'\n{split.upper()} SET:')
total = 0
for emotion in ['angry', 'disgust', 'fear', 'happy', 'neutral', 'sad', 'surprise']:
path = os.path.join(data_path, split, emotion)
if os.path.exists(path):
count = len([f for f in os.listdir(path) if f.endswith(('.jpg', '.jpeg', '.png'))])
print(f' {emotion:10s}: {count:6d} images')
total += count
print(f' {'Total':10s}: {total:6d} images')
"Expected Output:
TRAIN SET:
angry : 3995 images
disgust : 436 images
fear : 4097 images
happy : 7215 images
neutral : 4965 images
sad : 4830 images
surprise : 3171 images
Total : 28709 images
TEST SET:
angry : 958 images
disgust : 111 images
fear : 1024 images
happy : 1774 images
neutral : 1233 images
sad : 1247 images
surprise : 831 images
Total : 7178 images
cd backend
# Activate virtual environment
source venv/bin/activate # macOS/Linux
# OR
venv\Scripts\activate # Windows
# Start training
python train_model.py======================================================================
FACIAL EMOTION RECOGNITION - CNN MODEL TRAINING
======================================================================
Training started at: 2024-12-18 10:30:00
Verifying dataset structure...
β Found 28,709 training images
β Found 7,178 validation images
β Number of classes: 7
Creating CNN model...
Model Architecture:
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 48, 48, 32) 320
batch_normalization (None, 48, 48, 32) 128
conv2d_1 (Conv2D) (None, 48, 48, 32) 9,248
...
=================================================================
Total params: 3,456,789
Trainable params: 3,452,693
Non-trainable params: 4,096
Training Configuration:
Batch Size: 64
Steps per Epoch: 448
Validation Steps: 112
Max Epochs: 50
Learning Rate: 0.001
======================================================================
Starting training...
======================================================================
Epoch 1/50
448/448 [==============================] - 67s 149ms/step
loss: 1.7234 - accuracy: 0.3456 - val_loss: 1.5432 - val_accuracy: 0.4123
Epoch 2/50
448/448 [==============================] - 64s 143ms/step
loss: 1.5123 - accuracy: 0.4234 - val_loss: 1.4123 - val_accuracy: 0.4567
...
Epoch 50/50
448/448 [==============================] - 62s 138ms/step
loss: 0.7234 - accuracy: 0.7856 - val_loss: 0.8234 - val_accuracy: 0.7456
======================================================================
Evaluating model on test set...
======================================================================
π Final Results:
Test Loss: 0.9298
Test Accuracy: 65.54%
======================================================================
Generating visualizations...
======================================================================
β Training history plot saved to plots/training_history.png
β Confusion matrix saved to plots/confusion_matrix.png
======================================================================
Classification Report:
======================================================================
precision recall f1-score support
Angry 0.56 0.62 0.59 958
Disgust 0.68 0.41 0.51 111
Fear 0.56 0.33 0.42 1024
Happy 0.87 0.87 0.87 1774
Neutral 0.54 0.75 0.63 1233
Sad 0.56 0.49 0.52 1247
Surprise 0.75 0.78 0.76 831
accuracy 0.66 7178
macro avg 0.64 0.61 0.62 7178
weighted avg 0.66 0.66 0.65 7178
β Model saved to models/emotion_model.keras
======================================================================
Training completed at: 2025-12-17 16:38:33
======================================================================
======================================================================
β TRAINING COMPLETE!
======================================================================
Next steps:
1. Check 'models/emotion_model.h5' - your trained model
2. Check 'plots/' folder for training visualizations
3. Run 'python app.py' to start the Flask backend
4. Connect your React frontend to http://localhost:5000
======================================================================
Edit backend/train_model.py to customize:
# Dataset path
DATASET_PATH = 'data/1'
# Model save location
MODEL_SAVE_PATH = 'models/emotion_model.keras'
# Hyperparameters
IMG_SIZE = 48 # Image dimensions
BATCH_SIZE = 64 # Training batch size
EPOCHS = 50 # Number of epochs
LEARNING_RATE = 0.001 # Adam optimizer learning rate| Hardware | Approximate Time |
|---|---|
| CPU only (i7/i9) | 3-5 hours |
| GPU (GTX 1660) | 1.5-2 hours |
| GPU (RTX 3060) | 1-1.5 hours |
| GPU (RTX 4090) | 30-45 minutes |
# When using Docker
http://localhost/api
# When running manually
http://localhost:8000/api
GET /healthResponse:
{
"status": "healthy",
"model_loaded": true,
"face_cascade_loaded": true,
"spotify_connected": true,
"supported_emotions": [
"angry", "disgust", "fear", "happy",
"neutral", "sad", "surprise"
],
"timestamp": "2024-12-18T10:30:00.000Z"
}POST /api/detect-emotion
Content-Type: application/jsonRequest:
{
"image": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAA..."
}Success Response:
{
"success": true,
"emotion": "happy",
"confidence": 87.34,
"face_location": {
"x": 120,
"y": 80,
"width": 200,
"height": 200
},
"playlists": [
{
"playlist_name": "π Happy Vibes Collection",
"playlist_url": "https://open.spotify.com/playlist/...",
"tracks": [
{
"title": "Happy",
"artist": "Pharrell Williams",
"duration": "3:53",
"genre": "Pop",
"spotify_url": "https://open.spotify.com/track/...",
"preview_url": "https://p.scdn.co/mp3-preview/...",
"album_art": "https://i.scdn.co/image/..."
}
]
}
],
"timestamp": "2024-12-18T10:30:00.000Z"
}Error Response:
{
"success": false,
"error": "No face detected in the image. Please ensure your face is clearly visible."
}GET /api/emotionsResponse:
{
"emotions": [
"angry", "disgust", "fear", "happy",
"neutral", "sad", "surprise"
],
"count": 7,
"descriptions": {
"angry": "Feeling mad or frustrated",
"disgust": "Feeling repulsed or averse",
"fear": "Feeling scared or anxious",
"happy": "Feeling joyful or content",
"neutral": "Feeling calm or indifferent",
"sad": "Feeling down or melancholic",
"surprise": "Feeling shocked or amazed"
}
}GET /api/test-spotifyResponse:
{
"connected": true,
"message": "Spotify API working correctly",
"test_result": true
}emotion-music-recommender/
β
βββ .env # Environment variables
βββ docker-compose.yml # Docker Compose configuration
βββ README.md # This file
β
βββ backend/
β βββ Dockerfile # Backend container definition
β βββ requirements.txt # Python dependencies
β βββ app.py # Flask API server
β βββ train_model.py # Model training script
β β
β βββ data/
β β βββ 1/
β β β βββ train/ # Training images
β β β β βββ angry/
β β β β βββ disgust/
β β β β βββ fear/
β β β β βββ happy/
β β β β βββ neutral/
β β β β βββ sad/
β β β β βββ surprise/
β β β βββ test/ # Test images
β β β βββ angry/
β β β βββ disgust/
β β β βββ fear/
β β β βββ happy/
β β β βββ neutral/
β β β βββ sad/
β β β βββ surprise/
β β βββ dataset_download.py # Dataset download helper
β β
β βββ models/
β β βββ emotion_model.keras # Trained model (created after training)
β β
β βββ plots/
β βββ training_history.png # Training metrics visualization
β βββ confusion_matrix.png # Model performance matrix
β
βββ frontend/
βββ Dockerfile # Frontend container definition
βββ nginx.conf # Nginx configuration
βββ package.json # Node.js dependencies
βββ package-lock.json # Locked dependencies
βββ tailwind.config.js # Tailwind CSS configuration
βββ README.md # Frontend documentation
β
βββ public/
β βββ index.html # HTML template
β βββ favicon.ico # App icon
β βββ manifest.json # PWA manifest
β βββ robots.txt # SEO robots file
β
βββ src/
βββ App.js # Main React component
βββ App.css # App styles
βββ index.js # Entry point
βββ index.css # Global styles
βββ ... # Other components
File: docker-compose.yml
version: "3.9"
services:
backend:
build:
context: ./backend
container_name: emotion-backend
expose:
- "8000"
environment:
SPOTIFY_CLIENT_ID: ${SPOTIFY_CLIENT_ID}
SPOTIFY_CLIENT_SECRET: ${SPOTIFY_CLIENT_SECRET}
restart: unless-stopped
frontend:
build:
context: ./frontend
container_name: emotion-frontend
ports:
- "80:80"
environment:
REACT_APP_BACKEND_URL: http://your_server_ip:8000
depends_on:
- backend
restart: unless-stoppedFile: backend/app.py
# Port configuration
# When using Docker, Flask listens on port 8000
if __name__ == '__main__':
app.run(debug=False, host='0.0.0.0', port=8000)
# Model path
MODEL_PATH = 'models/emotion_model.keras'
# Emotion labels (MUST match training order)
EMOTION_LABELS = ['angry', 'disgust', 'fear', 'happy',
'neutral', 'sad', 'surprise']File: frontend/nginx.conf
Key configurations:
- Proxies
/api/*requests tobackend:8000 - Serves React static files
- Enables gzip compression
- Caches static assets
- Security headers
Environment Variables: Create frontend/.env for development:
REACT_APP_BACKEND_URL=http://localhost:8000File: .env (project root)
# Required: Spotify API credentials
SPOTIFY_CLIENT_ID=your_client_id_here
SPOTIFY_CLIENT_SECRET=your_client_secret_here
# Optional: Flask configuration
FLASK_ENV=production
FLASK_DEBUG=FalseIn docker-compose.yml, update the frontend environment:
frontend:
environment:
REACT_APP_BACKEND_URL: http://YOUR_SERVER_IP:8000# Transfer files to server
scp -r emotion-music-recommender user@your_server:/home/user/
# SSH into server
ssh user@your_server
# Navigate to project
cd /home/user/emotion-music-recommender
# Create .env file
nano .env
# Add your Spotify credentials
# Build and start
docker-compose up -d --build
# Check logs
docker-compose logs -f
# Check status
docker-compose ps# Allow HTTP traffic
sudo ufw allow 80/tcp
# Allow HTTPS (if using SSL)
sudo ufw allow 443/tcp
# Enable firewall
sudo ufw enable- Trained model exists at
backend/models/emotion_model.keras -
.envfile configured with Spotify credentials - Backend URL updated in
docker-compose.yml - Firewall rules configured
- Docker and Docker Compose installed
- Sufficient disk space (5GB+)
- Sufficient RAM (8GB+)
Use Let's Encrypt with Certbot:
# Install Certbot
sudo apt-get install certbot python3-certbot-nginx
# Update nginx.conf to include your domain
# Then get certificate
sudo certbot --nginx -d yourdomain.com
# Auto-renewal (optional)
sudo certbot renew --dry-run# View logs
docker-compose logs -f backend
docker-compose logs -f frontend
# Check container stats
docker stats
# Restart containers
docker-compose restart
# Update application
git pull
docker-compose up -d --buildβ Model not found at models/emotion_model.keras
Solution:
cd backend
source venv/bin/activate
python train_model.py# Check logs
docker-compose logs backend
docker-compose logs frontend
# Common issue: Port already in use
# Solution: Stop conflicting service or change port in docker-compose.ymlSolutions:
- Ensure good lighting
- Face camera directly
- Remove glasses/masks if possible
- Check camera permissions
- Verify image is not too dark/bright
Solution: Verify nginx.conf proxy settings and Flask-CORS configuration
# In app.py, ensure CORS is enabled
from flask_cors import CORS
app = Flask(__name__)
CORS(app)# Check credentials
cat .env
# Verify in container
docker-compose exec backend printenv | grep SPOTIFY
# Test connection
curl http://localhost/healthSolutions:
- Train for more epochs
- Add more training data
- Verify preprocessing matches training
- Check emotion label order
# In train_model.py, reduce batch size
BATCH_SIZE = 32 # or 16
# Or enable GPU memory growth
import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
tf.config.experimental.set_memory_growth(gpus[0], True)cd frontend
# Clear node modules and reinstall
rm -rf node_modules package-lock.json
npm install
# Build again
npm run build# Check Docker containers
docker-compose ps
docker-compose logs -f
# Enter backend container
docker-compose exec backend /bin/bash
# Enter frontend container
docker-compose exec frontend /bin/sh
# Check backend health
curl http://localhost/health
# Test API directly
curl -X POST http://localhost/api/detect-emotion \
-H "Content-Type: application/json" \
-d '{"image": "data:image/jpeg;base64,..."}'
# Rebuild specific service
docker-compose up -d --build backend
# Remove all containers and rebuild
docker-compose down
docker-compose up --build- Model Optimization:
# Use TensorFlow Lite for faster inference
# Add to app.py
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()- Caching:
# Cache face detection results
from functools import lru_cache
@lru_cache(maxsize=128)
def get_cached_predictions(image_hash):
return model.predict(image)- Code Splitting: Already implemented via React.lazy()
- Image Compression: Compress before sending to backend
- Lazy Loading: Load components on demand
Already configured in nginx.conf:
- Gzip compression
- Static asset caching
- Connection pooling
- Real-time video emotion tracking
- Multi-face detection and analysis
- Emotion history timeline
- User authentication and profiles
- Save favorite songs/playlists
- Custom playlist creation
- Mobile app (React Native)
- Voice emotion detection
- Emotion analytics dashboard
- Share emotions on social media
- Kubernetes deployment
- Redis caching
- PostgreSQL database
- WebSocket for real-time updates
- GraphQL API
- Microservices architecture
- Load balancing
- CI/CD pipeline (GitHub Actions)
- Automated testing
- Performance monitoring (Prometheus/Grafana)
- Transfer learning (VGG, ResNet)
- Ensemble models
- Larger training dataset
- Data augmentation techniques
- Multi-task learning (age, gender)
- Attention mechanisms
- Model quantization for edge deployment
Contributions are welcome! Please follow these guidelines:
- Fork the repository
- Clone your fork:
git clone https://github.com/yourusername/emotion-music-recommender.git - Create a branch:
git checkout -b feature/amazing-feature - Make your changes
- Test thoroughly
- Commit:
git commit -m 'Add amazing feature' - Push:
git push origin feature/amazing-feature - Open a Pull Request
Python (Backend):
- Follow PEP 8 style guide
- Use type hints where applicable
- Add docstrings to functions
- Write unit tests for new features
JavaScript (Frontend):
- Follow Airbnb style guide
- Use ESLint and Prettier
- Write component tests
- Use meaningful variable names
type(scope): subject
body
footer
Types: feat, fix, docs, style, refactor, test, chore
Example:
feat(backend): add emotion confidence threshold
Added configurable confidence threshold for emotion detection
to filter out low-confidence predictions.
Closes #42
## Description
Brief description of changes
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
## Testing
- [ ] Tests pass locally
- [ ] Added new tests
- [ ] Manual testing completed
## Screenshots (if applicable)
## Checklist
- [ ] Code follows style guidelines
- [ ] Self-review completed
- [ ] Commented complex