Transform fluid dynamics and more into stunning generative art with real-time AI processing
Diffusion Lab is an advanced WebGL fluid simulation combined with real-time AI stream diffusion, designed for interactive art creation, live performances, and creative exploration. Control it through touch, remote devices, or Telegram commands.
- 🤖 AI Stream Diffusion: Real-time AI processing with ControlNet support (Pose, HED, Canny, Depth, Color)
- 🌊 Advanced Fluid Simulation: WebGL-powered fluid dynamics with customizable physics
- 📱 Telegram Integration: Remote prompt control and preset management via Telegram bot
- 🎛️ OSC Control: Professional TouchOSC integration for live performance
- 📷 Camera Integration: Live camera feed as fluid background or AI input
- 🖼️ Media Backgrounds: Support for images and videos as fluid backgrounds
- 📱 Tablet Optimized: Touch-friendly interface designed for iPad and tablets
- 🎵 Audio Reactive: Audio input visualization with customizable parameters
- 🎨 ControlNet Presets: Pre-configured AI processing setups for different artistic styles
npm install./start-diffusion-lab.shThe startup script automatically:
- Starts the web server (port 3000)
- Launches the OSC server (port 8000)
- Initializes WebSocket communication (port 8001)
- Activates the Telegram bot (if configured)
- Opens the application in your browser
- Closes any conflicting browser tabs
- Open the application in your browser
- Click the API configuration button
- Enter your AI service API key
- Select your preferred AI model
- Create a bot via @BotFather on Telegram
- In the web interface:
- Enable "Receive from Telegram" toggle
- Open "Telegram Bot Configuration" section
- Enter your bot token in the "Bot Token" field
- Check "Remember token" to save it locally
- Use the QR code or bot link to access @DiffusionPromptBot
- Real-time Processing: Live AI generation with minimal latency
- ControlNet Support: Multiple ControlNet models for different artistic effects
- Prompt Management: Dynamic prompt switching and interpolation
- Parameter Control: Fine-tune inference steps, guidance scale, and more
- Advanced Physics: Density, velocity, pressure, and vorticity controls
- Visual Effects: Bloom, sunrays, 3D shading, and transparency
- Interactive Drawing: Multiple input modes including velocity-based drawing
- Animation System: Automated movement with chaos, breathing, and color evolution
- Touch/Mouse: Direct interaction with the fluid simulation
- Camera Feed: Use live camera as background or AI input source
- Media Files: Load images/videos as backgrounds or overlays
- Audio Input: Microphone-driven audio visualization
- OSC Control: Professional remote control via TouchOSC or similar
- Telegram Commands: Text-based control and preset management
- Background Modes: Fluid-only, camera, media, or combined backgrounds
- Scaling Controls: Independent scaling for different media layers
- Real-time Switching: Seamless transitions between input modes
- Text Messages: Send any text to use as an AI prompt
/start: Welcome message and setup instructions/preset: Browse and apply prompt or ControlNet presets
- 🎨 Prompt Presets: Ready-made visual themes and artistic styles
- ⚙️ ControlNet Presets: AI processing parameter configurations
- Queue Management: Smart waitlist system with configurable intervals
- Real-time Feedback: Queue position and processing time estimates
- Public Broadcasting: Share prompts to community Telegram groups
- Smart Processing: Intelligent queue management with priority handling
- Multi-user Support: Handle multiple users simultaneously
- Install TouchOSC on your phone/tablet
- Configure connection:
- Host: Your computer's IP address (shown at startup)
- Send Port: 8000
- Protocol: OSC
/toggle/paused # Pause/resume simulation
/toggle/colorful # Enable colorful mode
/toggle/animate # Enable animation
/toggle/bloom # Bloom effect
/toggle/sunrays # Sunrays effect
/toggle/shading # 3D shading
/toggle/transparent # Transparency
/toggle/velocity_drawing # Velocity-based drawing
/toggle/fluid_drawing # Fluid drawing mode
/toggle/audio # Audio input
/toggle/camera # Camera input
/toggle/media # Media input
/fluid/density (0.0-1.0) # Density dissipation
/fluid/velocity (0.0-1.0) # Velocity dissipation
/fluid/pressure (0.0-1.0) # Pressure
/fluid/vorticity (0.0-1.0) # Vorticity/curl
/fluid/splat (0.0-1.0) # Splat radius
/ai/inference_steps (1-100) # AI inference steps
/ai/seed (0-1000) # Random seed
/ai/guidance_scale (1-20) # Guidance scale
/ai/delta (0.0-1.0) # Delta parameter
/controlnet/pose (0.0-1.0) # Pose detection strength
/controlnet/hed (0.0-1.0) # HED edge detection
/controlnet/canny (0.0-1.0) # Canny edge detection
/controlnet/depth (0.0-1.0) # Depth estimation
/controlnet/color (0.0-1.0) # Color preservation
/multixy/1 # Channel 1 XY control
/multixy/2 # Channel 2 XY control
/multixy/3 # Channel 3 XY control
/multixy/4 # Channel 4 XY control
/multixy/5 # Channel 5 XY control
/telegram/receive (0/1) # Enable/disable prompt reception
/telegram/clear (1) # Clear waitlist
/telegram/interval (1-30) # Processing interval (seconds)
/action/reset # Reset all values
/action/screenshot # Capture screenshot
/action/record # Toggle video recording
- Toggle buttons: paused, colorful, animate, bloom, sunrays
- RGB sliders for brush and background colors
- Physics controls: density, velocity, pressure
- Inference steps, seed, guidance scale sliders
- ControlNet strength controls (pose, HED, canny, depth, color)
- Delta and animation parameters
- 5 XY pads for
/multixy/1through/multixy/5 - Real-time velocity-based drawing
- Independent channel control
- Input mode toggles (drawing, audio, camera, media)
- Media scaling controls
- Animation and audio parameters
- Telegram receive toggle
- Waitlist interval slider
- Clear waitlist button
- Status indicators
- Application won't start: Check that ports 3000, 8000, and 8001 are available
- Browser conflicts: The startup script automatically closes conflicting tabs
- Performance issues: Try reducing AI inference steps or disabling some visual effects
- "Disconnected" status: Verify the OSC server is running and you're on the same WiFi network
- TouchOSC not responding: Check IP address and port 8000 in TouchOSC settings
- Message lag: Enable "High Frequency" mode in TouchOSC
- Bot not responding: Verify the bot token is correctly configured in
local-osc-server.js - Commands not working: Ensure the bot has proper permissions and polling is active
- Queue not processing: Check the waitlist interval setting and server logs
- Camera access denied: Grant camera permissions in browser settings
- AI generation slow: Reduce inference steps or try a different model
- Media files not loading: Check file formats and browser compatibility
- Ports: 3000 (web), 8000 (OSC), 8001 (WebSocket)
- WiFi: All devices must be on the same network
- Firewall: Ensure ports are not blocked
- Host Computer: Run
./start-diffusion-lab.sh - iPad/Tablet: Open browser to
http://HOST_IP:3000 - Phone (TouchOSC): Connect to
HOST_IP:8000 - Telegram: Send messages to your configured bot
- Use lower inference steps (20-30) for real-time performance
- Disable unnecessary visual effects on slower devices
- Consider reducing ControlNet scales for better performance
- Start with ControlNet presets to establish artistic style
- Use Telegram for collaborative prompt creation
- Combine multiple input modes for complex compositions
- Record sessions for later playback and editing
- Set up TouchOSC layouts in advance
- Test OSC connections before performances
- Use the multi-touch XY pads for expressive control
- Configure Telegram for audience interaction
Edit the preset definitions in local-osc-server.js to create custom AI processing configurations.
The system supports multiple AI service providers. Configure your preferred service in the web interface.
Set DEBUG=true in start-diffusion-lab.sh for verbose logging and development features.
Need help? Check the console output for detailed debugging information, or refer to the troubleshooting sections above.