Enhancement: Multi-step AI material generation with image creation

[fernandotonon]

Summary

Implement a multi-step AI workflow that can generate both material scripts AND the required textures using image generation models.

Current State

• LLM generates material scripts that reference textures
• User must have textures already available
• If LLM references non-existent textures, material won't work

Proposed Enhancement

Phase 1: RAG with Project Assets

• Index project textures using a vision model (LLaVA, Moondream, etc.)
• Classify textures: diffuse, normal, specular, roughness, heightmap, etc.
• Describe texture content: "green grass", "rusty metal", etc.
• Store embeddings for semantic search
• When generating materials, retrieve relevant existing textures

Phase 2: Image Generation Integration

• Integrate stable-diffusion.cpp (supports Metal on macOS)
• Multi-step workflow:
  1. User describes desired material
  2. LLM plans what textures are needed
  3. SD generates missing textures (diffuse, normal, etc.)
  4. LLM generates material script using created textures

Technical Considerations

• stable-diffusion.cpp supports Metal acceleration (same as llama.cpp)
• Memory management: may need to swap models (LLM ↔ SD)
• Texture types to generate: diffuse, normal maps, roughness, emission
• Consider ControlNet for generating normal maps from diffuse

Architecture

User Prompt: "Create a mossy stone material"
        ↓
┌─────────────────────────────────────────┐
│ Step 1: LLM Plans                       │
│ "Need: stone_diffuse, stone_normal,     │
│        moss_diffuse, moss_mask"         │
└─────────────────────────────────────────┘
        ↓
┌─────────────────────────────────────────┐
│ Step 2: Check existing assets (RAG)     │
│ Found: stone_normal.png                 │
│ Missing: stone_diffuse, moss_*          │
└─────────────────────────────────────────┘
        ↓
┌─────────────────────────────────────────┐
│ Step 3: Generate missing with SD        │
│ → stone_diffuse.png                     │
│ → moss_diffuse.png                      │
│ → moss_mask.png                         │
└─────────────────────────────────────────┘
        ↓
┌─────────────────────────────────────────┐
│ Step 4: LLM generates material script   │
│ Using all available textures            │
└─────────────────────────────────────────┘

References

• stable-diffusion.cpp
• llama.cpp vision models

Labels

enhancement, AI, future