🧠 LLM Reflection Lab

An interactive research tool for exploring recursive reasoning in Large Language Models through iterative self-reflection. Watch as LLMs refine their thinking across multiple iterations without external feedback.

🎯 Overview

This project implements a thinking loop system where LLMs iteratively improve their responses through self-reflection. Unlike traditional single-shot prompting, this approach allows models to:

• 🔄 Reflect on their previous reasoning
• 🎯 Identify gaps and assumptions
• 📈 Refine their answers progressively
• 🧩 Explore different reasoning paths

## ✨ Features

Core Functionality

• Multi-Model Support: Works with Ollama, vLLM, and OpenRouter APIs
• Reasoning Extraction: Captures explicit reasoning from <think> tags or native fields
• 🎯 YOLO Mode: Run iterations until convergence is detected automatically
  • Configurable convergence threshold (80-99%)
  • Choose similarity comparison mode: "Response Only" (default) or "Reasoning + Response"
• 📚 Prompt Templates: Pre-configured epistemic approaches
  • Socratic Method, Empirical-Scientific, Dialectical Synthesis, Systems Thinking, and more
  • Easy template switching via dropdown in prompt editor
• Customizable Prompts: Edit system prompts and reflection templates via UI
• Auto-Save: Experiments saved automatically in JSON format
• 📄 Export Options:
  • PDF Reports: Professional reports with visualizations, charts, and complete appendix
  • HTML Reports: Interactive web-based reports
  • Smart Filenames: AI-generated descriptive filenames based on question content

📊 Interactive Visualizations

• 🕸️ Concept Evolution Graph: Network showing how concepts emerge and connect
• 🔥 Similarity Heatmap: Matrix of iteration similarities to identify convergence
• 📈 Confidence Tracking: Evolution of certainty/uncertainty markers
• 📊 Complexity Metrics: Vocabulary diversity and logical connector usage
• 🌊 Topic Flow Sankey: How topics persist or change between iterations
• ↔️ Convergence Timeline: Tracks exploration vs exploitation phases

🚀 Quick Start

Prerequisites

• Python 3.10+
• One of: Ollama, vLLM server, or OpenRouter API key

Installation

Using uv (Recommended)

# Clone the repository
git clone https://github.com/chheplo/llm-reflection-lab.git
cd llm-reflection-lab

# Install with uv
uv sync

Using pip

# Clone the repository
git clone https://github.com/chheplo/llm-reflection-lab.git
cd llm-reflection-lab

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Running the Application

# With uv
uv run streamlit run app.py

# With pip
streamlit run app.py

Alternative: Run with minimal UI and headless mode

# With uv (minimal toolbar, headless server, no usage stats)
uv run streamlit run app.py --client.toolbarMode=minimal --server.headless true --browser.gatherUsageStats false

# With pip
streamlit run app.py --client.toolbarMode=minimal --server.headless true --browser.gatherUsageStats false

The app will open at http://localhost:8501

🎮 Usage

1. Configure Your Model

For Ollama (Local)

• Install Ollama
• Pull a model: ollama pull gpt-oss:20b
• Start Ollama: ollama serve
• Select "Ollama (Local)" in the app

For vLLM

• Start your vLLM server
• Enter the server URL and API key
• Click "Load Available Models"

For OpenRouter

• Get an API key from OpenRouter
• Select "OpenRouter" and enter your key
• Choose from available models

2. Run an Experiment

1. Enter a Question: Complex questions work best
2. Set Iterations: Choose 3-10 iterations (or more!)
3. Optional - Enable YOLO Mode:
   • Toggle "🎯 YOLO Mode" to run until convergence
   • Adjust convergence threshold (0.80-0.99)
   • Iterations continue until consecutive responses are similar enough
4. Start Loop: Click to begin the thinking process
5. Watch Evolution: See reasoning improve in real-time
6. Explore Visualizations: Click visualization buttons for insights

3. Customize Prompts

Click "✏️ Prompts" to:

• Load Templates: Choose from epistemic approaches
  • Default: Standard iterative reasoning
  • Socratic Method: Question-driven inquiry
  • Empirical-Scientific: Evidence-based analysis
  • Dialectical Synthesis: Thesis-antithesis resolution
  • Systems Thinking: Holistic interconnected analysis
  • Iterative Refinement: Precision-focused improvement
• Edit Prompts: Customize system and reflection prompts
• Save Changes: Store your customizations

4. Export Results

Click "📤 Export" to generate reports:

• PDF Report: Professional document with:
  • Colorful title page with research question
  • Executive summary and key findings
  • Visualization charts (token usage, convergence analysis)
  • Detailed experiment results
  • Complete appendix with all iterations
  • Smart AI-generated filename based on question
• HTML Report: Web-based interactive report

📁 Project Structure

llm-reflection-lab/
├── app.py                 # Main Streamlit application
├── src/
│   ├── visualizations.py  # Visualization modules
│   ├── pdf_export.py      # PDF report generation
│   └── prompts.json       # Current active prompts (user customized)
├── templates/            # Prompt template library
│   ├── default.json      # Standard reasoning template
│   ├── socratic-method.json
│   ├── empirical-scientific.json
│   ├── dialectical-synthesis.json
│   ├── systems-thinking.json
│   └── iterative-refinement.json
├── saves/                # Auto-saved experiments
├── pyproject.toml        # Project dependencies (uv)
├── requirements.txt      # Project dependencies (pip)
└── README.md            # This file

🔬 How It Works

The Thinking Loop Process

1. Initial Response: Model answers the question
2. Self-Reflection: Model reviews its previous answer
3. Improvement: Model provides refined response
4. Repeat: Process continues for N iterations (or until convergence in YOLO Mode)

Reasoning Extraction

The system extracts reasoning through:

• Native reasoning fields (e.g., OpenAI o1 models)
• <think>...</think> tags in responses
• Configurable extraction patterns

Convergence Patterns

Through visualizations, you can observe:

• Convergence: Ideas stabilizing (high similarity)
• Divergence: Exploring new concepts (low similarity)
• Phase Transitions: Shifts between exploration/exploitation

YOLO Mode (You Only Loop Once... Until Convergence)

When enabled, YOLO Mode:

• Automatic Stopping: Detects when consecutive iterations reach similarity threshold
• Dynamic Duration: Runs as many iterations as needed (up to 100 for safety)
• Real-time Monitoring: Shows convergence progress chart during execution
• Efficiency: Stops early when the model's responses stabilize
• Configurable Threshold: Adjust sensitivity from 80% to 99% similarity
• Comparison Modes:
  • "Reasoning + Response": Compare full thought process
  • "Response Only": Focus on answer convergence

📊 Example Insights

From a typical 10-iteration experiment:

• Iterations 1-2: Initial exploration
• Iterations 3-6: First convergence cluster
• Iteration 7: Divergence/pivot point
• Iterations 8-10: Final convergence

🛠️ Configuration

Environment Variables

• OPENROUTER_API_KEY: Your OpenRouter API key
• VLLM_API_KEY: Your vLLM server API key

Prompt Customization

Edit prompts.json or use the UI to modify:

• System prompts
• Reflection templates
• Reasoning extraction patterns

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Development Setup

# Clone your fork
git clone https://github.com/YOUR_USERNAME/llm-reflection-lab.git
cd thinking-loop-experiment

# Install in development mode
uv sync --dev

# Create a feature branch
git checkout -b feature/your-feature

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with Streamlit
• Visualizations powered by Plotly and PyVis
• LLM integration via OpenAI Python SDK

📚 Citation

If you use this tool in your research, please cite:

@software{llm_reflection_lab,
  title = {LLM Reflection Lab},
  author = {Your Name},
  year = {2024},
  url = {https://github.com/chheplo/llm-reflection-lab}
}

🔗 Links

• Report Issues
• Documentation
• Discussions

---

Made with ❤️ for the AI research community