Crucible: Quantifying the Potential of Control Algorithms through LLM Agents

This is the official code repository for the paper "Crucible: Quantifying the Potential of Control Algorithms through LLM Agents" presented at NeurIPS 2025 (Paper Link).

Overview

Crucible is a novel framework that quantifies the Tuning Potential of control algorithms using LLM-driven expert simulation. Unlike traditional approaches that focus solely on default performance, Crucible systematically evaluates how much improvement can be achieved through parameter tuning and logic optimization across different algorithms and domains.

Requirements

• Python 3.7+
• Dependencies listed in requirements.txt

Installation

git clone https://github.com/thu-media/Crucible
cd crucible
pip install -r requirements.txt

Quick Start

1. Set API Key in the target module's server.py:

   api_key = "your_key"

2. Run Analysis for any domain:

   cd ABR/  # or CartPole/ or Scheduling/
   python run_test.py      # Generate baseline logs

3. Set the RL log directory in server.py:

   rl_log_dir = 'rl_path'  # Replace rl_path with the actual log path

4. Run Crucible analysis:

   python run_server.py    # Run Crucible analysis

Project Structure

├── ABR/              # Adaptive Bitrate algorithms
│   │                 # Includes BBA, MPC, HYB, Pitree, Pensieve
│   │                 # and our modified BBA_C from Section 4.4.1
├── CartPole/         # Control algorithms (Bang-Bang, PID, LQR, DQN)
├── Scheduling/       # Scheduling algorithms (coming soon)
└── Log/              # Example potential calculations and Table 4 data

Note: The ABR module includes our modified BBA_C algorithm, which demonstrates the practical application of Crucible's tuning methodology as described in Section 4.4.1 of the paper.

Reproducing Results

The Log/ directory contains calculation scripts and example data for reproducing the potential quantification results reported in Table 4 of our paper.