LiteCoT This project is the official open-source implementation of the LiteCoT paper. Currently, we have fully open-sourced both the training data and training scripts.
🔥 [2025/05/31] (update #2): We release all the evaluation scripts and part of recipes for different models.
🔥 [2025/05/20] (update #1): We release the traing datasets and all the corresponding Liter Models.
We use DAP pipeline to generate LiteCoT datasets and trained Liter based on this LiteCoT. Overview of the DAP pipeline. First, the DeepSeek R1 model generates initial Long CoT data from collected user queries. Then, a difficulty-aware prompt guides the model to rewrite concise CoT outputs, adapting reasoning length dynamically based on problem difficulty
- Open source all the evaluation scripts
- Open source Liter series model weights
- Open source training scripts
- Open source training data
1. git clone https://github.com/Evanwu1125/LiteCoT.git
2. cd LiteCoT
3. conda create --name <env_name> --file requirements.txtYou can find the corresponding dataset here
If you have problems directly having access to huggingface, you can use the following command.
pip install -U huggingface_hub # download huggingface_hub package
export HF_ENDPOINT=https://hf-mirror.com # using huggingface mirror
huggingface-cli download --repo-type dataset --resume-download SmallDoge/SmallThoughts --local-dir /path/to/your/dir # terminal download command
Using the training_data/convert_parquet.py to preprocess the downloaded datasets.
Here we provide different recipes for training different sizes of models from 1.5B to 32B, all the models are STF based on the Qwen2.5 models.
If you have problems directly having access to huggingface, you can use the following command.
export HF_ENDPOINT=https://hf-mirror.com # using huggingface mirror
huggingface-cli download --resume-download Qwen/Qwen2.5-Math-1.5B --local-dir model_results/qwen2.5-math-1.5b # terminal download command
#We use qwen2.5-math-7b here as an example.
run scripts/run_7b-math-training-short.shYou can see all the checkpoints preserved in model_results.
In case you want you directly use our trained models, we provide all the weights here.
| Our Model Name | Original Model Weights |
|---|---|
| LiteCoT-1.5B | Qwen2.5-Math-1.5B-Base |
| LiteCoT-7B | Qwen2.5-Math-7B-Base |
| LiteCoT-14B | Qwen2.5-14B-Base |
| LiteCoT-32B | Qwen2.5-32B-Base |
The results are shown below:
Our evaluation scripts are from Open-R1 and Qwen-Math projects.
Given the stochastic nature of the model outputs, it is normal for the reproduced evaluation results to differ slightly from those reported in our paper.
# Remember to install lighteval before evaluating
cd LiteCoT/eval/open-r1
pip install lighteval
sh eval.sh# Remember to install corresponding requirements
cd LiteCoT/eval/Qwen2.5-Math
sh eval.sh@article{wu2025concise,
title={Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting},
author={Wu, Yifan and Shi, Jingze and Wu, Bingheng and Zhang, Jiayi and Lin, Xiaotian and Tang, Nan and Luo, Yuyu},
journal={arXiv preprint arXiv:2505.19716},
year={2025}
}