File tokenizer.json is missing for GD-ML/Open-RS1 model

[GangLii]

Thank you for sharing the great work. On Huggingface, I noticed that the tokenizer.json is missing for GD-ML/Open-RS1 model so that I can't load the model directly with Huggingface API. Could you help upload tokenizer.json to the repo?

[xiaozhang23957]

@GangLii Thank you for your interest and support in our work. We update the tokenizer.json on Hugging Face: https://huggingface.co/GD-ML/Open-RS1.

[GangLii]

@undyingjoker Thank you for the updates. I'm trying to reproduce the numbers reported in the paper for the GPG-RS1 1.5B model. I used the commands below, provided in the repo. However, I only get 0.3 with one GPU and 0.27 with 4 GPUs for evaluation. Both of them are much lower than 0.33 reported in the paper. Also, I notice that lighteval computes pass@1 with only one response for each question, rather than averaging over k responses as did in the deepseekR1 paper, which might lead to very large evaluation variance. Could you help confirm if my evaluation method is correct? How did you evaluate the model?

MODEL=GD-ML/Open-RS1
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

# Example: AIME 2024
TASK=aime24
lighteval vllm "$MODEL_ARGS" "custom|$TASK|0|0" \
  --custom-tasks src/open_r1/evaluate.py \
  --use-chat-template \
  --output-dir "$OUTPUT_DIR"