Question: No parallel path tags during evaluation despite using Parallel-R1 checkpoints

[larasneves]

Hi, thanks a lot for this work and for open-sourcing the code!!

I’ve been trying to reproduce the results for a course project this semester and ran into an issue during evaluation.

What I did:
•Ran submit_rl_exp_s1 to train with RL using the base model Parallel-R1/Parallel-SFT-Unseen from Hugging Face.
• Evaluated the resulting checkpoint using verl/recipe/r1/run_r1_distill_qwen.sh.
• I also tried evaluating the released checkpoint Parallel-R1/Parallel-R1-Unseen_Step_200.

Issue:
• I noticed that I had to manually modify the evaluation prompt to match the Parallel Thinking prompt described in the report.
• Even after doing so, the evaluation outputs do not contain any parallel path tags, which I find unexpected (using run_r1_distill_qwen.sh )

Questions:
1. Are parallel path tags supposed to appear automatically during evaluation, or is there an additional configuration or prompt formatting step required?
2. Is there a specific evaluation script or flag needed to enable parallel path generation?

Any guidance on how to properly reproduce the parallel reasoning behavior and evaluation would be greatly appreciated.
Thank you!

[zhengkid]

Thank you for your interest in our work.

For you Q1, parallel path tags supposed to appear automatically during evaluation when you use same prompt and same inference engine as in our paper.

For you Q2: Yes — reproducing the parallel reasoning behavior requires
(1) using the same inference engine as in our experiments, and
(2) strictly matching the inference prompt to the one used during training.

If either of these is changed (e.g., using vanilla evaluation script or a different prompt format), the model will typically fall back to standard single-path reasoning, even when using a Parallel-R1 checkpoint.

If you have tried to train the model yourself, you can track the model whether using parallel thinking through the metric num_turns/mean. Also you can track the generation logs, e.g., rollout-logs/validation-logs to check the results of each training step.

Moreover, if you have a specific checkpoint, and you want to eval offline.

You can try the following training commands as follows:

add     
trainer.val_before_train=True \
trainer.val_only=True \

and change the checkpoint you want to evaluate.

The example command is as follows:

#!/usr/bin/env bash
pkill python
sleep 5


set -x

DAPO_train_path=./verl/data_preprocess_scripts/data/dapo/adaptive_parallel_thinking_final_with_prompt_v3/rl_all_accuracy_reward/train.parquet
APO_combiene_test_path=./verl/data_preprocess_scripts/data/APO_combine/adaptive_parallel_thinking_final_with_prompt_v3/rl_all_accuracy_reward/test.parquet
train_files="['$DAPO_train_path']"
test_files="['$APO_combiene_test_path']"

project_name=Parallel-R1
experiment_name=Parallel-R1-Unseen-S1
default_local_dir=./$project_name/$experiment_name
validation_data_dir=$default_local_dir/val-log
base_model= # change to Your trained checkpoint
use_dynamic_bsz=False
offload=False
gradient_ckpt=True
infer_micro_batch_size=4

max_length=3000
max_prompt_len=2000
train_rollout_n=8

# sampling parameters
temperature=1.0
top_p=1.0
top_k=-1 # 0 for HF rollout, -1 for vLLM rollout
export VLLM_USE_V1=1
# 
python -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    algorithm.use_kl_in_reward=False \
    algorithm.gamma=1 \
    algorithm.lam=1 \
    data.train_files="$train_files" \
    data.val_files="$test_files" \
    data.train_batch_size=256 \
    data.return_raw_chat=True \
    data.max_prompt_length=${max_prompt_len} \
    data.max_response_length=${max_length} \
    data.filter_overlong_prompts=False \
    data.truncation='left' \
    actor_rollout_ref.actor.use_dynamic_bsz=${use_dynamic_bsz} \
    actor_rollout_ref.ref.log_prob_use_dynamic_bsz=${use_dynamic_bsz} \
    actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=${use_dynamic_bsz} \
    actor_rollout_ref.actor.ppo_max_token_len_per_gpu=$((max_length + max_prompt_len)) \
    actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=$((max_length + max_prompt_len)) \
    actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=$((max_length + max_prompt_len)) \
    actor_rollout_ref.model.path="${base_model}" \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=False \
    actor_rollout_ref.actor.ppo_mini_batch_size=128 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=2 \
    actor_rollout_ref.actor.fsdp_config.param_offload=${offload} \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=${offload} \
    actor_rollout_ref.actor.use_kl_loss=False \
    actor_rollout_ref.model.enable_gradient_checkpointing=${gradient_ckpt} \
    actor_rollout_ref.rollout.n=${train_rollout_n} \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=${infer_micro_batch_size} \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    actor_rollout_ref.rollout.agent.max_path_response_length=4096 \
    actor_rollout_ref.rollout.agent.num_workers=32 \
    actor_rollout_ref.rollout.agent.max_iterations_for_parallel_thinking=4 \
    actor_rollout_ref.rollout.agent.num_paths=2 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.mode=async \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.7 \
    actor_rollout_ref.rollout.top_p=${top_p} \
    actor_rollout_ref.rollout.top_k="${top_k}" \
    actor_rollout_ref.rollout.val_kwargs.temperature=${temperature} \
    actor_rollout_ref.rollout.val_kwargs.top_p=${top_p} \
    actor_rollout_ref.rollout.val_kwargs.top_k=${top_k} \
    actor_rollout_ref.rollout.val_kwargs.do_sample=True \
    actor_rollout_ref.rollout.val_kwargs.n=1 \
    trainer.logger=['console','wandb'] \
    trainer.project_name="${project_name}" \
    trainer.experiment_name="${experiment_name}" \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=1 \
    trainer.save_freq=10 \
    trainer.test_freq=2 \
    trainer.total_epochs=100 \
    trainer.validation_data_dir="$default_local_dir/validation-log" \
    trainer.rollout_data_dir="$default_local_dir/rollout-log" \
    trainer.default_local_dir="$default_local_dir" \
    trainer.val_before_train=True \
    trainer.val_only=True \
    trainer.resume_mode=auto $@

[larasneves]

Thank you so much!

Could you please clarify why the 200-step checkpoint was shared instead of the final checkpoint? Unfortunately, we don’t have enough resources to reproduce the full training, so this makes direct comparison a bit difficult.

Also, could you clarify for which types of questions we should expect more than one reasoning path? In our evaluations so far, we usually observe only a single path, and in some cases, it does not fully follow the prompt instructions for at least two paths. We’re still working on this and trying to better understand the behavior.

Thank you very much for your help!