GRPO multi-GPU reproduction issue

[DT6A]

Hello! Thanks a lot for your awesome work.

I'm trying to reproduce the results GRPO results using multi-GPU setup. I did not change any of your code and just ran the training using provided notebook. However, the model doesn't seem to learn anything -- there is no any reward improvement from near-zero level.

Here is the wandb run: https://wandb.ai/tarasovd/GRPO-Qwen-1.5-Instruct-Multi-GPU/runs/ukdas6bn

Do you have any ideas why this might happen? Were there any changes in the code after the launch you reported (sorry, it's hard to track changes for notebooks with git diff)?

My best guess is that libraries versions might be the issue and I couldn't find those in your repository. Could you please share those or say where they can be found?

[DT6A]

After updating the libraries which are installed in notebook to latest versions it started to work. I suggest authors to fix the versions of the libraries as it heavily affects the performance.

Here is the run with updated versions:
https://wandb.ai/tarasovd/GRPO-Qwen-1.5-Instruct-Multi-GPU/runs/syqn4qpb
The versions were not changed significantly from what I used (i.e. I don't think there were major updates between those I used and latest) but made a huge difference.

[aburkov]

Hi! Thanks for letting me know. Which versions did you use? The notebook was configured to run on Lambda's stack. The versions of installed packages are shown in the console output log after the respective cell. Did you use these?

[DT6A]

The versions that worked (I had to change numpy, sklearn and scipy versions too as there were some conflicts):

transformers-4.49.0
numpy-2.2.3
scikit_learn-1.6.1
scipy-1.15.2
accelerate-1.4.0
torch-2.5.1+cu124
flash_attn-2.7.4.post1

You are right that versions you used are available in output log, I forgot about those:)

[wenzhifang]

My results appear to be more fluctuating and sometimes drop finally

[Titulus233]

thx! I can not reproduce too until i came across this issue and fixed it after I updated my transformers XD