About Visual-RFT GPG implement

[yyl494]

Hello, thank you for your great work.

Why I can't see the implementation of alpha in the visual-rft code? Looking at this line of code, does the GPG version of Visual-RFT just remove the KL divergence?
https://github.com/AMAP-ML/GPG/blob/main/Visual-RFT/src/virft/src/open_r1/trainer/grpo_trainer.py#L493

[lerogo]

Hi,
thank you for your attention to our work. We just casually verified the effectiveness of GPG in multimodal tasks and found it to be good. The code you mentioned is the first version of our GPG, and the latest version of GPG implemented on Visual RFT has been committed to the repository. Other task can refer to the implementation. The results we obtained using the latest GPG method are (slightly improved compared to the first version):

re-run log & report:
https://wandb.lerogo.com/lerogo/visual-rft/reports/Visual-RFT-re-run-gpg-f_norm-alpha--VmlldzozOA?accessToken=918juirquast2ye4hecemzs3yukh0gyh4ulhwh2daoqms0vyr4xzb6wdxr2gzi90

re-run results:

Table: Reasoning grounding results on LISA. GPG surpasses GRPO in reasoning grounding with 239 training images.

Models	mIoUtest	mIoUval	gIoUtest
Qwen2-VL-2B	26.9	30.1	25.3
+ SFT	28.3	29.7	25.3
+ GRPO	37.6	34.4	34.4
+ GPG	51.8	51.3	50.4

Table: 4-shot Results on Four Fine-grained Classification Datasets. GPG shows consistently better results than GRPO on $4$ classification datasets.

Models	Average	Flower102	Pets37	FGVC	Cars196
Qwen2-VL-2B	56.0	54.8	66.4	45.9	56.8
+ SFT	55.6	58.5	55.5	67.9	40.5
+ GRPO	81.9	71.4	86.1	74.8	95.3
+ GPG	86.0	76.0	87.7	83.3	97.0

Thank you again for your attention and support to our work.

[lerogo]

Hi, thank you for your attention to our work. We just casually verified the effectiveness of GPG in multimodal tasks and found it to be good. The code you mentioned is the first version of our GPG, and the latest version of GPG implemented on Visual RFT has been committed to the repository. Other task can refer to the implementation. The results we obtained using the latest GPG method are (slightly improved compared to the first version):

re-run log & report: https://wandb.lerogo.com/lerogo/visual-rft/reports/Visual-RFT-re-run-gpg-f_norm-alpha--VmlldzozOA?accessToken=918juirquast2ye4hecemzs3yukh0gyh4ulhwh2daoqms0vyr4xzb6wdxr2gzi90

re-run results:

Table: Reasoning grounding results on LISA. GPG surpasses GRPO in reasoning grounding with 239 training images.

Models mIoUtest mIoUval gIoUtest
Qwen2-VL-2B 26.9 30.1 25.3

• SFT 28.3 29.7 25.3
• GRPO 37.6 34.4 34.4
• GPG 51.8 51.3 50.4
  Table: 4-shot Results on Four Fine-grained Classification Datasets. GPG shows consistently better results than GRPO on 4 classification datasets.

Models Average Flower102 Pets37 FGVC Cars196
Qwen2-VL-2B 56.0 54.8 66.4 45.9 56.8

• SFT 55.6 58.5 55.5 67.9 40.5
• GRPO 81.9 71.4 86.1 74.8 95.3
• GPG 86.0 76.0 87.7 83.3 97.0
  Thank you again for your attention and support to our work.

There is a slight problem with the easy problem in printing logs in this part, which should be determined based on the upper limit of the reward function, that is, modify easy_mask = gathered_mean_grouped_rewards == MaxReward.

However, there is no problem with the calculation of $\alpha$ in the paper: identical_value_mask = gathered_std_grouped_rewards == 0

[lerogo]

We have updated the multimodal version code and results of GPG, welcome to try.
commit: 12843f8