Seeking training recipe/advice for SDPO on Math tasks

[AzureStarz]

Could you please share a training recipe for applying SDPO to math tasks?

Currently, I am training Qwen2.5-3B-Instruct on a Math training split using your SDPO implementation, but the val-core metric keeps degrading throughout the training progress. I have already tried swapping in other models and different datasets, but the training still isn't working as expected.

I would love to know if you have any empirical experience, recommended hyperparameters, or general advice for adapting your method successfully to math-heavy tasks. Thanks!

[JianghaoWu]

I’m facing the same issue and would also appreciate any guidance or practical suggestions.

[euclidgame]

I think this is somewhat expected... I ran a lot of math training with SDPO as well, so the training rewards went up because the model started memorizing the correct demonstrations for the training problems. The model can't learn to reason through SDPO, and thus the improvement can't transfer to the validation set.

[joejajo]

Hi ,
Im trying one shot rlvr paper on SDPO , is this a good idea ?