Performance Drop on MGSM After Finetuning - QWEN2.5 - 3B

[baselmousi]

Hi,

I was trying to replicate the finetuning of the QWEN2.5-3B-Instruct model . I tried to evaluate on the mgsm dataset after training but I obtained a performance difference of 14 points between the released model and the model that I trained.

I used the same setup as the one shared in the sft.sh script.

I was wondering if you used a different hyperparameters setup for training the smaller models released, and if you did I'd appreciate if you can share it with me.

Thanks

[Muennighoff]

Our run is here: https://wandb.ai/hashimoto-group/o1/runs/stcs4jwi/overview

[baselmousi]

Thank you for sharing the details. I reproduced the training exactly as outlined in your run, but the resulting model shows a discrepancy from the released version on the MGSM evaluation. In particular:

Model	BN	DE	EN	ES	FR	JA	RU	SW	TE	TH	ZH	Average
Qwen2.5-1.5B-s1-trained	0.25	0.52	0.66	0.53	0.54	0.20	0.56	0.04	0.08	0.42	0.53	0.39
Qwen2.5-1.5B-s1-released	0.22	0.54	0.63	0.66	0.60	0.43	0.54	0.03	0.06	0.48	0.61	0.44
QWEN2.5-3B-s1-trained	0.26	0.54	0.81	0.66	0.62	0.42	0.54	0.06	0.08	0.47	0.67	0.47
Qwen2.5-3B-s1-released	0.57	0.63	0.82	0.74	0.72	0.62	0.71	0.09	0.13	0.69	0.69	0.58

Could you please suggest ways to address this discrepancy ?

[Muennighoff]

Hmm that is odd - what are the thinking trace lengths of trained vs released?
Did you use simplescaling/s1K-1.1_tokenized ?

[baselmousi]

Hi,

Thanks a lot for your reply.

Yes I used simplescaling/s1K-1.1_tokenized.

Apparently the trained model has shorter traces than the released one. This table shows a comparison of the lengths of the traces

Task	QWEN2.5-3B-Trained	QWEN2.5-3B-Released	QWEN2.5-1.5B-Trained	QWEN2.5-1.5B-Released
mgsm_direct_en	861.036	1888.44	1567.612	4204.392
mgsm_direct_th	986.684	2428.3	1800.42	4506.22
mgsm_direct_ru	1194.496	1923.872	2528.196	4576.392
mgsm_direct_te	2283.16	3194.92	4843.592	6982.868
mgsm_direct_bn	4432.772	2924.24	3226.664	5579.932
mgsm_direct_es	934.864	2026.276	2858.548	4359.916
mgsm_direct_ja	752.596	2556.376	1572.26	5053.348
mgsm_direct_sw	2025.408	6094.496	2546.068	5395.62
mgsm_direct_fr	1024.876	2142.156	2650.404	5009.552
mgsm_direct_de	1006.328	1676.412	1715.252	3901.456
mgsm_direct_zh	359.012	1986.496	1909.916	4455.972

I am running these tasks using lm-harness using a temperature of 0.

[Muennighoff]

Interesting did you use the same block size (i.e. seq len) as the released model?

[baselmousi]

yes I used the same one (23000) as shared in the wandb logs.
The only difference with the run shared is the use of gradient accumulation steps of 16 and a batch size of 1 (which is equivalent to using a batch size of 16 since you ran on 16 gpus with a micro batch size of 1 but I am only running on 1 gpu)

[Muennighoff]

Hm there might be some randomness - I guess you could try rerunning and see how much scores differ.