Inconsistent AIME scores when porting evaluation to newer vLLM versions

[asparius]

Description

Hi Niklas(@Muennighoff),
I'm trying to reproduce s1/s1.1 evaluation results using newer vLLM versions so I can evaluate newer instruct and base models that aren't supported on the original repo's vLLM (~0.6.x). To do this, I modified causal_vllms and the corresponding utils files in the tasks directory.

Two problems:

1. AIME scores deviate from published values.
2. Budget forcing appears to have no effect in most configurations.

I can reproduce the exact published scores using the original unmodified repository, so this is specific to my vLLM migration.

Published reference scores from the paper

Model	MATH500	GPQA	AIME24	AIME25
s1 w/o BF	92.6	56.6	50.0	26.7
s1 BF "Wait" 1x	92.8	59.6	53.3	30.0
s1 BF "Wait" 2x	93.0	59.6	53.3	33.3
s1.1 w/o BF	94.4	60.6	56.7	50.0
s1.1 BF "Wait" 1x	95.4	62.6	56.7	50.0
s1.1 BF "Wait" 2x	95.4	63.6	56.7	50.0

My s1.1 results (modified vLLM)

Config	AIME24	AIME25	GPQA	MATH
Paper w/o BF	56.7	50.0	60.6	94.4
Paper BF 1x	56.7	50.0	62.6	95.4
Paper BF 2x	56.7	50.0	63.6	95.4
My Auto (no BF)	70.0	43.3	63.1	94.2
My Ignore 1 wait	70.0	43.3	63.1	94.2
My Ignore 2 wait	53.3	43.3	61.6	94.4

Key observations:

• Auto and Ignore 1 wait produce identical results, suggesting BF 1x is not actually changing the generation.
• AIME24 is 70.0 without BF (paper: 56.7), then drops to 53.3 with BF 2x — the opposite direction from expected.
• AIME25 consistently undershoots the paper's 50.0 at 43.3.
• GPQA and MATH are reasonably close to published values.

My s1 results (modified vLLM)

Config	AIME24	AIME25	GPQA	MATH
Paper w/o BF	50.0	26.7	56.6	92.6
Paper BF 1x	53.3	30.0	59.6	92.8
Paper BF 2x	53.3	33.3	59.6	93.0
My Auto (no BF)	40.0	26.7	56.1	92.4
My Ignore 1 wait	40.0	26.7	56.1	92.6
My Ignore 2 wait	40.0	26.7	56.1	92.4

Key observations:

• All three configs produce essentially identical scores. Budget forcing has zero effect, confirming the wait injection is not functioning.
• AIME24 is 40.0 across all configs, 10 points below the paper's w/o BF (50.0).
• AIME25 matches the paper's w/o BF (26.7) but doesn't improve with BF as expected.
• GPQA and MATH are close to but slightly below published values.

What I've investigated so far

To debug this, I looked at the s2 branch which uses a local simple-verify library instead of the original chat-style extraction logic. Suspecting my budget forcing implementation might be incorrect, I tried bypassing it and using the original chat-style extraction approach directly, but still get incorrect AIME scores. This suggests the issue isn't solely in the extraction pipeline, and likely also involves how the generation itself differs on newer vLLM versions.

What I'm looking for

Given the symptoms, there appear to be two distinct issues:

1. Budget forcing is non-functional: The identical scores across Auto/Ignore 1/Ignore 2 (especially for s1) strongly suggest the "Wait" injection is not actually modifying the generation. Any guidance on how the end-of-thinking token suppression and wait appending should interact with newer vLLM versions would be helpful.

2. AIME score divergence even without BF: The s1 AIME24 score (40.0) is 10 points below the paper's w/o BF (50.0), while for s1.1 it overshoots (70.0 vs 56.7). MATH and GPQA are close in both cases. This points to either a generation-level difference in how newer vLLM handles these models, or a subtle difference in the AIME answer extraction for integer answers (000–999).

Any tips on debugging either of these would be appreciated. Since the original repo gives exact scores, the root cause is in my vLLM migration.

[Muennighoff]

just made that repo public here in case useful to you: https://github.com/Muennighoff/simpleverify
though I havn't tried the Wait budget forcing on that branch I think.

I'm not sure what the issue is with the newer vLLM.
@cursor / @copilot can you investigate and figure it out?

[asparius]

Thanks for the quick reply, I will try with simpleverify and let you know. I have asked opus 4.6 etc, so at the end of the day it was hinting to different vllm version.

[asparius]

Update: pretty sure this is just vLLM numerical non-determinism

I dug into this more and I'm now fairly confident that both the AIME score differences and the budget forcing issue come down to numerical non-determinism between vLLM 0.6.1 and 0.11, not anything wrong with my migration code.

What I did

I ran the same s1-32B eval with BF "Wait" 1x on both vLLM 0.6.1 (original repo) and vLLM 0.11, both with --log_samples. Then I did two things:

First, a token-level diff: I tokenized the outputs from both runs and compared them token by token for each problem to find the exact position where the two versions first produce a different token.

Second, an LLM-as-judge analysis: for every problem where one vLLM version got the right answer and the other didn't, I sent both full reasoning traces along with the problem and correct answer to an LLM, asking it to figure out where the reasoning actually diverges and what mathematical error the failing version made.

Prompts are identical

The prompt_hash in the logged samples matches exactly between both runs for every problem. No template or encoding difference.

Token diff results

Almost every AIME sample diverges mid-generation. On AIME24, 28 out of 30 samples diverge, with the first divergence happening between token 227 and token 7308 (average around 2619). On AIME25, 25 out of 30 diverge, first divergence between token 37 and 6884 (average 2293). Zero samples diverge at the prompt level on either benchmark.

So the model gets the exact same input, starts generating the same tokens, and then hundreds or thousands of tokens in it picks a different token and the whole chain of thought goes in a different direction.

The diverging tokens are basically synonyms:

Problem	Token position	vLLM 0.6.1 picks	vLLM 0.11 picks
4	80	" integer"	" number"
5	4619	" question"	" problem"
6	4211	"P"	"A" (variable name)
16	391	" for"	" of"
25	1382	" indices"	" points"
17	1391	".\n\n"	".\n"

These are all cases where two tokens presumably have very close logprobs and some floating-point difference in the backend tips it one way. I haven't actually compared logprobs between the two versions (that would need more instrumentation), but the pattern is consistent: the divergences are always at semantically equivalent tokens, never at points where one version produces something obviously wrong.

Why MATH and GPQA are fine but AIME isn't

MATH500 has shorter traces (avg ~4700 tokens) and 500 problems, so per-problem divergences cancel out. AIME has much longer traces (avg 7000-10000 tokens, more chances to diverge) and only 30 problems, so flipping 3-4 answers moves the score by 10+ points.

Benchmark	n	Avg tokens (0.6.1)	Avg tokens (0.11)	Token delta	Score diff
MATH500	500	4727.8	4487.5	~5%	0.2 pts
AIME24	30	7420.7	8932.2	~20%	13.3 pts
AIME25	30	9805.6	8061.2	~18%	3.3 pts

LLM judge results

For each AIME problem where one vLLM version got it right and the other didn't, the LLM judge analyzed both reasoning traces.

On AIME24 there were 4 flipped samples, all cases where 0.6.1 was correct and 0.11 was wrong. On problem #6 (target=721), both traces initially misinterpret the problem as minimization, but the 0.6.1 trace self-corrects and realizes it needs the maximum while the 0.11 trace never catches this. On #13 (target=197), the 0.11 trace incorrectly identifies chain length as perpendicular separation. On #14 (target=480), the 0.11 trace introduces a factor-of-2 error simplifying a hyperbola equation. On #28 (target=127), the 0.11 trace picks a wrong geometric model for a torus intersection while 0.6.1 pivots to an approach that works.

On AIME25 there were 3 flipped samples. Problem #4 (target=279): the 0.11 trace omitted one valid subset {1,3,6,8} when enumerating, undercounting by 288 permutations. Problem #11 (target=510): here 0.11 actually did better, 0.6.1 made a sign error that 0.11 avoided. Problem #24 (target=907): the 0.11 trace computed 12870−11583=1297 instead of 1287, a simple arithmetic slip.

These aren't random failures. They're cases where the model is on a knife's edge and a slightly different reasoning path leads to missing one case in an enumeration, dropping a factor of 2, or failing to self-correct.

Budget forcing

I now think the budget forcing issue is probably related to the same root cause. If the generation itself is already diverging between vLLM versions, the "Wait" injection interacts with a different reasoning state and may not have the same effect. The identical scores between Auto and BF 1x on vLLM 0.11 could be a consequence of this rather than the injection mechanism being broken. Still looking into this.

Bottom line

The AIME score differences aren't a bug in my migration. The prompts are identical, the divergence is always mid-generation at synonym-level token choices, and the LLM judge confirms the resulting errors are consistent with fragile reasoning being perturbed by numerical differences in the inference backend.

For anyone else trying to reproduce s1 results on newer vLLM: MATH and GPQA will be within ~0.5 points, but AIME scores can shift by ±10 points and this is expected.

@Muennighoff