Illegal submissions megathread

[0hq]

Hey all! We're seeing a lot of invalid solutions due to autoresearch and some rules confusion. Clarifying everything here.

[0hq]

From @valerio-oai:

PR ##589 is a potential candidate, but it only beats #549 by 0.002 nats.
PR ##473 is valid and technically sota over the current leader ##374, but also does not clear the bar, it's like 0.001 nats better
Invalid record runs:

PR ##410, ##415, ##417: invalid TTT; two-phase/multi-epoch adaptation on validation, then reported score after adaptation.
PR ##442, ##462, ##481, ##486, ##517, ##518, ##532, ##555, ##581, ##595: invalid TTT; adapt on validation before the reported eval pass.
PR ##461: non-record track submission.
PR ##467: explicit non-record; also keeps the final-epoch score in multi-epoch LoRA TTT.
PR ##503, ##528, ##529: not compliant submission packages; root train_gpt.py only, no proper /records folder; #503 also says total eval is ~616s (>10 min).
PR ##505: incomplete submission; missing required submission.json and train logs; also claims ~18 min/seed.
PR ##512, ##548, ##568: invalid TTT; multi-epoch TTT reports the final-pass/final-epoch score after earlier adaptation on the same doc/chunk.
PR ##535, ##544, ##545, ##569, ##585, ##593: invalid eval; GPTQ/Hessian calibration uses fineweb_train_* during evaluation.
PR ##573: invalid eval; oracle/hindsight min() selection across multiple passes.
PR ##576: invalid eval; re-scores the full validation set after TTT with the fully adapted model.
PR ##587: invalid eval; GPTQ/Hessian calibration uses fineweb_train_* during evaluation. Also not a compliant /records submission package.
#659 At eval-time, it constructs a 5-gram model, where it scores a token, places it into the model, etc. As it's doing that, at every token, before scoring the token it checks the NLL of both the model and the 5-gram and picks the one where the loss is lower -- clearly this just relies on knowledge of the true token/teacher forcing.

Also a tactic more PRs have been using that it's good to be aware of: people are trying to sneak in extra compute between training and eval by arguing it's part of "artifact construction" (i.e. quantization and whatnot). This is most commonly done through "GPTQ calibration", which uses training data to somehow calibrate the model (hence legal at training time, but it's illegal at eval time) -- this takes 4-40s depending on implementation but people still train for the full 10min, then do this, and then do a full 10min eval, which is not legal

[valerio-oai]

Another pass:

#606: eval-time GPTQ recalibrates on fineweb_train_* after the 600s training cap.
#610: Unclear if this is meant to be a record run, but 1200s training disqualifies it.
#615: same eval-time GPTQ-on-training-data issue.
#626: same eval-time GPTQ-on-training-data issue.
#639: not a compliant /records submission and also has eval-time GPTQ on training data.
#656: not a compliant /records submission; mean score does not beat #549; code also has eval-time GPTQ on training data.
#674: not a compliant /records submission, and its GPTQ calibration explicitly reopens fineweb_train_* during eval/export.
#684: I'm pretty sure this multi-epoch TTT scheme trains on eval tokens, since it seems to do multiple passes and take the loss at the last epoch. Closing for now.
#685: Closing for now, min(NLL) over multiple passes means you're training on the eval set.