Gerbench

[jphme]

Datasets:

https://huggingface.co/datasets/ellamind/gerbench_sentence_errors

https://huggingface.co/datasets/ellamind/gerbench_next_word

Target:

• Improve/test the benchmark code
• Implement a version for next word not reliant on multiple choice -> sample the continuation (investigate how to handle different token lengths of correct answers properly)
• Optional: Also add a perplexity benchmark (based on the satz_bis_zum_letzten_wort column) and compare (relative) results

Models to test/eval at start:

• Mistral Nemo 12b
• Llama 3.1 8b
• Llama 3.2 3b
• Qwen 2.5 7b / 14b
• Optinal: Gpt-4o / Gpt-4o-mini / Claude (if possible?)

[notion-workspace]

☑️ Finish Lighteval “Gerbench” to evaluate German language capabilities systematically