Group-based Splitting for Benchmarks + Batch-layer Evaluation + GPU Support

[BKHMSI]

No description provided.

[KartikP]

Hi @BKHMSI did you intend to remove Pereira2018.243sentences-linear?

[BKHMSI]

Hi @BKHMSI did you intend to remove Pereira2018.243sentences-linear?

Yes, I did intend to change all benchmarks to use ridge regression instead of linear.

[mschrimpf]

let's keep the original ones for reference? (at least in code, don't have to display on the website.)
I agree we should use ridge going forward

[BKHMSI]

Re-added the linear metrics for all benchmarks and fixed ceiling for ridge.

Note that for Pereira2018, we need to cache ceilings for the new metrics.

[mike-ferguson]

Hi all,

Please note a few things about the state of the language repo:

1. Automerging currently disabled for benchmark-only PRs. Merging will require manual approval by Brain-Score admin (Kartik, me, or Martin most likely)
2. Scoring is disabled as well until language scoring infra comes online (~ end of next week)