Request for Access to Human Dataset from Baker et al. 2022 Benchmark

[mond-res]

Hello Brain-Score Team,
I am currently conducting research related to human shape perception and would like to evaluate my models on the benchmark introduced in Baker et al. (2022). For this purpose, I am seeking access to the human behavioral dataset used in that benchmark (Baker 2022).
Could you please let me know how I can obtain the dataset ?

Thanks in advance.

[KartikP]

@mond-res If you are referring to the raw behavioral dataset that was processed into the Brain-Score benchmark dataset, you can find the OSF link in the original paper. This is the OSF link.

If you are referring to the actual Brain-Score benchmark data (i.e., the post-processed version that is used on Brain-Score), you can use the following:

import brainscore_vision

brainscore_vision.load_stimulus_set("Baker2022_normal_distortion")
brainscore_vision.load_dataset("Baker2022_normal_distortion")

brainscore_vision.load_stimulus_set("Baker2022_inverted_distortion")
brainscore_vision.load_dataset("Baker2022_inverted_distortion")

You will then find the contents of the dataAssembly and stimulusSet in your .brainio/ folder.

Then refer to the /benchmark/Baker2022/benchmark.py for the full implementation of the benchmark.

In the /data/Baker2022/ folder, you will also see the data_packaging/ that describes how the Brain-Score dataset was created from the source dataset.

[mond-res]

@mond-res If you are referring to the raw behavioral dataset that was processed into the Brain-Score benchmark dataset, you can find the OSF link in the original paper. This is the OSF link.

If you are referring to the actual Brain-Score benchmark data (i.e., the post-processed version that is used on Brain-Score), you can use the following:

import brainscore_vision

brainscore_vision.load_stimulus_set("Baker2022_normal_distortion")
brainscore_vision.load_dataset("Baker2022_normal_distortion")

brainscore_vision.load_stimulus_set("Baker2022_inverted_distortion")
brainscore_vision.load_dataset("Baker2022_inverted_distortion")

You will then find the contents of the dataAssembly and stimulusSet in your .brainio/ folder.

Then refer to the `/benchmark/Baker2022/benchmark.py` for the full implementation of the benchmark.

In the /data/Baker2022/ folder, you will also see the data_packaging/ that describes how the Brain-Score dataset was created from the source dataset.

Thanks for your answer what is the difference between the brain score human Dataset and the OSF link ?

[KartikP]

The main difference is that the Brain-Score DataSet is a standardized repackaging of the OSF dataset.

Often times there are no rigid rules between how labs must submit data to open science repositories. This makes it very difficult to plug-and-play models to be evaluated on various benchmarks (DataSet + Metric). As such, in Brain-Score we specify a data standard with a minimum set of criteria to allow models to be easily evaluated on a benchmark.

For the Baker2022 (and most Behavioral Benchmarks), this is just taking a tabular file (CSV; Pandas dataframe), and mapping the columns to Xarray dimensions. You can see it here.