C2F-LDM

Code for Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from human brain activity

1  Download

1.1  Datasets

Brain2Sound Dataset https://github.com/KamitaniLab/SoundReconstruction

Brain2Music Dataset https://openneuro.org/datasets/ds003720

Brain2Speech Dataset https://openneuro.org/datasets/ds003020/versions/1.1.1

1.2  Checkpoints

download pretrained.pth from AudioMAE

download audioldm2-full.pth and audioldm2-speech-gigaspeech.pth from AudioLDM2

2  Environment

Follow the steps below to set up the virtual environment.

Create and activate the environment:

conda create -n c2f_ldm python=3.10
conda activate c2f_ldm

Install dependencies in the listed order:

pip install -r requirements.txt

3  Coarse-to-fine brain decoding

3.1  Coarse-grained semantic decoding

First, extract the semantic features of the ground truth:

python semantic_decoding/extract_gt_feat.py -d brain2sound
python semantic_decoding/extract_gt_feat.py -d brain2music
python semantic_decoding/extract_gt_feat.py -d brain2speech

Next, perform L2-regularized linear regression:

python semantic_decoding/sound_decoding.py
python semantic_decoding/music_decoding.py
python semantic_decoding/speech_decoding.py

3.2  Fine-grained acoustic decoding

Specify the subject ID in the configuration file and then run:

CUDA_VISIBLE_DEVICES=1 python acoustic_decoding/train_AcousticDecoder.py -c configs/brain2sound.yaml
CUDA_VISIBLE_DEVICES=1 python acoustic_decoding/train_AcousticDecoder.py -c configs/brain2music.yaml
CUDA_VISIBLE_DEVICES=1 python acoustic_decoding/train_AcousticDecoder.py -c configs/brain2speech.yaml

4  Brain-to-audio reconstruction

Specify the subject ID and the checkpoint path of the pretrained AcousticDecoder in the configuration file and then run:

CUDA_VISIBLE_DEVICES=1 python reconstruction/train_LDM.py -c configs/brain2sound.yaml --reload_from_ckpt audioldm2-full
CUDA_VISIBLE_DEVICES=1 python reconstruction/train_LDM.py -c configs/brain2music.yaml --reload_from_ckpt audioldm2-full
CUDA_VISIBLE_DEVICES=1 python reconstruction/train_LDM.py -c configs/brain2speech.yaml --reload_from_ckpt audioldm2-speech-gigaspeech