InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions

Kushal Kedia, Atiksh Bhardwaj, Prithwish Dan, Sanjiban Choudhury

Cornell University

ICRA 2024

Project Page | arxiv

Installation

Follow these steps to install InteRACT:

1. Create and activate the conda environment:

   cd interact
   conda create --name interact python=3.8.16
   conda activate interact
   pip install -r requirements.txt
   pip install -e . 

Preliminaries

1. Set the base_dev_dir to your working directory in all of the config files

2. Create the data directory: Create a new directory called "data" under interact so that the repo has the following structure:

Repo Structure

├── config
│   ├── *.yaml files 
├── interact
|   ├── checkpoints
|     ├── HH_checkpoints
|     ├── HR_checkpoints
|   ├── data
|     ├── comad_data
|     ├── amass
|     ├── cmu_mocap (optional)     
|   ├── model
|     ├── model architecture files...
|   ├── utils
|     ├── utility files...
|   ├── scripts
|     ├── eval_hh.py / eval_hr.py        <- evaluation scripts
|     ├── pretrain_intent_forecaster.py  <- pretraining on H-H 
|     |── finetune_intent_forecaster.py  <- finetuning on H-H
|     |── hr_transfer.py                 <- transferring to H-R
|   ├── mapping
|     ├── files for joint mapping...
|   ├── body_models
|     ├── SMPL skeleton file (used for AMASS data)
|
├── environment.yml
├── README.md
├── setup.py

Dataset Installation and Preprocessing

AMASS

Download datasets listed in configs/synthetic_amass.yaml from the official AMASS website. Use any of the SMPL-X or SMPL-H links to download each dataset.

The AMASS dataset contains data of single human motion. Preprocess this data to create synthetic two-human data:

python scripts/create_synthetic_amass.data.py

Note: Update the config file configs/synthetic_amass.yaml with the specific datasets you have in your data directory before running this script.

CoMaD

Download the data from this link Data into the correct data directory.

You can download the .zip file and extract the data using the following terminal commands:

wget https://cornell.box.com/shared/static/6ss0mfojdof8q1z9ru7go58rwxqbnel5.zip -O comad_data.zip
unzip comad_data.zip -d data/comad_data/
rm comad_data.zip

Training

1. Run the pretraining script on large-scale H-H data:

   python scripts/pretrain_intent_forecaster.py

2. Run the finetuning script on H-H interaction data.

   python scripts/finetune_intent_forecaster.py

3. Run the script to transfer the model to the H-R setting.

   python scripts/hr_transfer.py

Evaluation

1. Run the evaluation script for H-H:

   python scripts/eval_hh.py

2. Run the evaluation script for H-R:

   python scripts/eval_hr.py

BibTeX

@article{kedia2023interact,
 title={InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions},
 author={Kedia, Kushal and Bhardwaj, Atiksh and Dan, Prithwish and Choudhury, Sanjiban},
 journal={arXiv preprint arXiv:2311.12943},
 year={2023}
}

Acknowledgement

• MRT is adapted from MRT