Custom Google Colabs RT1_pytorch_training

All implementation have been moved to google colabs here

All explanation below to run on google colabs, if want to run local refer to the forked repository.

Data

Get RT1 dataset

gsutil cp -R "gs://gresearch/robotics/fractal20220817_data/0.1.0" destination-directory

# if you want to limit, change XXXXXX to what you want
!gsutil cp "gs://gresearch/robotics/fractal20220817_data/0.1.0/fractal20220817_data-train.tfrecord-0000*-of-XXXXX" VLMART/fractal20220817_data/0.1.0/

# and because we limit not include all, we need include ourself the .json file

!gsutil cp "gs://gresearch/robotics/fractal20220817_data/0.1.0/*.json" VLMART/fractal20220817_data/0.1.0/

Env Explanation

• rt-tf is for dataset segment use
• rt only its for datapreprocess

So its sequential, but in google colabs implementation in the end will be merge the requirement for both env.

Dataset segment

# for starter install depend requirement from env rt-tf ref: env/rt-tf_requirements.txt, then continue.

python data_preprocessing/rt_data_generator_all.py


episode section length
# adjust this in program for smaller size or bigger
# episode_sec_len = 20   

# The memory size of 100 个 episode:6.12 G
# The memory size of 20 个 episode:1.2 G

Batch will generate .h5 files for episode_sec_len episodes

continue with

run rt_data_generator_pick_batch.py

how big is your sec_length, or the dataset you download its affecting the task you need. If you follow my configuration it will only generate pick object only task.

DataPreprocess

# install depend requirement from env rt ref: env/rt_requirements.txt, then continue.

python DataPreprocess_batch_all.py

# run DataPreprocess_batch_pick.py   # only pick object task

To realize the data preprocessing, specific preprocessing operation details of the subsequent update or read the source code themselves.

Generate data index for dataloader

python data_emerge.py

you will Get two .csv file

chang the filepath_or_buffer and self.size of train and val.

Training

Google colab configuration use Single GPU and A100 GPU configuration, all adjustment from the original repo already adjust on this repo.

python -m torch.distributed.launch --nproc_per_node=4 --use_env train_ddp.py

Start training, one epoch takes about 4 hours

---

note：pip install robotic-transformer-pytorch .After installation, if you can not scientifically surf the Internet, you need to download the T5 model and adjust the path of T5.

change path
file: /mnt/anaconda3/envs/rt/lib/python3.10/site-packages/classifier_free_guidance_pytorch/t5.py

DEFAULT_T5_NAME = '/mnt/robotic-transformer-pytorch/t5/t5-v1_1-base'(Downloaded model file address)

---

Appreciation

• Stability.ai for the generous sponsorship to work and open source cutting edge artificial intelligence research.
• https://github.com/lucidrains/robotic-transformer-pytorch for RT1model(pytorch).
• https://github.com/google-deepmind/open_x_embodiment for dataset process.

Citations

@inproceedings{rt12022arxiv,
    title    = {RT-1: Robotics Transformer for Real-World Control at Scale},
    author   = {Anthony Brohan and Noah Brown and Justice Carbajal and  Yevgen Chebotar and Joseph Dabis and Chelsea Finn and Keerthana Gopalakrishnan and Karol Hausman and Alex Herzog and Jasmine Hsu and Julian Ibarz and Brian Ichter and Alex Irpan and Tomas Jackson and  Sally Jesmonth and Nikhil Joshi and Ryan Julian and Dmitry Kalashnikov and Yuheng Kuang and Isabel Leal and Kuang-Huei Lee and  Sergey Levine and Yao Lu and Utsav Malla and Deeksha Manjunath and  Igor Mordatch and Ofir Nachum and Carolina Parada and Jodilyn Peralta and Emily Perez and Karl Pertsch and Jornell Quiambao and  Kanishka Rao and Michael Ryoo and Grecia Salazar and Pannag Sanketi and Kevin Sayed and Jaspiar Singh and Sumedh Sontakke and Austin Stone and Clayton Tan and Huong Tran and Vincent Vanhoucke and Steve Vega and Quan Vuong and Fei Xia and Ted Xiao and Peng Xu and Sichun Xu and Tianhe Yu and Brianna Zitkovich},
    booktitle = {arXiv preprint arXiv:2204.01691},
    year      = {2022}
}

@inproceedings{Tu2022MaxViTMV,
    title   = {MaxViT: Multi-Axis Vision Transformer},
    author  = {Zhengzhong Tu and Hossein Talebi and Han Zhang and Feng Yang and Peyman Milanfar and Alan Conrad Bovik and Yinxiao Li},
    year    = {2022}
}

@misc{peebles2022scalable,
    title   = {Scalable Diffusion Models with Transformers},
    author  = {William Peebles and Saining Xie},
    year    = {2022},
    eprint  = {2212.09748},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}