teaser_video.mp4
💫 Meet LingBot-VA! We've built an AR diffusion framework for simultaneous world modeling and action! 🤖✨
LingBot-VA has focused on:
- Autoregressive Video-Action World Modeling: Architecturally unifies visual dynamics prediction and action inference within a single interleaved sequence while maintaining their conceptual distinction.
- High-efficiency Execution: A dual-stream mixture-of-transformers(MoT) architecture with Asynchronous Execution and KV Cache.
- Long-Horizon Performance and Generalization: High improvements in sample efficiency, long-horizon success rates, and generalization to novel scenes.
- [2026-02-17] Post-training code and dataset released! Support fine-tuning LingBot-VA on custom robotic manipulation datasets.
- [2026-01-29] Weights and code for shared backbone released! Please stay tuned for our separated version!
- Pretrained Checkpoints for Post-Training
| Model Name | Huggingface Repository | ModelScope Repository | Description |
|---|---|---|---|
| lingbot-va-base | 🤗 robbyant/lingbot-va-base | 🤖 Robbyant/lingbot-va-base | LingBot-VA w/ shared backbone |
| lingbot-va-posttrain-robotwin | 🤗 robbyant/lingbot-va-posttrain-robotwin | 🤖 Robbyant/lingbot-va-posttrain-robotwin | LingBot-VA-Posttrain-Robotwin w/ shared backbone |
- Post-Training Dataset
| Dataset Name | Repository | Description |
|---|---|---|
| robotwin-clean-and-aug-lerobot | 🤗 robbyant/robotwin-clean-and-aug-lerobot | Cleaned & augmented RoboTwin dataset in LeRobot format for post-training |
Requirements • Python == 3.10.16 • Pytorch == 2.9.0 • CUDA 12.6
pip install torch==2.9.0 torchvision==0.24.0 torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu126
pip install websockets einops diffusers==0.36.0 transformers==4.55.2 accelerate msgpack opencv-python matplotlib ftfy easydict
pip install flash-attn --no-build-isolationYou MUST change the
attn_modesetting depending on whether you are training or running inference. Since LingBot-VA is loaded viafrom_pretrained, this parameter is read from the model folder'stransformer/config.json. You need to manually edit this file before launching.
Mode attn_modevalueNotes Training "flex"Required for training. Will not work for inference. Inference / Evaluation "torch"or"flashattn"Required for inference. "flex"will cause errors at eval time.How to change: Open
<your-model-path>/transformer/config.json, find the"attn_mode"field, and set it to the appropriate value.
LingBot-VA supports both standalone execution and Server-Client architecture which separates the model environment from simulation. By isolating dependencies, the design avoids package clashes and supports distributed inference on GPUs, clusters, and other devices.
Preparing the Environment
You can follow the official instructions from the original RoboTwin-2.0 repository:
https://robotwin-platform.github.io/doc/usage/robotwin-install.html
In summary:
sudo apt install libvulkan1 mesa-vulkan-drivers vulkan-toolsgit clone https://github.com/RoboTwin-Platform/RoboTwin.git && cd RoboTwin && git checkout 2eeec322- modify script/requirements.txt
transforms3d==0.4.2
sapien==3.0.0b1
scipy==1.10.1
mplib==0.2.1
gymnasium==0.29.1
trimesh==4.4.3
open3d==0.18.0
imageio==2.34.2
pydantic
zarr
openai
huggingface_hub==0.36.2
h5py
# For Description Generation
azure==4.0.0
azure-ai-inference
pyglet<2
wandb
moviepy
imageio
termcolor
av
matplotlib
ffmpeg- modify line 8 of script/_install.sh:
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable" --no-build-isolation- run:
bash script/_install.sh- run:
bash script/_download_assets.shDeploying the Inference Server
# single GPU
bash evaluation/robotwin/launch_server.sh
# multi-GPU
bash evaluation/robotwin/launch_server_multigpus.shExecuting the Inference Client
# single GPU
task_name="adjust_bottle";
save_root="results/";
bash evaluation/robotwin/launch_client.sh ${save_root} ${task_name}
# multi-GPU
save_root="results/"
task_group_id=0;
bash evaluation/robotwin/launch_client_multigpus.sh ${save_root} ${task_group_id}Related experiments results will be save in /path/to/your/RoboTwin/${save_root}. Please note that an eval_result folder is also generated. This is a native output from RoboTwin and is identical to the contents in the results folder; it can be safely ignored.
It is important to note that the inference server and client must be deployed on the same machine. For launching multi-GPU client, we padded the original 50 tasks to 56 via duplication and partitioned them into 7 groups to align with the 8-GPU configuration of our inference node. You can specify the task_group_id (0-6) to select a particular group for inference. For detailed grouping configurations, please refer to evaluation/robotwin/launch_client_multigpus.sh.
GPU Memory Requirements: Approximately 24GB VRAM for single-GPU RoboTwin evaluation with offload mode enabled (VAE and text_encoder offloaded to CPU).
We also provide a script for image to video-action generation:
NGPU=1 CONFIG_NAME='robotwin_i2av' bash script/run_launch_va_server_sync.shGPU Memory Requirements: Approximately 18GB VRAM for single-GPU i2av inference with offload mode enabled (VAE and text_encoder offloaded to CPU).
We support post-training (fine-tuning) LingBot-VA on custom robotic manipulation datasets. The training pipeline uses FSDP for distributed training and integrates with LeRobot dataset format.
On top of the base installation, post-training requires:
pip install lerobot==0.3.3 scipy wandb --no-depsDownload the post-training dataset from HuggingFace:
huggingface-cli download --repo-type dataset robbyant/robotwin-clean-and-aug-lerobot --local-dir /path/to/your/datasetIf you want to fine-tune LingBot-VA on your own robotic manipulation data, follow these steps:
When preparing your custom dataset, the data goes through the following processing pipeline:
- Raw Data → Convert to LeRobot format (with metadata and video files)
- Add Action Segmentation → Add
action_configtoepisodes.jsonl - Extract Latents → Process videos through VAE according to video specifications
- Dataset Loading → Load processed data with proper action dimensions for training
The final data should conform to these specifications:
Action Format:
- Output dimension: 30 dimensions, structured as follows:
- Left arm EEF (end-effector): 7 dimensions
- Right arm EEF (end-effector): 7 dimensions
- Left arm joints: 7 dimensions
- Right arm joints: 7 dimensions
- Left arm gripper: 1 dimension
- Right arm gripper: 1 dimension
- In your dataset class loader, map your robot's action dimensions to this standard 30-dimensional format. Missing dimensions are padded with 0.
Video Format:
- During VAE latent extraction, resize videos to ~256 × 256 pixels and downsample to 5-15 fps as a reference (adjust based on your task requirements).
Step 1: Convert your data to LeRobot format
Follow the official LeRobot dataset documentation to convert your raw data (e.g., HDF5, video files, etc.) into the standard LeRobot dataset format. Ensure that each episode contains the required observation videos, actions, and metadata.
Step 2: Add action_config field to episodes.jsonl
After converting to LeRobot format, you need to modify the meta/episodes.jsonl file to add an action_config field to each line. This field describes the temporal segmentation and natural language description of the robot's actions within each episode.
Each line in episodes.jsonl should follow this format:
{
"episode_index": 0,
"tasks": ["task description"],
"length": 450,
"action_config": [
{
"start_frame": 0,
"end_frame": 450,
"action_text": "Natural language description of the robot action in this segment.",
}
]
}start_frame/end_frame: The frame range (0-indexed) of the action segment within the episode.action_text: A natural language description of what the robot does in this segment.
For episodes with a single continuous action, start_frame should be 0 and end_frame should equal the episode length. You can also define multiple segments per episode if your data contains sequential sub-tasks.
Step 3: Extract video latents with Wan2.2 VAE
LingBot-VA operates on video latent representations rather than raw pixels. You need to extract the latent features using the Wan2.2 VAE encoder and place them under the converted LeRobot dataset directory. Please refer to the Wan-Video documentation for instructions on how to run the VAE encoder.
The extracted latent files should be placed under latents/ in your dataset directory, mirroring the structure of videos/:
your_dataset/
├── videos/
│ └── chunk-000/
│ └── observation.images.cam_high/
│ ├── episode_000000.mp4
│ └── ...
├── latents/
│ └── chunk-000/
│ └── observation.images.cam_high/
│ ├── episode_000000_0_450.pth # named as episode_{index}_{start_frame}_{end_frame}.pth
│ └── ...
└── meta/
└── episodes.jsonl
Each .pth file is a dictionary containing the following fields:
| Key | Type | Description |
|---|---|---|
latent |
Tensor [N, C] (bfloat16) |
Flattened VAE latent features (e.g., shape [latent_num_frames * latent_height * latent_width, C]) |
latent_num_frames |
int |
Number of temporal frames in the latent space |
latent_height |
int |
Spatial height in the latent space |
latent_width |
int |
Spatial width in the latent space |
video_num_frames |
int |
Number of frames in the (sampled) source video |
video_height |
int |
Original video height in pixels |
video_width |
int |
Original video width in pixels |
text_emb |
Tensor [L, D] (bfloat16) |
Text embedding of the action description (encoded by Wan2.2 text encoder) |
text |
str |
The raw action description text |
frame_ids |
list[int] |
Sampled frame indices from the original episode (at target fps) |
start_frame |
int |
Start frame index matching action_config in episodes.jsonl |
end_frame |
int |
End frame index matching action_config in episodes.jsonl |
fps |
int |
Target sampling fps used for latent extraction |
ori_fps |
int |
Original fps of the episode data |
The latent file naming convention episode_{index}_{start_frame}_{end_frame}.pth corresponds to the action_config segments defined in episodes.jsonl. For example, an episode with "start_frame": 0, "end_frame": 450 produces a latent file named episode_000000_0_450.pth.
NGPU=8 bash script/run_va_posttrain.shFor better training performance, use a larger global batch size (e.g., 32, 64). If you have limited GPU resources, you can increase gradient_accumulation_steps to achieve a larger effective batch size.
We evaluate our model on both simulation benchmarks and real-world scenarios, and achieve state-of-the-art performance.
- RoboTwin 2.0
We are the first to propel RoboTwin 2.0 metrics performance past the 90+ threshold！
* All metrics are reported in percentage (%). Higher values are bolded.
| Method (Average 50 Tasks) | Easy SR (%) | Hard SR (%) |
|---|---|---|
| X-VLA | 72.9 | 72.8 |
| π0 | 65.9 | 58.4 |
| π0.5 | 82.7 | 76.8 |
| Motus | 88.7 | 87.0 |
| LingBot-VA (Ours) | 92.9 (+4.2) | 91.6 (+4.6) |
- LIBERO
* All metrics are reported in percentage (%). Higher values are bolded.
| Methods | Spatial | Object | Goal | Long | Avg |
|---|---|---|---|---|---|
| π0 | 96.8 | 98.8 | 95.8 | 85.2 | 94.1 |
| π0.5 | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 |
| OpenVLA | 84.7 | 88.4 | 79.2 | 53.7 | 76.5 |
| X-VLA | 98.2 | 98.6 | 97.8 | 97.6 | 98.1 |
| LingBot-VA (Ours) | 98.5 ± 0.3 | 99.6 ± 0.3 | 97.2 ± 0.2 | 98.5 ± 0.5 | 98.5 |
Six manipulation tasks across three categories: longhorizon tasks (Make Breakfast, Pick Screws), precision tasks (Insert Tube, Unpack Delivery), and deformable & articulated object manipulation (Fold Clothes, Fold Pants). Our method achieves state-of-the-art performance on both metrics (Progress Rate and Success Rate) with only 50 trials per task, substantially outperforming strong baseline π0.5.
PS = Average_Progress / Max_Steps × 100%
SR = Successful_Trials / N × 100%
* All metrics are reported in percentage (%). Higher values are bolded.
| Task | Make Breakfast | Pick Screws | Insert Tube | Unpack Delivery | Fold Clothes | Fold Pants | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PS | SR | PS | SR | PS | SR | PS | SR | PS | SR | PS | SR | |
| π0.5 | 73.0 | 70.0 | 74.0 | 50.0 | 79.2 | 30.0 | 73.0 | 25.0 | 62.9 | 30.0 | 30.0 | 30.0 |
| LingBot-VA (Ours) | 97.0 | 75.0 | 82.5 | 70.0 | 85.8 | 40.0 | 84.5 | 65.0 | 48.8 | 35.0 | 76.7 | 70.0 |
This project is released under the Apache License 2.0. See LICENSE file for details.
@article{lingbot-va2026,
title={Causal World Modeling for Robot Control},
author={Li, Lin and Zhang, Qihang and Luo, Yiming and Yang, Shuai and Wang, Ruilin and Han, Fei and Yu, Mingrui and Gao, Zelin and Xue, Nan and Zhu, Xing and Shen, Yujun and Xu, Yinghao},
journal={arXiv preprint arXiv:2601.21998},
year={2026}
}This work builds upon several excellent open-source projects:
- Wan-Video - Vision transformer backbone
- MoT - Mixture-of-Transformers architecture
- The broader open-source computer vision and robotics communities
For questions, discussions, or collaborations:
- Issues: Open an issue on GitHub
- Email: Contact Dr. Qihang Zhang (liuhuan.zqh@antgroup.com) or Dr. Lin Li (fengchang.ll@antgroup.com)
