GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
TULLUS
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
TULLUS
Model Tree
robotics-diffusion-transformer/RDT2-VQ
Input Modalities
Output Modalities
Supported Functionality
20-D per step = right (10) + left (10):
Output tensor shape: (T=24, D=20), relative deltas, float32.
The RVQ tokenizer yields a fixed-length token sequence; see tokenizer card for exact code lengths.
robotics-diffusion-transformer/RVQActionTokenizerApproximate single-GPU requirements (Qwen2.5-VL-7B-Instruct scale):
| Mode | RAM | VRAM | Example GPU |
|---|---|---|---|
| Inference | ≥ 32 GB | ≥ 16 GB | RTX 4090 |
| LoRA FT | – | ≥ 32 GB | A100 40GB |
| Full FT | – | ≥ 80 GB | A100 80GB / H100 / B200 |
For deployment on real robots, follow your platform’s end-effector + camera choices and perform hardware setup & calibration (camera stand/pose, flange, etc.) before running closed-loop policies.
Tested OS: Ubuntu 24.04.
python
# Run under repository: https://github.com/thu-ml/RDT2 import torchfrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration from vqvae import MultiVQVAEfrom models.normalizer import LinearNormalizerfrom utils import batch_predict_action # assuming using gpu 0device = "cuda:0" processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")model = Qwen2_5_VLForConditionalGeneration.from_pretrained( "robotics-diffusion-transformer/RDT2-VQ" torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", device_map=device).eval()vae = MultiVQVAE.from_pretrained("robotics-diffusion-transformer/RVQActionTokenizer").eval()vae = vae.to(device=device, dtype=torch.float32) valid_action_id_length = ( vae.pos_id_len + vae.rot_id_len + vae.grip_id_len)# TODO: modify to your own downloaded normalizer path# download from http://ml.cs.tsinghua.edu.cn/~lingxuan/rdt2/umi_normalizer_wo_downsample_indentity_rot.ptnormalizer = LinearNormalizer.from_pretrained("umi_normalizer_wo_downsample_indentity_rot.pt") # result = batch_predict_action( model, processor, vae, normalizer, examples=[ { "obs": { # NOTE: following the setting of UMI, camera0_rgb for right arm, camera1_rgb for left arm "camera0_rgb": ..., # RGB image in np.ndarray of shape (1, 384, 384, 3) with dtype=np.uint8 "camera1_rgb": ..., # RGB image in np.ndarray of shape (1, 384, 384, 3) with dtype=np.uint8 }, "meta": { "num_camera": 2 } }, ..., # we support batch inference, so you can pass a list of examples ]， valid_action_id_length=valid_action_id_length, apply_jpeg_compression=True, # Since model is trained with mostly jpeg images, we suggest toggle this on for better formance instruction="Pick up the apple." # We suggest using Instruction in format "verb + object" with Capitalized First Letter and trailing period ) # get the predict action from example 0action_chunk = result["action_pred"][0] # torch.FloatTensor of shape (24, 20) with dtype=torch.float32# action_chunk (T, D) with T=24, D=20# T=24: our action_chunk predicts the future 0.8s in fps=30, i.e. 24 frames# D=20: following the setting of UMI, we predict the action for both arms from right to left# - [0-2]: RIGHT ARM end effector position in x, y, z (unit: m)# - [3-8]: RIGHT ARM end effector rotation in 6D rotation representation# - [9]: RIGHT ARM gripper width (unit: m)# - [10-12]: LEFT ARM end effector position in x, y, z (unit: m)# - [13-18]: LEFT ARM end effector rotation in 6D rotation representation# - [19]: LEFT ARM gripper width (unit: m) # rescale gripper width from [0, 0.088] to [0, 0.1]for robot_idx in range(2): action_chunk[:, robot_idx * 10 + 9] = action_chunk[:, robot_idx * 10 + 9] / 0.088 * 0.1For installation and fine-tuning instructions, please refer to the official GitHub repository.
Intended uses
Limitations
Safety & responsible use
| Symptom | Likely cause | Suggested fix |
|---|---|---|
| Drifting / unstable gripper widths | Scale mismatch | Apply LinearNormalizer; rescale widths ([0,0.088] → [0,0.1]). |
| Poor instruction following | Prompt format | Use “Verb + Object.” with capitalization + period. |
| No improvement after FT | OOD actions | Check RVQ bounds & reconstruction error; verify normalization. |
| Vision brittleness | JPEG gap | Enable --image_corruption; ensure 384×384 inputs. |
bibtex
@article{liu2026rdt2, title={RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization}, author={Liu, Songming and Li, Bangguo and Ma, Kai and Wu, Lingxuan and Tan, Hengkai and Ouyang, Xiao and Su, Hang and Zhu, Jun}, journal={arXiv preprint arXiv:2602.03310}, year={2026}}# Run under repository: https://github.com/thu-ml/RDT2 import torchfrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration from vqvae import MultiVQVAEfrom models.normalizer import LinearNormalizerfrom utils import batch_predict_action # assuming using gpu 0device = "cuda:0" processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")model = Qwen2_5_VLForConditionalGeneration.from_pretrained( "robotics-diffusion-transformer/RDT2-VQ" torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", device_map=device).eval()vae = MultiVQVAE.from_pretrained("robotics-diffusion-transformer/RVQActionTokenizer").eval()vae = vae.to(device=device, dtype=torch.float32) valid_action_id_length = ( vae.pos_id_len + vae.rot_id_len + vae.grip_id_len)# TODO: modify to your own downloaded normalizer path# download from http://ml.cs.tsinghua.edu.cn/~lingxuan/rdt2/umi_normalizer_wo_downsample_indentity_rot.ptnormalizer = LinearNormalizer.from_pretrained("umi_normalizer_wo_downsample_indentity_rot.pt") # result = batch_predict_action( model, processor, vae, normalizer, examples=[ { "obs": { # NOTE: following the setting of UMI, camera0_rgb for right arm, camera1_rgb for left arm "camera0_rgb": ..., # RGB image in np.ndarray of shape (1, 384, 384, 3) with dtype=np.uint8 "camera1_rgb": ..., # RGB image in np.ndarray of shape (1, 384, 384, 3) with dtype=np.uint8 }, "meta": { "num_camera": 2 } }, ..., # we support batch inference, so you can pass a list of examples ]， valid_action_id_length=valid_action_id_length, apply_jpeg_compression=True, # Since model is trained with mostly jpeg images, we suggest toggle this on for better formance instruction="Pick up the apple." # We suggest using Instruction in format "verb + object" with Capitalized First Letter and trailing period ) # get the predict action from example 0action_chunk = result["action_pred"][0] # torch.FloatTensor of shape (24, 20) with dtype=torch.float32# action_chunk (T, D) with T=24, D=20# T=24: our action_chunk predicts the future 0.8s in fps=30, i.e. 24 frames# D=20: following the setting of UMI, we predict the action for both arms from right to left# - [0-2]: RIGHT ARM end effector position in x, y, z (unit: m)# - [3-8]: RIGHT ARM end effector rotation in 6D rotation representation# - [9]: RIGHT ARM gripper width (unit: m)# - [10-12]: LEFT ARM end effector position in x, y, z (unit: m)# - [13-18]: LEFT ARM end effector rotation in 6D rotation representation# - [19]: LEFT ARM gripper width (unit: m) # rescale gripper width from [0, 0.088] to [0, 0.1]for robot_idx in range(2): action_chunk[:, robot_idx * 10 + 9] = action_chunk[:, robot_idx * 10 + 9] / 0.088 * 0.1@article{liu2026rdt2, title={RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization}, author={Liu, Songming and Li, Bangguo and Ma, Kai and Wu, Lingxuan and Tan, Hengkai and Ouyang, Xiao and Su, Hang and Zhu, Jun}, journal={arXiv preprint arXiv:2602.03310}, year={2026}}