Model Details
Table with columns: Attribute, Value| Attribute | Value |
|---|
| Base Model | Qwen3.5-VL-397B-A17B |
| Architecture | Qwen3.5 MoE (Mixture of Experts) |
| Total Parameters | 397B |
| Active Parameters | ~17B (10 of 512 experts per token) |
| Text Backbone | 60 layers, hidden=4096 |
| Vision Encoder | 27 layers, hidden=1152, patch=16 |
| Training | SFT on robot manipulation data |
| Max Context | 262,144 tokens |
| License | Apache 2.0 |
Capabilities
- Robotic Video VQA: Visual question answering on robot manipulation videos (multi-view supported)
- Fine-Grained Manipulation Annotation: Step-by-step captioning across 10 dimensions:
Table with columns: Dimension, What it captures| Dimension | What it captures |
|---|
| Action Sequence | Step-by-step execution order |
| Active Actor | Which arm / end-effector to use |
| Target Object | Object disambiguation |
| Initial Configuration | Starting state of objects and robot |
| Final Configuration | End state after manipulation |
| Contact & Approach | Where and how contact is made |
| Trajectory & Orientation | Motion path and tool orientation |
| Body Motion | Full-body or joint-level movement |
Benchmark Results
Evaluated on RoboFine-Bench, a comprehensive benchmark for robotic video understanding.
VQA (VLM4Robotics Benchmark)
Evaluated on 500 robot manipulation samples across 10 datasets.
Table with columns: Model, Overall, Gnd., Act., State| Model | Overall | Gnd. | | | Act. | | | | State | | |
|---|
| | AA | TO | IC | AS | C&A | T&O | BM |
Caption (VLM4Robotics Benchmark)
Fine-grained step decomposition of robot manipulation videos.
Table with columns: Model, Easy, Hard| Model | Easy | | | | Hard | | | |
|---|
| Overall | Cons. | Cov. | A-Hal. | Overall | Cons. | Cov. | A-Hal. |
| Qwen3-VL-Plus | 75.4 | 75.2 |
Settings: fps=4, max_frames=512, temperature=0.0, top_p=0.95, thinking=able(only RoboFine-VLM don't use thinking)
Quick Start
[!Important]
RoboFine-VLM is built on Qwen3.5-VL-397B-A17B. Inference efficiency and throughput vary significantly across frameworks.
We recommend using the latest framework versions to ensure optimal performance and compatibility.
Using RoboFine-VLM via the Chat Completions API (OpenAI SDK)
After serving the model with a framework like vLLM or SGLang, you can use the following code:
from openai import OpenAIimport httpx client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", http_client=httpx.Client(timeout=httpx.Timeout(600.0, connect=60.0)),) # Single-view video (frame URLs as video type)messages = [{"role": "user", "content": [ { "type": "video", "video": [ "https://your-bucket.oss.aliyuncs.com/frame_0000.jpg", "https://your-bucket.oss.aliyuncs.com/frame_0001.jpg", # ... more frame URLs ], "max_frames": 512, }, {"type": "text", "text": "Describe the robot manipulation in this video."},]}] response = client.chat.completions.create( model="RoboFine-VLM-opensource", messages=messages, temperature=0.0, top_p=0.95, max_tokens=32768, extra_body={"chat_template_kwargs": {"enable_thinking": False}},)print(response.choices[0].message.content)
Hardware Requirements
Table with columns: Precision, Model Weights, Minimum GPUs, Total VRAM| Precision | Model Weights | Minimum GPUs | Total VRAM |
|---|
| BF16 | ~752GB | 8× H200-141GB | 1128GB |
| FP8 | ~376GB | 8× A100/H100-80GB | 640GB |
- 8× H200-141GB: BF16 full precision, no quantization needed
- 8× A100/H100-80GB: Requires FP8 quantization (
--quantization fp8)
Training Data
Fine-tuned on FineVLA-Data: 47,159 human-verified trajectories with fine-grained instructions, generated by FineVLA-Tool and validated through human inspection.
Citation
@misc{FineVLA, title={FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies}, author={Xintong Hu and Xuhong Huang and Jinyu Zhang and Yutong Yao and Yuchong Sun and Qiuyue Wang and Mingsheng Li and Sicheng Xie and Yitao Liu and Junhao Chen and Yixuan Chen and Yingming Zheng and Shuai Bai and Tao Yu}, year={2026}, eprint={2605.27284}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2605.27284}, }
