YijiaFan

CaPtain-4B-SFT

Deploy Dedicated

README

License: apache-2.0

What the model is

A code-writing agent, not a VLA. The model never outputs actions directly. At every request it writes Python for the harness's single python tool: a persistent, sandboxed Python 3.11 session. The robot API in that session has 22 low-level capabilities (perception, geometric measurement, IK-level guarded motion) plus finish(), and no grasp or place macros.
What it sees: camera images, its own code and output, and one-line receipts of its robot calls.
What it never sees: object poses and the success predicate.
Perception services. SAM 3 segmentation and Molmo2-8B pointing run behind the harness tools.
Format. bf16 safetensors with the Qwen3_5ForConditionalGeneration architecture and an untied lm_head. The tokenizer, processor and chat template are the same as Qwen3.5-4B. The vision tower is unchanged from the base model: it was frozen during SFT.

How to use

Use it exactly like CaPtain-4B, inside CaPtain-Harness.

bash

vllm serve YijiaFan/CaPtain-4B-SFT --served-model-name captain-4b --dtype bfloat16 --max-model-len 81920 \  --enable-prefix-caching --limit-mm-per-prompt '{"image":12}' --max-num-seqs 16 \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml --port 8600   # vLLM 0.29 # in a CaPtain-Harness checkout (https://github.com/waltstephen/CaPtain-Harness, installed as in INSTALL.md):services/start_perception.sh                                    # SAM 3 :8114 + Molmo2-8B :8122export POLICY_URL=http://127.0.0.1:8600/v1 STUDENT_PYTHON=~/envs/student-py311/bin/python EGL_DEVICE=0eval/run_episode.sh libero_spatial 0 11 runs/demo

The paper's evaluation protocol:

sampling T = 1.0, top_p = 1.0, thinking on;
budgets per episode: 60 model requests, 1,080 s and 150 physical actions;
context: a 64k-token window with compaction.

To start RL from this checkpoint, see rl/README.md in the CaPtain repository.

Prompt format

The model was trained on the harness's exact request template:

the system prompt and environment context;
TASK.md, inlined in the system prompt;
the python tool schema (python-runtime/v1);
receipts, compaction notes and image placement.

The released SFT data show this template row by row. The model uses the standard Qwen3.5 chat template, with <think> reasoning and XML tool calls. Run it through CaPtain-Harness rather than with hand-written prompts, and do not change model-visible text when evaluating it.

Results

Success rates (%) in simulation, host / strict. Strict success also requires that the agent called finish() itself. The LIBERO row uses held-out official initial states 11–16 of the 40 tasks of the four standard suites.

Table with columns: Benchmark, Episodes, Qwen3.5-4B, CaPtain-4B-SFT, CaPtain-4B (after RL)
Benchmark	Episodes	Qwen3.5-4B	CaPtain-4B-SFT	CaPtain-4B (after RL)
LIBERO, official held-out states 11–16	240	0.0 / 0.0	55.4 / 52.1	73.8 / 67.5
LIBERO-PRO Pos+Task, 6-cell mean	600	0.3 / 0.2	59.5 / 56.7	78.5 / 71.5
LIBERO-PRO Pos+Task, 8-cell mean	800	0.2 / 0.1	49.2 / 45.1	68.6 / 58.9
LIBERO-Plus, 7 categories	700	0.0 / 0.0	48.6 / 44.6	66.1 / 59.0
LIBERO-PRO, 4 categories	160	0.0 / 0.0	48.8 / 48.1	68.8 / 60.6
LIBERO-MAX Dynamic	80	0.0 / 0.0	40.0 / 40.0	61.2 / 50.0
LIBERO-MAX Base	80	0.0 / 0.0	51.2 / 45.0	70.0 / 63.8

The zero-shot transfer results of the paper (RoboTwin 2.0 22.6 on the official Hard protocol, SimplerEnv WidowX 83.3, CaP-Bench 75.0, RoboDojo Score 1.7) were measured with CaPtain-4B, not with this checkpoint.

Training

Base: Qwen3.5-4B, with the lm_head untied (sft/scripts/untie_base.py).
Stage 1: an earlier agent mix, trained for 1,700 steps. This checkpoint initialises stage 2.
Stage 2 (this model): CaPtain-SFT-Data. The stage-2 mix consists of:
- synthetic agent trajectories collected in CaPtain-Harness, generated with frontier models, plus successful episodes of an earlier RL run;
- visual QA at loss weight 0.5.
Supervision: only assistant turns. A text-hint mask (p = 0.2) replaces earlier mark_point pixel coordinates with <masked>, so the model has to read the image.
Optimisation: full fine-tuning of the language model with the vision tower frozen. Peak LR 1e-5, 30 warmup steps, cosine schedule. One row per rank on 16 ranks, with a constant global per-token loss normaliser. DeepSpeed ZeRO-3, bf16, gradient clipping 1.0.
Length: 2,435 steps (1.5 epochs).

The trainer, launch script and exact configuration are in the CaPtain repository (sft/).

Limitations

Simulation-trained. The SFT trajectories are simulated episodes, almost all on LIBERO. The model was not trained on real-robot data.
Weaker than the RL model. It is clearly worse than CaPtain-4B on every benchmark above. Missed finishes and long-horizon or articulated tasks are its main failures.
Mid-training checkpoints are much worse. If you retrain SFT, judge the run only at the end of the cosine schedule.
Inputs. It depends on the harness's perception services (SAM 3, Molmo2-8B), depth, camera calibration, and the exact prompt and tool format.
One arm at a time. Two-arm coordination is not supported.
Research artifact. Not validated for safety-critical use or for unsupervised use on physical robots.

License

Apache-2.0, the license of the base model Qwen3.5-4B. The SFT data have their own terms; see the dataset card. Components you run with the model (LIBERO, SAM 3, Molmo2, simulators) keep their own licenses.

Citation

bibtex

@article{fan2026captain,  title   = {CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents},  author  = {Fan, Yijia and He, Xilin and Jiang, Yilei and Wu, Dongming and Wang, Jiaxu and Yue, Xiangyu},  journal = {arXiv preprint},  year    = {2026}}

Contact: Xiangyu Yue (xyyue@ie.cuhk.edu.hk), MMLab, The Chinese University of Hong Kong.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

YijiaFan

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality