Lineage
Qwen/Qwen3.5-4B -> YijiaFan/CaPtain-4B (CaPtain SFT + RL on LIBERO, RL round 300 = "ck300") -> ck300-r1113c (SFT on 555 native SO-101 twin rollouts, bf16; not released here) -> CaPtain-4B-SO101-Fast (this repo: W4A16 + W8A16 lm_head, trained MTP head, draft vocabulary)
SO-101 SFT (ck300-r1113c). CaPtain-4B, fine-tuned on 555 successful native SO-101 MuJoCo-twin rollouts (16,557 rows)
(CaPtain-SO101-SFT-Data) collected with GPT-6.x
teacher agents in CaPtain-Harness. Data harness: intro v5, prompt profile compact_v1, real perception only
(SAM 3 + Molmo2-8B + MoGe-2 monocular depth, no privileged state), 64k context with compaction keeping the last 16k
tokens, at most 12 images, 100 model requests per episode, T = 1.0. SFT: 8x H100, 2 epochs, 1,228 steps, peak LR 1e-5
with a cosine schedule, frozen vision tower.
Files
Table with columns: file, size, what| file | size | what |
|---|
model.safetensors | 4,432,275,784 B | policy weights, compressed-tensors pack-quantized: W4A16 (int4, group 128, symmetric, RTN) for all decoder Linear layers, W8A16 (int8, group 128) for lm_head; vision tower, embeddings, norms and GDN in_proj_a/b stay bf16 |
mtp.safetensors | 241,200,880 B | MTP drafter head (1 layer, bf16), "run13" (see below); ignored by the quantization config |
config.json | 10,855 B | Qwen3_5ForConditionalGeneration, mtp_num_hidden_layers = 1, quantization_config (compressed-tensors) |
recipe.yaml | 332 B | the llm-compressor recipe used for quantization |
draft_vocab/draft_vocab16384_r13.safetensors | 84,017,312 B | drafter vocabulary subset: ids int64 [16384] + weight bf16 [16384, 2560] (rows of lm_head) |
| tokenizer / processor / chat template / generation config | | unchanged from Qwen3.5-4B / CaPtain-4B |
SHA256SUMS | | sha256sum -c SHA256SUMS |
The draft vocabulary lives in a subfolder on purpose: vLLM loads every top-level *.safetensors of a model directory.
How it was built
Quantization (round-to-nearest, no calibration data), CaPtain branch so101-fast:
cd robots/soarm && source env.shpython tools/quantize_policy.py <ck300-r1113c bf16 dir> <out dir> --scheme W4A16 --lm-head W8A16
The resulting recipe.yaml: W4A16 on every Linear except the vision tower and the GDN in_proj_a / in_proj_b
projections, W8A16 on re:.*lm_head$.
MTP drafter head ("run13"). Qwen3.5-4B ships a one-layer MTP head; vLLM uses it for speculative decoding. run13 was
initialised from an earlier SO-101 drafter (run3, fine-tuned for an earlier SO-101 policy from the public Qwen3.5-4B MTP head) and trained on
ck300-r1113c's own recorded outputs on the twin (24 episodes, 68 request bodies, 3 epochs, T = 1.0, 5 unrolled draft
steps) with robots/soarm/tools/drafter/train_mtp.py; the model directory was assembled with
tools/drafter/make_model_dir.py.
Draft vocabulary. robots/soarm/tools/vllm_patches/build_draft_vocab.py --k 16384: the special tokens, every token
the policy emitted in the same recorded outputs, then the lowest BPE ids up to 16,384 rows (362 of 78k held-out output
tokens fall outside). Lossless: the target model still verifies every draft token.
Serving (vLLM 0.29, 2x RTX 4090)
The serving patches live in CaPtain branch so101-fast, robots/soarm/tools/vllm_patches (loaded as sitecustomize
into the vLLM process only):
Table with columns: switch, effect, lossless| switch | effect | lossless |
|---|
on by default with the patches on PYTHONPATH | split-KV Triton attention for speculative verify steps; FlashAttention-2 for prefill on the TRITON_ATTN KV layout | yes |
SOARM_DRAFT_VOCAB=<file> | the MTP drafter scores only the 16k-row vocabulary subset | yes |
SOARM_VLLM_PATCH_NO_EAGLE_DROP=1 | keep the full prefix-cache hit with the MTP drafter | yes (target outputs unchanged) |
SOARM_VLLM_PATCH_FP8_PREFILL=1 | W4 layers keep an FP8 copy; prefill chunks run as FP8 GEMMs, decode stays W4 Marlin | no (prefill only); needs VLLM_DISABLE_COMPILE_CACHE=1 |
git clone -b so101-fast --recurse-submodules https://github.com/waltstephen/CaPtain.gitcd CaPtain/robots/soarm && source env.sh # settings: env.example; VLLM_VENV = a vLLM 0.29 venvhf download YijiaFan/CaPtain-4B-SO101-Fast --local-dir "$MODEL_DIR" # policy on GPU0, port 8700 (SPEC_VERIFY_PATCH=1 puts tools/vllm_patches on the vLLM PYTHONPATH)SPEC_VERIFY_PATCH=1 SOARM_DRAFT_VOCAB="$MODEL_DIR/draft_vocab/draft_vocab16384_r13.safetensors" \SOARM_VLLM_PATCH_NO_EAGLE_DROP=1 SOARM_VLLM_PATCH_FP8_PREFILL=1 VLLM_DISABLE_COMPILE_CACHE=1 POLICY_MEM=0.62 \POLICY_ARGS='--attention-backend TRITON_ATTN --max-num-batched-tokens 8192 --speculative-config {"method":"mtp","num_speculative_tokens":5,"draft_sample_method":"probabilistic","attention_backend":"TRITON_ATTN"}' \ bash tools/serve_policy.sh "$MODEL_DIR" 8700 0 captain-4b # perception: SAM 3 on GPU0 :8114 (8-image encoder cache), Molmo2-8B FP8 on GPU1 :8122, MoGe-2 on GPU1 :8130 (level 0)SOARM_PROFILE=fast_v5 bash tools/serve_perception.sh sam3 0MOLMO_MEM=0.62 bash tools/serve_perception.sh molmo 1bash tools/serve_perception.sh depth 1python tools/warm_services.py --body <a recorded first request body> # must print no error
tools/serve_policy.sh wraps this plain vLLM command:
PYTHONPATH=<CaPtain>/robots/soarm/tools/vllm_patches:$PYTHONPATH \SOARM_DRAFT_VOCAB=... SOARM_VLLM_PATCH_NO_EAGLE_DROP=1 SOARM_VLLM_PATCH_FP8_PREFILL=1 VLLM_DISABLE_COMPILE_CACHE=1 \VLLM_USE_FLASHINFER_SAMPLER=0 CUDA_VISIBLE_DEVICES=0 \vllm serve "$MODEL_DIR" --served-model-name captain-4b --dtype bfloat16 --gpu-memory-utilization 0.62 \ --max-model-len 65536 --enable-prefix-caching --limit-mm-per-prompt '{"image":12}' --max-num-seqs 8 \ --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \ --attention-backend TRITON_ATTN --max-num-batched-tokens 8192 \ --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_sample_method":"probabilistic","attention_backend":"TRITON_ATTN"}' \ --port 8700
The model also loads without the patches (stock vLLM 0.29, MTP speculative decoding from mtp.safetensors); it is then
slower and the draft vocabulary is unused.
GPU layout (measured, 2x RTX 4090 24 GB):
Table with columns: GPU, services, memory| GPU | services | memory |
|---|
| GPU0 | policy (POLICY_MEM 0.62: W4 weights + W8 lm_head + MTP layer 4.7 GB, FP8 prefill copies 4.3 GB, 75k-token KV cache) + SAM 3 (7.7 GB, 8-image encoder cache) | 23.1 GB |
| GPU1 | Molmo2-8B FP8 (MOLMO_MEM 0.62, 13.8 GB) + MoGe-2 depth (resolution level 0, 3.8 GB) | 17.6 GB |
Do not raise the SAM 3 encoder cache above 8 images next to the policy (SAM 3 runs out of memory).
Prompt profile must match training: intro v5 + compact_v1 TASK.md, 100 requests, 12 images, keep 16k tokens after
compaction, T = 1.0 (CaPtain robots/soarm/configs, speed profile speed_fast_v5s.json for the twin).
Results (digital twin, not real-arm success rates)
Strict success = the hidden goal predicate holds and the agent called finish() itself. All rows use realistic
perception (RGB + MoGe-2 depth + SAM 3 + Molmo2), the training prompt profile and one episode at a time.
Policy quality (bf16, data harness):
Table with columns: eval, CaPtain-4B (ck300), ck300-r1113c| eval | CaPtain-4B (ck300) | ck300-r1113c |
|---|
| div3 seen task families, strict | 4/169 | 45/169 |
| held-out cube task families, strict | 0/40 | 10/40 |
Fast stack (this repo) on pick_lift / pick_place_mat / put_in_container, 45 episodes (15 seeds x 3 families):
Table with columns: config, strict, s/request, real-arm timeline (model / motion), requests / episode, strict-success median| config | strict | s/request, real-arm timeline (model / motion) | requests / episode | strict-success median |
|---|
| bf16, release serving and release motion, 22-episode subset | 14/22 | 5.48 (3.06 / 2.08) | 39.9 | 94.3 s |
| this repo, same 22 family/seed pairs | 16/22 | 1.21 (0.74 / -) | - | 31.5 s |
| W4 stack with the public Qwen3.5-4B MTP head | 28/45 | 1.55 (0.93 / 0.41) | 54.7 | 37.5 s |
| this repo (run13 head + r13 draft vocabulary) | 27/45 (10 / 9 / 8 of 15) | 1.455 (0.81 / 0.45) | 51.3 | 32.3 s (12 of 27 under 30 s) |
Per request: model 0.814 s (prefill 0.299 + decode 0.469 = 195 tokens x 2.41 ms) + motion 0.447 + perception 0.071 +
observe 0.075 s; vLLM prefix-cache hit 0.89; 0 perception errors in 929 requests.
Drafter A/B (6 held-out episodes, 437 recorded requests replayed, s/request normalised to 190 output tokens):
Table with columns: drafter, s/request, accepted tokens / step, decode ms/token| drafter | s/request | accepted tokens / step | decode ms/token |
|---|
| public Qwen3.5-4B MTP head | 0.948 | 3.04 | 2.94 |
| run3 head + new draft vocabulary | 0.906 | 3.29 | 2.72 |
| run13 head + new draft vocabulary (this repo) | 0.843 | 3.78 | 2.37 |
Limitations
- Twin numbers. Every success rate and timing above was measured on the SO-101 MuJoCo digital twin with realistic
perception. The model was deployed on a real SO-101 arm, but no real-arm success rate is reported here. Expect slower
wall time on hardware (real camera frames, release motion speeds).
- Real arm: use a motion-safe profile. The twin speed profile (
speed_fast_v5*.json) contains motion speed-ups that
were validated only in the twin (motion_speed_scale 3, above-cap step_profile, gripper_settle_min_steps). For a
real arm use configs/speed_fast_v5_real.json (release motion speeds: motion_speed_scale 1, 0.05 m/s, 0.5 rad/s, no
step profile), enforce joint soft limits on every command, keep an e-stop at hand, and start slowly with an operator
watching. All model-side speed-ups are motion-independent.
- Embodiment- and harness-specific. Trained only on SO-101 twin rollouts with the
compact_v1 prompt profile; it
needs CaPtain-Harness, the SO-101 robot API, the perception services and a calibrated camera / kinematics setup.
Outside the harness it behaves like an ordinary small VLM.
- Quantization. W4A16 RTN (no calibration) and the FP8 prefill patch are lossy; on the twin the success rate was not
significantly different from bf16, but small differences cannot be ruled out. The drafter, draft vocabulary and the
other serving patches are lossless.
- Scope. Simple tabletop tasks (pick-and-lift, pick-and-place on a mat, put in a container, cube families); success
varies strongly with seed and scene. Research artifact, not validated for safety-critical or unsupervised use.
License
Apache-2.0, the license of the base model Qwen3.5-4B (and of CaPtain-4B). When you run the model together with other
components (SAM 3 under the SAM License, Molmo2, MoGe-2, MuJoCo, simulators), their licenses also apply; see
THIRD_PARTY_LICENSES.md in the CaPtain repositories.
Citation
@article{fan2026captain, title = {CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents}, author = {Fan, Yijia and He, Xilin and Jiang, Yilei and Wu, Dongming and Wang, Jiaxu and Yue, Xiangyu}, journal = {arXiv preprint arXiv:2610.40799}, year = {2026}}
Contact: Xiangyu Yue (xyyue@ie.cuhk.edu.hk), MMLab, The Chinese University of Hong Kong.