EmpathicRobotics

vla-1.7b-qwen3-v7

README

License: apache-2.0

Key facts

Table

Architecture	Qwen3 (28 layers, hidden 2048, intermediate 6144, 16 attn heads / 8 KV heads (GQA), qk-layernorm, RoPE θ=1e6, tied embeddings)
Parameters	1.97B (including embeddings for 274,688 padded vocab)
Vocab size	274,688 padded (same tokenizer as v6 — no vocab expansion needed; `<think>`/`</think>`/`<\|im_start\|>`/`<\|im_end\|>`/`<\|endoftext\|>` were already atomic special tokens in the base Qwen3 tokenizer)
Tokenizer	EmpathicRobotics/tokenizer-vla-qwen3-v2
Training data	21.34B tokens across the same 6 sources as v6, reformatted (see Data mix below)
Training	2,544 iters (1 epoch), 64 nodes × 4 GH200 GPUs, global batch 1024, seq len 8192, micro batch size 2
Final loss	Train: 2.609 (iter 2500), Val: 2.6153 (PPL 13.67), Test: 2.5738 (PPL 13.12)
Precision	bf16
Context length	8,192 tokens

What this model does

Given a text prompt (activity description, image seed2 block, or partial modality sequence), the model generates an interleaved multimodal token sequence spanning 6 categories it was trained on:

markdown
<think> ... </think>                   # structural turn-boundary marker, usually empty
<seed2_N> ...                          # 1 FPS semantic image/video keyframes (vocab 8192)
<cosmos_N> ... </cosmos>               # 8-frame spatial video tokens (vocab 64000)
<listen> <snac_N> ... </listen>        # SNAC audio codec tokens, "heard" role
<speak> <snac_N> ... </speak>          # SNAC audio codec tokens, model-generated "spoken" role
<caption> ... </caption>               # inline visual caption text
<agent> <fps_30> <pelvis> ... </agent> # 3D human pose, 17 H36M joints

Prompt shape (FineVideo example, from investigations/format_standardization/reformat_finevideo.py):

markdown
USER: Continue this video activity titled "Morning stretch". Context: A person raises
both arms above their head. ASSISTANT:
<think>
</think>
<caption> ... </caption> <seed2_N> ... <agent> ... </agent> ...

Progress vs. v6 — sampling matters more than decoding strategy might suggest

Direct comparison run (same eval harness, same 5-prompt suite structure, multi-seed where noted) against v6 and v2:

Autonomous full-chain generation from a bare text header (no modality primed — the hardest test in the suite: does the model spontaneously open <think>, then <caption>, <seed2>, and eventually a decodable <agent> block on its own?) — v7 succeeded with a valid decoded pose in 4/4 sampled seeds tested. v6 and v2 both failed in 0/4 combined runs (greedy and sampled, seed 42) — decoding an all-zero pose every time. This is the clearest signal so far that the format-standardization changed something real, not just cosmetic, about the model's ability to self-initiate a full modality chain and close it with <\|im_end\|>.
Greedy decoding (no sampling) is not usable on this checkpoint — without repetition_penalty, generation collapses into token-repetition loops (a single token repeated up to the full max_new_tokens budget, or a <cosmos> chunk repeating the same handful of ids for hundreds of tokens). This is not unique to v7 (v6 and v2 show a related-but-different failure under greedy — technically more varied tokens, but still no valid decodable output on the hardest test above) — but it means this model must always be sampled (do_sample=True, temperature≈0.8, , ) for usable output. The bundled demo (Colab notebook in this repo) samples by default.

Known limitations

Instruction-following / content-accuracy gap, unchanged since v2: even when the model correctly opens <caption> and closes it cleanly, the actual described content is frequently wrong on genuinely novel prompts (e.g. describing unrelated objects/scenes). Format-standardization measurably improved structural autonomy (see above) but was never expected to fix this — it's a data-composition problem (most non-video sources are media→text, i.e. understanding/captioning direction, not text→media generation), flagged as a concrete target for the project's planned VLA-Instruct SFT stage.
Greedy decoding is unreliable (see above) — always sample.
Probabilistic, not universal, on the "from-scratch" test: even under sampling, 1 of 4 seeds tested failed the hardest "generate an agent block from just a seed2 prime, no agent context at all" sub-test (decoded an all-zero pose) — treat the from-scratch full-chain result as "usually works, not guaranteed every draw," consistent with the "modality drift is probabilistic" pattern documented across the whole v2→v7 lineage.
avc_lm tokens are essentially unused — discarded at the data-flatten stage before training, so the model rarely if ever produces them.
seed2→image reconstruction is generative, not a deterministic round-trip (see tools/decode/decode_seed2.py — conditions a diffusion img2img pipeline on the token embeddings; expect run-to-run pixel variation for the same input tokens, and note the diffusion decode step itself (default 20 inference steps) is the slowest single decode step in the demo by design, not a bug).

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "EmpathicRobotics/vla-1.7b-qwen3-v7",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("EmpathicRobotics/vla-1.7b-qwen3-v7")

prompt = (
    'USER: Continue this video activity titled "Morning stretch". Context: A person '
    "raises both arms above their head. ASSISTANT:\n<think>\n</think>\n"
)
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    input_ids, max_new_tokens=1200,
    do_sample=True, temperature=0.8, top_p=0.9, repetition_penalty=1.3,
)
print(tokenizer.decode(output[0]))

Always sample (do_sample=True + repetition_penalty>1.0) — greedy decoding on this checkpoint reliably collapses into repetition loops on generations longer than a couple hundred tokens (see Progress vs. v6 above).

Encoding real media into tokens (so you can actually prompt the model)

The ## Usage prompt above uses pre-picked token ids as a demo. To send the model real media -- e.g. "here's a photo, continue the scene" or "here's a real motion clip, keep going" -- encode it first with the 4 encoders below. Bundled in this repo the same way as the decoders (tools/encode/), no separate git clone needed.

bash
# Image -> <seed2_N> tokens (32 ids, auto-downloads the Q-Former checkpoint
# from ontocord/seed2 if not cached locally)
python tools/encode/encode_seed2.py --image photo.jpg

# 8 video frames -> <cosmos_N> tokens (200 ids -- window=8/square-crop
# convention; the window=24/aspect-preserving convention (896 tokens) was NOT
# used for this model)
python tools/encode/encode_cosmos.py --frames f0.png f1.png f2.png f3.png f4.png f5.png f6.png f7.png

# Audio/video file -> <snac_N> tokens, wrapped in <listen> (this model's
# "heard" convention)
python tools/encode/encode_snac.py --input clip.wav

# Real 3D pose (8 frames x 17 joints x xyz, metres, root-centred) -> <agent>
# tokens -- for "give the model a real motion capture / pose-pipeline output,
# have it continue"
python tools/encode/encode_agent.py --input pose.npy   # shape (8, 17, 3)

Splice the printed token block into your prompt (after ASSISTANT:\n<think>\n</think>\n) the same way the ## Usage example does, then call model.generate() as shown there.

Decoding generated tokens back to media

The decoder scripts + their vendored dependencies are bundled directly in this repo (tools/) -- one snapshot_download gets everything, no separate git clone needed.

bash
python -c "
from huggingface_hub import snapshot_download
snapshot_download('EmpathicRobotics/vla-1.7b-qwen3-v7', allow_patterns=['tools/*', 'tools/**/*'])
"
pip install scipy numpy torch torchvision imageio-ffmpeg soundfile snac huggingface_hub
cd <snapshot-download-cache-dir-printed-above>

Agent tokens -> 3D pose (pure Python, no extra downloads):

bash
python tools/eval/decode_agent_tokens.py --input generated_tokens.txt --output poses.json

Cosmos tokens -> video (auto-downloads the ~350MB decoder checkpoint from nvidia/Cosmos-Tokenizer-DV8x16x16 on first run):

bash
python tools/decode/decode_cosmos.py --tokens 58345,57843,... --output out.mp4
# this model's cosmos chunks are exactly 200 raw ids each (8 frames, 160x160,
# square-cropped) -- the window=24/aspect-preserving convention (896 tokens)
# does NOT apply to this model.

SNAC tokens -> audio (auto-downloads hubertsiuzdak/snac_24khz from HF):

bash
python tools/decode/decode_snac.py --tokens 130911,134940,... --format listen --output out.wav
# use --format listen for <listen>-wrapped tokens (input/"heard" role) or
# --format speak for <speak>-wrapped tokens (model-generated "spoken" role).

Seed2 tokens -> image (auto-downloads the ~2.6GB Q-Former checkpoint from the tokenizer's own public repo, ontocord/seed2, plus a ~5GB diffusion img2img pipeline on first run -- this one is a generative reconstruction, not a deterministic decode, so expect run-to-run and prompt-to-prompt variation in the exact pixels even for the same tokens):

bash
python tools/decode/decode_seed2.py --tokens 6750,680,2472,... --output out.png
# exactly 32 raw ids per image (Seed2Tokenizer's fixed Q-former query length)

Training details

Loss curve

Table with columns: Iter, Loss
Iter	Loss
50	8.214
500	3.998
1000	3.042
1500	2.830
2000	2.718
2500	2.609
2544 (val)	2.6153 (PPL 13.67)
2544 (test)	2.5738 (PPL 13.12)

Config

Batch: GBS 1024, seq_len 8192, micro_batch_size 2 → 21.34B tokens trained (1 epoch of the w8_new format-standardized corpus)
Infrastructure: 64 nodes × 4 GH200 GPUs (256 total), ~302 TFLOP/s/GPU, ~20,650 tok/s/GPU
Framework: Megatron-LM via oellm-autoexp
Runtime: 1h27m wall clock (job 1137346)

Data mix (same 6 sources as v6, reformatted -- 2 sources' real token counts changed after bug fixes)

Table with columns: Source, Tokens, % of total, Notes
Source	Tokens	% of total	Notes
FineVideo-VLA (window=8 rebuild)	10.93B	51.2%	Unchanged from v6 — flagship video+pose branch
MixtureVitae-Omni (mv_omni)	7.89B	37.0%	+0.04B vs v6's pre-fix count — the reformat bug fixes moved WHERE text landed (USER vs ASSISTANT), not how much content existed
OmniVideo-100K	1.98B	9.3%	Unchanged token count from v6 — the reformat bug fix moved the split point, ~5 tokens differ across the whole corpus
Harmony4D	0.32B

Citation

bibtex
@misc{{empathicrobotics2026vlaqwen3v7,
  title={{VLA 1.7B Qwen3 v7: Format-Standardized Prompt Wire Format Across 6 Multimodal Sources}},
  author={{EmpathicRobotics}},
  year={{2026}},
  url={{https://huggingface.co/EmpathicRobotics/vla-1.7b-qwen3-v7}}
}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider