Why the distilled model at all?
The distill (14k Claude Opus 4.6 reasoning traces, methodology by Jackrong) fixes the most painful production behaviour of the base reasoning model: burning the entire max_tokens budget on the thinking chain and returning content: null. Our production A/B (same NVFP4 precision, same TP2 setup):
Table with columns: Task, Official Qwen3.6-27B-NVFP4, This model| Task | Official Qwen3.6-27B-NVFP4 | This model |
|---|
| Chinese copywriting | 1671 tok / 15.2s | 688 tok / 6.4s |
| Coding task | ❌ burned 2048-tok budget, empty answer | ✅ complete, 722 tok |
| Math word problem | ✅ correct, 1624 tok | ✅ correct, 352 tok |
| Image understanding | ❌ burned budget, no answer | ✅ correct, 138 tok |
| Long-doc summary | 656 tok | 241 tok |
| Decode speed (2×RTX 5090, TP2) | 108.7 tok/s | 109.3 tok/s (parity) |
Same speed, ~1/2–1/3 the tokens, zero empty answers in our suite. Long-context verified to 240K tokens input on 2×32 GB.
Usage (vLLM ≥ 0.24)
vllm serve <this-repo> \
--tensor-parallel-size 2 \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser qwen3 \
--trust-remote-code
Single 32 GB GPU also works (reduce --max-model-len, add --enforce-eager if CUDA-graph capture OOMs).
⚠️ Two settings that are NOT optional
-
--tool-call-parser hermes — the distilled model emits <tool_call>{...}</tool_call> (hermes format). With qwen3_coder parser tool calls silently degrade to plain text.
-
Don't swap the bundled tokenizer/chat template. This repo intentionally mixes:
tokenizer.json / tokenizer_config.json from the official base (defines the vision special tokens — with the upstream distill tokenizer, video inputs crash Qwen3VLProcessor);chat_template.jinja from the distill (drives the hermes tool format — with the official template the model stops emitting tool calls).
Each mismatch breaks exactly one capability; we hit both so you don't have to.
How it was made
- Quantizer: llm-compressor (
QuantizationModifier, scheme="NVFP4", ignore=["lm_head"]), sequential layer-wise calibration on a single GPU with the model resident in CPU RAM. NVIDIA ModelOpt was tried first and failed on this hybrid GDN architecture (both latest and pinned versions), llm-compressor handled it cleanly and auto-excluded the quantization-sensitive linear_attn (GDN) layers — same treatment as NVIDIA's official NVFP4 build.
- Calibration: 512 mixed-language samples — 256 English news (cnn_dailymail) + 256 Chinese instruction pairs (alpaca-zh), max_length 1024.
- Multimodal graft: llm-compressor only saves the text model (which vLLM refuses to load for this architecture — only the wrapper arch is registered). We grafted the 333 BF16 vision-tower tensors from the upstream checkpoint back into the safetensors, rebuilt the wrapper-format
config.json (visual modules listed explicitly in the quantization ignore — regex patterns break vLLM weight loading with a KeyError), and restored the processor config set.
Lineage & licensing
Qwen/Qwen3.6-27B (Apache-2.0) → rico03/Qwen3.6-27B-Claude-Opus-Reasoning-Distilled (Apache-2.0, 14k Claude Opus 4.6 reasoning traces — see upstream repo for training-data details) → this NVFP4 quantization (Apache-2.0). Vision tower and tokenizer inherit from the official base checkpoint.