Scheme
Same scheme and the same module coverage as the official
Qwen/Qwen3.8-27B-FP8 release, so it runs on
the same well-trodden vLLM kernel path.
Table | |
|---|
| Weights | FP8 e4m3, static, 128×128 blocks |
| Activations | FP8 e4m3, dynamic, per-token groups of 128 |
| Format | compressed-tensors (float-quantized) |
| Calibration | none (data-free) |
400 Linear modules are quantized:
q_proj/k_proj/v_proj/o_proj of the 16 full-attention layers
in_proj_qkv/in_proj_z/out_proj of the 48 gated-delta-net (linear-attention) layers
gate_proj/up_proj/down_proj of all 64 MLPs
Kept in BF16: in_proj_a/in_proj_b, conv1d, all norms, embed_tokens, lm_head, the
whole vision tower, and the MTP head (mtp.*, so MTP speculative decoding works).
Every non-quantized tensor is byte-identical to upstream, and so are the tokenizer,
processor, generation config and chat template. This was checked tensor by tensor, and it
needed a fix: llm-compressor 0.13 converts Qwen3.5's offset-style RMSNorm weights
(1 + w) for its pipeline and converts them back in BF16, which rounds every
input_layernorm/post_attention_layernorm/q_norm/k_norm weight to a ~0.004 grid
(up to ~7% relative error on individual values). The shipped checkpoint takes those tensors
straight from the source instead.
Dequantized-vs-source relative error of the quantized weights is ~2.7%, as expected for
FP8 e4m3.
Serving with vLLM
vllm serve batsclamp/Qwopus3.8-27B-Flash-V2-FP8 \ --max-model-len 262144 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --chat-template <snapshot>/chat_template_qwen_xml.jinja \ --tool-call-parser qwen3_coder \ --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20}'
Which chat template
The repo carries two templates that differ only in the tool-call format:
Table with columns: File, Tool-call format, vLLM parser| File | Tool-call format | vLLM parser |
|---|
chat_template.jinja (upstream, unchanged) | JSON inside <tool_call> | hermes |
chat_template_qwen_xml.jinja (= stock Qwen3.8-27B template) | XML <function=…><parameter=…> | qwen3_coder |
The parser must match the template. chat_template.jinja is upstream's file byte for byte and
is what vLLM picks up by default; we recommend overriding it with the XML one. A/B on the
same weights, same server, reasoning_effort: xhigh, temperature 1.0:
Table with columns: upstream JSON + hermes, Qwen XML + qwen3_coder | upstream JSON + hermes | Qwen XML + qwen3_coder |
|---|
| tool-eval-bench v2.0.6, 69 scenarios | 84 (18 not fully passed) | 91 (10 not fully passed) |
| Python file written through a tool call compiles, temp 1.0 (n=10) | 7/10 | 9/10 |
| same, temp 0.3 (upstream's coding recommendation) (n=10) | 3/10 | 10/10 |
With the JSON format the model has to emit code as an escaped JSON string, and it often
collapses Python indentation there (if x:\n raise …). In raw XML parameters it does not.
Single runs at n=10 are noisy, but both measurements point the same way.
The chat template (upstream's and Qwen's alike) accepts only reasoning_effort values
xhigh, medium and low, and raises on anything else (high, max, minimal), which
an OpenAI-compatible server turns into HTTP 400. If your client sends other values, map
them in a small Jinja prelude before the template.
For coding and agentic work the upstream card recommends reasoning_effort: xhigh with
temperature: 0.3 supplied by the caller.
Measured on one DGX Spark (GB10)
vLLM 0.32.0rc1 (nightly of 2026-10-16), --max-model-len 262144, MTP k=3,
--gpu-memory-utilization 0.55, kv-cache-dtype auto:
Table | |
|---|
| Weights resident | 29.1 GiB |
| KV cache | 29.2 GiB, ~424k tokens (1.62× concurrency at 262k) |
| Decode, single stream, no thinking | 14.8 tok/s |
| TTFT, short prompt | 185 ms |
| MTP | 1.58 accepted tokens per draft |
| Kernel | CutlassFp8BlockScaledMMKernel |
The reasoning effort levels behave as upstream: with xhigh a simple factual question
used ~290 characters of reasoning.
Caveats
- Quantization was verified structurally (tensor-by-tensor against the source) and by
generation and tool-calling tests on vLLM; no perplexity or benchmark-suite numbers are
claimed beyond what is listed above.
- All model behaviour, strengths and limitations are upstream's; see the
original model card.