GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
dbrasdasilva
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
dbrasdasilva
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| Component | Precision |
|---|---|
| Vision tower (333 tensors) | BF16 — never quantized |
| MTP head (15 tensors) | BF16 — grafted from the base after export, so speculative decoding works |
lm_head | BF16 — see the note below, this is a requirement |
| Attention projections | FP8 E4M3 |
linear_attn norms, in_proj_a/b, conv1d | BF16 |
| MLP | FP8 E4M3 for all 64 layers |
| KV cache | FP8 E4M3 with calibrated k_scale/v_scale (16 each) |
Size on disk: 30 GB.
lm_head is left unquantizedNot because vLLM cannot load a quantized one — nvidia/Qwen3.8-27B-NVFP4 ships an
NVFP4 lm_head and loads fine. It is left in BF16 here for two reasons: it is a small
fraction of the weights, and it keeps quantization_config.ignore honest.
If you quantize this architecture yourself, the thing that actually bites is a
mismatch between the config and the weights. An earlier build of this model listed
lm_head in quantization_config.ignore while having quantized it, and vLLM refused
to load it:
text
ValueError: There is no module or parameter named 'lm_head.input_scale' in Qwen3_5ForCausalLM.The available parameters belonging to lm_head (ParallelLMHead) are: {'lm_head.weight'}vLLM builds the module from the config, then finds scale tensors the config said would not be there. Either quantize it and say so, or do not quantize it and say that — just do not ship both claims.
This checkpoint carries k_scale/v_scale for the 16 layers that use full attention
(the other 48 are gated-delta linear attention and have no KV cache). Without them,
serving with --kv-cache-dtype fp8 silently runs at scale 1.0:
text
Using KV cache scaling factor 1.0 for fp8_e4m3Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attentionq_scale/prob_scale are still absent — modelopt's KV quantizer covers K and V only,
and vLLM substitutes k_scale for q.
Verified on vLLM 0.30.0 (official image) on an RTX PRO 6000 Blackwell 96 GB,
driver 615.71.09. Note there is no --language-model-only:
bash
vllm serve Qwen3.8-27B-FP8-MTP \ --trust-remote-code --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --max-model-len 262144 --max-num-seqs 4 \ --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 \ --quantization modelopt \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Measured: weights load in ~14 s, 1,622,656 KV tokens, 6.19× concurrency at 262 k context. It answers questions about images — the vision path was tested by inference, not only by inspecting the files.
Budget for reasoning tokens. With --reasoning-parser qwen3 the model thinks before
answering and that counts against max_tokens. At 64 the entire budget went to reasoning
and content came back null. Use 256+.
nvidia-modelopt 0.43.0, torch 2.14.1+cu130, transformers 5.18.0neuralmagic/calibration (LLM split), 64 samples, seqlen 8192Calibration is text-only, which every public text-calibrated VLM quantization shares. The vision tower is unquantized so it is unaffected, but the quantized language layers are calibrated on text distributions.
Qwen3.8-27B-NVFP4-MTP — same base, same calibration, same everything except MLP precision,
so a comparison between the two attributes to precision alone.
Built in a disposable container on the host that owns the GPU: a CUDA image with
nvidia-modelopt, the driver userspace injected by nvidia-container-toolkit so it
matches the host exactly, and the base model plus exports on a separate volume that
outlives the container. The whole quantization run is ~90 seconds on this card; the
55.6 GB base download dominates everything.
The precision plan was read out of unsloth/Qwen3.8-27B-NVFP4's shipped
quantization_config rather than guessed, then corrected in two places: lm_head
moved out of the FP8 tier (see above), and *[kv]_bmm_quantizer added from
modelopt's own FP8_KV_CFG to get calibrated KV scales.
ValueError: There is no module or parameter named 'lm_head.input_scale' in Qwen3_5ForCausalLM.The available parameters belonging to lm_head (ParallelLMHead) are: {'lm_head.weight'}vllm serve Qwen3.8-27B-FP8-MTP \ --trust-remote-code --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --max-model-len 262144 --max-num-seqs 4 \ --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 \ --quantization modelopt \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}'