GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
malaiwah
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
malaiwah
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| concurrency | no MTP (aggr t/s) | MTP n=3 (aggr t/s) |
|---|---|---|
| 1 | 49.0 | 72.6 |
| 2 | 93.2 | 110.9 |
| 4 | 180.9 | 232.8 |
| 8 | 318.3 | 386.9 |
Single-stream prefill (prefix cache disabled, mean of 3): 3,820 tok/s @ 8k, 3,299 @ 32k, 2,363 @ 100k prompt tokens. GPU KV cache at TP=4/128k: 1,025,977 tokens (7.83× concurrency).
bash
vllm serve sakamakismile/Qwen3.8-27B-MTP-NVFP4 \--trust-remote-code --tensor-parallel-size 4 \--max-model-len 131072 --kv-cache-dtype fp8 \--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
On boards without P2P add NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce.
mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance and slower than no MTP.gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.max_tokens ≥ 4096 or the <think> phase eats the budget.--limit-mm-per-prompt '{"image":0,"video":0}' to be explicit); the vision tower is shipped in bf16 but multimodal serving was not benchmarked here.llm-compressor NVFP4 (W4A4, group 16), targets: [Linear], ignore: [lm_head, re:.*visual.*, re:.*conv1d.*, re:.*mtp.*], 32 calibration samples × 8192 tok from neuralmagic/calibration. MTP tensors grafted back in bf16 post-save and appended to quantization_config.ignore.
🙏 Qwen team for the model, vLLM & llm-compressor teams for the tooling.