GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
sakamakismile
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sakamakismile
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| concurrency | no MTP (aggr t/s) | MTP n=3 (aggr t/s) |
|---|---|---|
| 1 | 49.0 | 72.6 |
| 2 | 93.2 | 110.9 |
| 4 | 180.9 | 232.8 |
| 8 | 318.3 | 386.9 |
Single-stream prefill (prefix cache disabled, mean of 3): 3,820 tok/s @ 8k, 3,299 @ 32k, 2,363 @ 100k prompt tokens. GPU KV cache at TP=4/128k: 1,025,977 tokens (7.83× concurrency).
bash
vllm serve sakamakismile/Qwen3.8-27B-MTP-NVFP4 \--trust-remote-code --tensor-parallel-size 4 \--max-model-len 131072 --kv-cache-dtype fp8 \--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
On boards without P2P add NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce.
mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance and slower than no MTP.gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.reasoning_effort defaults to xhigh. Give it max_tokens ≥ 4096, and for long-form work send chat_template_kwargs: {"reasoning_effort": "medium"} — at the default effort the <think> phase can consume the entire budget and emit no answer at all (I saw exactly that on long single-file HTML generation: 32,768 tokens spent thinking, zero content). A 9-case gate (French/English long-form, temperature 0 and 0.7) passes at both xhigh and medium on this base model, but medium keeps thinking to ~1k characters instead of ~5–9k, so it is the safer default. These modes are three system prompts rather than three capability levels: injects "think carefully, validate key assumptions, consider alternatives", injects "keep your thinking brief", and injects nothing at all. Measured length follows suit (a few hundred chars / ~1k / 5–9k) with no quality difference on short verifiable tasks (96 runs).llm-compressor NVFP4 (W4A4, group 16), targets: [Linear], ignore: [lm_head, re:.*visual.*, re:.*conv1d.*, re:.*mtp.*], 32 calibration samples × 8192 tok from neuralmagic/calibration. MTP tensors grafted back in bf16 post-save and appended to quantization_config.ignore.
🙏 Qwen team for the model, vLLM & llm-compressor teams for the tooling.
xhighlowmedium<think>--limit-mm-per-prompt '{"image":0,"video":0}' to be explicit); the vision tower is shipped in bf16 but multimodal serving was not benchmarked here.