GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
OptimizeLLM
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
OptimizeLLM
Model Tree
Qwen/Qwen3.6-27B
Input Modalities
Output Modalities
Supported Functionality
Tested on vLLM 0.24 through 0.26.
bash
vllm serve OptimizeLLM/Qwen3.6-27B-heretic-MTP-FP8 \--max-num-seqs 32 \--reasoning-parser qwen3 \--reasoning-config '{}' \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--enable-prefix-caching \--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
--max-num-seqs 32 is required. vLLM's default of 1024 exceeds the available Mamba cache blocks (933) on this hybrid architecture, and the engine won't start.
Single RTX PRO 6000 Blackwell, vLLM 0.26.0, MTP=4, thinking off:
| Metric | Value |
|---|---|
| Single-stream generation | ~86 tok/s |
| MTP acceptance length | 2.5-3.4 tokens/step |
| Per-position acceptance | 0.81 / 0.65 / 0.51 / 0.42 |
| Weights on GPU | ~30GB |
Acceptance varies with workload. That range covers conversational and tool-calling traffic.
This is what we run, not a suggested starting point.
num_speculative_tokens: 4. Positions 3 and 4 accept least, so 2 or 3 looks better on paper. 4 has been faster for us on conversational and tool-calling traffic. It's one flag, so test it against your own.--enable-prefix-caching. How much this buys you depends on your traffic rather than the model. A long stable system prompt with short turns on top caches well; one-shot requests sharing no prefix gain nothing. Watch anything that varies near the front of the prompt, like a timestamp or session id, since it invalidates everything after it.--reasoning-parser qwen3 with --reasoning-config '{}' for thinking control. thinking_token_budget works. Cap it. Uncapped, the model will sometimes think at length before an easy answer, and that shows up in tail latency.--tool-call-parser qwen3_coder. Clean tool-call JSON, including against large MCP schemas.--gpu-memory-utilization at its 0.9 default gives a large KV cache on a 96GB card. Lower it first if you're sharing the card with something else.One template constraint: system messages are only accepted at position 0. A mid-conversation system turn fails with a template error. Merge runtime context into a user message instead.
Quantized with llm-compressor (compressed-tensors 0.14.1.dev28), FP8_BLOCK preset. Needs transformers >= 5.0 for the Qwen3.6 classes.
Scheme:
float-quantizedFP8_BLOCK is data-free RTN, about 30 minutes on CPU.lm_head, embed_tokens, visual.*, the linear_attn in_proj_a/in_proj_b/in_proj_ba gates, and mtp.*. The delta-rule gates are precision-sensitive. in_proj_qkv, in_proj_z and are quantized.Five things this checkpoint does that a plain oneshot() run won't:
CUDA_VISIBLE_DEVICES="". The data-free pipeline dispatches to visible GPUs and will OOM against a card that's already busy.visual.* tensors and flattens config.json to a text-only causal LM. Splice the tensors back and restore the multimodal config with quantization_config grafted in.mtp.*, so take all 15 in BF16 from official Qwen3.6-27B. Splicing only needs the source shards holding those keys, not the whole checkpoint.tokenizer.json, tokenizer_config.json and chat_template.jinja. The source's tokenizer was damaged by a round-trip through an older tokenizers version: a 768-token truncation cap, a pre-tokenizer regex missing the Unicode Mark class, and seven audio tokens mapped to untrained embedding rows. The chat template is byte-identical either way, so prompting is unchanged.Then check the index against the shards before serving: 1599 tensors, no orphans in either direction.
out_projre: patterns. vLLM fuses in_proj_a and in_proj_b into in_proj_ba, and llm-compressor writes resolved concrete names, so a literal ignore list misses the fused module and vLLM refuses to load the checkpoint.