GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
hyperhuzaifa
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
hyperhuzaifa
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
| Method | AWQ (AWQModifier, duo_scaling="both") + QuantizationModifier W4A16 |
| Scheme | int4, symmetric, group size 128 |
| Calibration | 128 samples × 1024 tokens, HuggingFaceH4/ultrachat_200k (chat-templated) |
| Pipeline | sequential, moe_calibrate_all_experts=True |
| Tooling | llm-compressor 0.12.0, compressed-tensors 0.17.1, transformers 5.10.1 |
moe_calibrate_all_experts matters here: with top-8-of-256 routing, each expert would otherwise see only ~3% of
calibration tokens.
All language-model Linear layers — including the ones many public MoE quants skip: the 256 routed experts per
layer, the shared experts, the 30 gated-DeltaNet (linear-attention) projections (in_proj_qkv, in_proj_z,
in_proj_b, in_proj_a), and the 10 full-attention blocks.
| component | size | why |
|---|---|---|
embed_tokens | 0.95 GB | quantizing input embeddings is a known quality cliff |
lm_head | 0.95 GB | required — see Known issues |
vision tower (model.visual.*) | 0.83 GB | calibration was text-only, so its activation scales would be meaningless |
MTP head (mtp.*) | 1.57 GB | draft head, kept intact for optional speculative decoding |
Requires vLLM ≥ 0.26.0 (architecture Qwen3_5MoeForConditionalGeneration).
--max-num-seqsThis is a hybrid architecture: 30 of 40 layers are gated DeltaNet ("Mamba"-style). vLLM allocates one recurrent
cache block per decode sequence regardless of context length, and on a 24 GB card only ~46 fit. vLLM's default
--max-num-seqs 256 makes CUDA graph capture abort:
markdown
ValueError: max_num_seqs (256) exceeds available Mamba cache blocks (46).
Pass --max-num-seqs 46 or lower on a single 24 GB GPU. This caps concurrency independently of KV tokens.
bash
vllm serve hyperhuzaifa/Qwen3.6-35B-A3B-W4A16-AWQ \--max-model-len 32768 \--max-num-seqs 16 \--kv-cache-dtype fp8 \--gpu-memory-utilization 0.94
bash
vllm serve hyperhuzaifa/Qwen3.6-35B-A3B-W4A16-AWQ \--tensor-parallel-size 2 \--max-model-len 262144 \--max-num-seqs 32 \--kv-cache-dtype fp8 \--gpu-memory-utilization 0.90 \--disable-custom-all-reduce
The draft head is included. Enable with:
bash
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'
mtp_num_hidden_layers=1, so 1 speculative token is the maximum.
The MTP weights cost nothing until you enable this flag — vLLM only instantiates the module under
--speculative-config, so otherwise they sit on disk. Enabling it makes the draft head resident (+1.57 GiB)
and raises per-token KV cost.
⚠️ Do not enable MTP on a single 24 GB card. The draft head consumes ~92% of the KV budget and even 8K context then fails to allocate:
| 1 × 24 GB | max context |
|---|---|
| MTP inactive | 171,872 |
| MTP active | < 8,192 (unusable) |
Measured MTP behaviour on 2 × 24 GB over PCIe (no NVLink): the draft head is very good — mean acceptance length 1.90 / 2.0, ~90% acceptance — but end-to-end throughput still drops, because each draft forward adds an all-reduce and the draft is a full 256-expert MoE block:
| TP=2 config | KV pool | concurrency @256K | decode |
|---|---|---|---|
| no MTP | 1,965,063 | 7.50× | 127 tok/s |
MTP, --max-num-batched-tokens 4096 | 1,656,910 | 6.32× | 107 tok/s |
MTP, --max-num-batched-tokens 16384 | 1,552,853 | 5.92× | 102 tok/s |
Raising --max-num-batched-tokens (which vLLM itself suggests) made it worse. On this hardware, serve
without speculative decoding. On an NVLink pair the 1.90 acceptance length would likely convert into a real
speedup — the bottleneck is interconnect, not the draft head.
Measured on RTX 3090 / RTX 4090 (24 GB each), fp8 KV cache, vLLM 0.26.0.
| config | max context | concurrency | decode |
|---|---|---|---|
| 1 × 24 GB, CUDA graphs | 32K | 2.86× | 156 tok/s |
1 × 24 GB, --enforce-eager | 171,872 | 1× | 19.5 tok/s |
| 2 × 24 GB (TP=2), CUDA graphs | 262,144 | 7.5× @ 256K | 127 tok/s |
TP=2 gives a KV pool of 1,965,063 tokens. On a single card, full 262K context is not reachable — it needs 2.58 GiB of KV against ~1.71 GiB available.
Context and throughput trade off sharply on one card: --enforce-eager buys ~17K tokens of context but costs ~8×
decode speed.
lm_head is intentionally NOT quantized. vLLM builds it as a plain ParallelLMHead for this architecture and
requires lm_head.weight. A packed head fails at load with:
markdown
ValueError: There is no module or parameter named 'lm_head.weight_packed'
Load with the multimodal class. AutoModelForCausalLM resolves to Qwen3_5MoeForCausalLM (text-only) and will
silently drop the vision tower and MTP head. vLLM does not register that architecture at all. Use
Qwen3_5MoeForConditionalGeneration.
Vision is preserved but was not calibrated. The tower is bf16 and untouched, so image quality should match the base model, but it received no activation-aware scaling.
Not evaluated on benchmarks. Coherence was verified by generation (code synthesis, instruction following, long-context recall) rather than by lm-eval-harness or similar. No perplexity or task-accuracy numbers are published here — treat quality claims as unverified.
Apache 2.0, inherited from the base model. See the base model's LICENSE.
| routers / gates / norms | tiny | quantizing routers mis-routes tokens |