GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
frischeDaten
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
frischeDaten
Model Tree
omnisecureai/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
At the time of writing, no quantizer supported this architecture's fused MoE experts:
monkey_patching, but qwen3_5_moe needs ≥5.2).Qwen3_5MoeExperts (3-D tensors), and its exporter can't serialize them. → filed upstream: NVIDIA/Model-Optimizer#2011.qwen3_5_vl_moe weight_loader only loads the fused experts as bf16 (drops NVFP4 scales). → filed upstream: vllm-project/vllm#49638.This repo ships the model plus the two patches needed to produce and serve it (see below). Both fixes are also filed against the upstream projects so they can be resolved at the source.
| Project | Issue | What it fixes |
|---|---|---|
| vLLM | vllm-project/vllm#49638 | qwen3_5_vl_moe weight_loader dropping NVFP4 scales for the fused experts |
| NVIDIA ModelOpt | NVIDIA/Model-Optimizer#2011 | quantizer/exporter skipping the fused Qwen3_5MoeExperts |
| Metric | Value |
|---|---|
| Weights | 22 GB (bf16 was 67 GB) |
| Vision | ✅ intact (verified OCR + counting) |
| Context | 512K (YARN ×2 from 256K native) |
| KV cache @ util 0.78 (fp8) | ~65 GB → ≈11× a full 512K sequence |
| Decode, single-stream (no spec) | ~34–35 tok/s (MARLIN NVFP4 MoE kernel) |
| Decode, single-stream + MTP | ~44 tok/s short ctx, ~41 tok/s @64k (lossless, peaks ~50–53) |
| Decode, aggregate (concurrent) | ~230 tok/s @ 8 seqs, ~300 tok/s ceiling (details) |
Single-stream is kernel-bound: the fast FlashInfer NVFP4-MoE kernel crashes on the vision path on GB10, so we use the slower-but-stable MARLIN backend. Aggregate throughput scales with concurrency up to a point — see Concurrency scaling — which suits multi-agent / sub-agent workloads. On top of that, the shipped MTP head raises single-stream decode by ~1.26–1.44× losslessly — see below.
GB10 is memory-bandwidth-bound, so aggregate decode throughput saturates at ~300 tok/s and is reached by ~8–16 concurrent streams — pushing more sequences past that doesn't add throughput, it just splits the same budget into slower, higher-latency slices. Measured with llama-benchy, MTP on, ~20K-token context per request (a realistic sub-agent size), fp8 KV:
| Concurrency | Aggregate decode (peak) | Per-request decode | TTFT* |
|---|---|---|---|
| 1 | 52 tok/s | 49 tok/s | ~5 s |
| 4 | 143 tok/s | 38 tok/s | ~14 s |
| 8 | 231 tok/s | 33 tok/s | ~25 s |
| 16 | 295 tok/s | 25 tok/s | ~46 s |
* time-to-first-token for 16–48 requests each carrying a 20K-token prompt arriving simultaneously (worst-case burst; prefill-dominated).
Practical takeaway: --max-num-seqs 16 is the sweet spot — it captures ~94 % of the aggregate ceiling while keeping first-token latency tolerable. Raising it to 32/48 adds ≤6 % throughput but halves per-request speed and doubles TTFT. The limit here is decode bandwidth, not KV capacity (fp8 KV holds ~5.9M tokens — 512K × ≈11) and not the sequence cap. Note the aggregate ceiling is measured with MTP enabled; spec-decode is a large win at low concurrency (the common case) but spends compute on drafts once the GPU is already saturated, so a pure-throughput, always-saturated server may prefer to disable it.
This build ships the base model's native MTP head, and it works. Serve with vLLM's qwen3_5_mtp speculative method (the Qwen3_5MoeMTP draft class) and you get a lossless decode speedup while keeping fp8 KV cache — so you pay no concurrency penalty for it. Enable it with one flag:
markdown
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'
⚠️ Use
qwen3_5_mtp, notqwen3_next_mtp. This architecture (qwen3_5_moe) has its own dedicated MTP class. Pointing vLLM at theqwen3_nextMTP path loads the wrong layout and gives ~0 % acceptance — the origin of the myth that "MTP doesn't work on quantized Qwen3.6." It does.
Measured on the same DGX Spark (GB10), fp8 KV, same-session A/B (llama-benchy, decode tok/s, prefill excluded):
| Depth | No spec-decode | MTP (nspec=2) | Speedup | Draft acceptance |
|---|---|---|---|---|
| 0 (short ctx) | 34.5 tok/s | 43.6 (peak 51) | 1.26× | 60–94 % |
| 32k | 32.7 tok/s | 34.9 (peak 49) | 1.07× | content-dependent |
| 64k | 28.3 tok/s | 40.9 (peak 53) |
It is lossless — MTP drafts are verified by the target model, so output is identical to greedy decoding; only speed changes. The head costs ~2 GB and drops 512K concurrency only marginally (fp8 KV is retained → still ≈11×). num_speculative_tokens=2 is the sweet spot; 3 is faster at short/mid context but regresses below baseline at 64k, so it's not recommended for long-context use.
The strongest external option is DFlash (Z-Lab / Modal, z-lab/Qwen3.6-35B-A3B-DFlash), a block-diffusion drafter packaged in the AEON-7 build. We benchmarked it head-to-head on the same GB10 at 512K. It gives a bigger short-context single-stream boost (~1.8×), but:
| This build + MTP | AEON-7 + DFlash | |
|---|---|---|
| Concurrency headroom @ 512K | ≈11× (fp8 KV kept) | 2.65× (BF16 KV required) |
| Short-ctx single-stream | ~44 tok/s (1.26×) | ~64 tok/s (1.8×) |
| Long-ctx (64k–110k) | ~41 tok/s, ≈73 % acceptance | ~9 tok/s, 0 % acceptance (drafter trained @40k) |
| Lossless | ✅ | ✅ |
| Dependencies | stock vLLM class (in-image) | unmerged vLLM PR + external drafter |
DFlash's speedup evaporates past ~40k tokens (its drafter's training length) and it mandates BF16 KV, costing ~4× concurrency at full context. MTP wins exactly where this build is meant to live: long context and high concurrency — while still handing you a solid short-context speedup for free. If your workload is predominantly short-context single-stream chat, DFlash's 1.8× is the higher ceiling; otherwise MTP is the better trade.
Note: long-context decode on GB10 is ultimately memory-bandwidth-bound; MTP helps because high acceptance amortizes the KV reads, but it is not a substitute for more bandwidth.
Requires the patched qwen3_5_vl_moe.py (see vllm-qwen3_5_vl_moe-nvfp4-fused-experts.patch in this repo — it maps the fused-expert NVFP4 scales; tracked upstream at vllm-project/vllm#49638). Mount it over vLLM's copy, and disable the FlashInfer MoE backend:
bash
docker run --gpus all --ipc=host \-e VLLM_USE_FLASHINFER_MOE_FP4=0 -e VLLM_NVFP4_GEMM_BACKEND=marlin \-v $(pwd)/qwen3_5_vl_moe_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5_vl_moe.py \-v /path/to/this-model:/models/omni-nvfp4 -p 8000:8000 \<your-gb10-vllm-image> \vllm serve /models/omni-nvfp4 --served-model-name omni \--max-model-len 524288 --gpu-memory-utilization 0.78 --max-num-seqs 16 \--kv-cache-dtype fp8 --max-num-batched-tokens 8192 \--enable-chunked-prefill --enable-prefix-caching \--limit-mm-per-prompt '{"image":2,"video":0}' --trust-remote-code \--tool-call-parser qwen3_coder --reasoning-parser qwen3 --enable-auto-tool-choice \--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}' \--hf-overrides '{"text_config":{"rope_parameters":{"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":2.0,"original_max_position_embeddings":262144,"mrope_interleaved":true,"mrope_section":[11,11,10]}}}'
Notes / gotchas
VLLM_USE_FLASHINFER_MOE_FP4=0 is required on GB10 — the FlashInfer NVFP4-MoE kernel throws illegal instruction on the image forward. MARLIN is stable.--hf-overrides (and set --max-model-len 262144) if you don't need 512K.--speculative-config line above. Use method qwen3_5_mtp (not qwen3_next_mtp — wrong class for this arch → ~0 % acceptance). It's lossless and keeps fp8 KV; drop the line to disable. See Speculative decoding.quantize.py (in this repo): registers a custom _QuantQwen3_5MoeExperts quantizer with ModelOpt, calibrates on wikitext, and manually packs the fused experts into vLLM's w13/w2 NVFP4 layout (packed uint8 + fp8 block scales + fp32 per-shard global scales + input scales), keeping everything else bf16. It's memory-safe on unified memory (frees expert weights as it packs; avoids ModelOpt's export_hf_checkpoint, which OOMs).
The MTP head is carried over verbatim from the base model in bf16 (the 19 mtp.* tensors, in model-mtp.safetensors) — it is not quantized. vLLM's Qwen3_5MoeMTP class loads it and auto-excludes mtp. from NVFP4, sharing the target model's embeddings and lm_head with the draft. No re-training or re-alignment was needed: the base head accepts 60–94 % of drafts against the NVFP4 body out of the box.
omnisecureai/Qwen3.6-35B-A3B (Apache-2.0), itself derived from Qwen3.6 by Alibaba Qwen.Released under Apache-2.0, inheriting the base model's license.
| Prefill (pp512) | ~4400 tok/s |
| 32 | 314 tok/s | 14 tok/s | ~87 s |
| 48 | 307 tok/s | 10 tok/s | ~129 s |
| ≈73 % |