Twu31

Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency

README

License: apache-2.0

Quick start (vLLM ≥ 0.17; measured on 0.24.0, 0.27.1, 0.30.0, 0.31.0)

bash

vllm serve Twu31/Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency \  --served-model-name qwen36 --max-model-len 12288 --max-num-seqs 6 \  --enable-prefix-caching --prefix-match-unit 16 --mamba-ssm-cache-dtype bfloat16 \  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

--mamba-ssm-cache-dtype bfloat16 halves the hybrid model's prefix-cache block (1056 → 544 tokens); outputs were identical with and without it on our harness.
--prefix-match-unit 16 (measured on vLLM 0.30 and 0.31) also stores the GDN state at the prompt's last 16-token boundary, so a request that shares a fixed system prompt hits within 16 tokens of it instead of at the last 544-token block: TTFT 50 → 25 ms on an RTX 6000 Ada with vLLM 0.31, no prompt padding needed. Drop the flag on vLLM older than 0.26.
MTP (--speculative-config '{"method":"mtp","num_speculative_tokens":2}'): on vLLM 0.24 and 0.27.1 it made the turn slower (below); on 0.30 and 0.31 with --prefix-match-unit 16 it is a small win, 3.7 vs 4.1 ms/token at 56 % acceptance and about 5 % shorter turns for the same reply length. Measure on your own replies.
Vision tower is BF16, images work out of the box (--mm-processor-kwargs '{"min_pixels":50176,"max_pixels":200704}' to bound image tokens). Fits a 24 GB card for text; on 32 GB at --gpu-memory-utilization 0.85 vLLM reports 298 k KV tokens. SGLang: untested.

Latency (measured)

Same harness for every row: a ~1.4 k-token fixed persona system prompt + 2 turns of history + 18 real spoken-Chinese user turns × 3, greedy, streaming, thinking off; medians of 54 requests; replies 20–27 tokens. One RTX 5090 32 GB, vLLM 0.27.1, flags as above without --prefix-match-unit. The anchor is QuantTrio/Qwen3.6-35B-A3B-AWQ (data-free, experts only, AWQ format), a widely used community INT4 build.

Table with columns: Checkpoint, Weights in vLLM, fp32 state, 1478-tok prompt — TTFT / turn / ms·tok, bf16 state, block-aligned 1693-tok prompt, + MTP k=2 (acceptance)
Checkpoint	Weights in vLLM	fp32 state, 1478-tok prompt — TTFT / turn / ms·tok	bf16 state, block-aligned 1693-tok prompt	+ MTP k=2 (acceptance)
anchor (experts-only INT4, data-free)	22.2 GiB	53 / 142 / 4.6	56 / 141 / 4.5	53 / 149 / 4.1 (56.9 %)
this build, INT4 body only	19.0 GiB	58 / 148 / 3.6	63 / 153 / 3.5	57 / 168 / 3.5 (54.5 %)
this build (`main`, + INT8 lm_head)

vLLM 0.24.0 on the same card: this build 3.2–3.3 ms/token, anchor 4.7. Five concurrent streams (MTP off): this build 329 ms/turn, 9.4 ms/token, 424 tok/s aggregate vs anchor 348 / 12.2 / 321.

MTP did not pay on vLLM 0.27.1: the draft head is a full BF16 MoE decoder layer, the INT4 main body is cheap, replies are 20–30 tokens and acceptance is 53–57 % — k=2 buys ~10 % on ms/token and gives it back in overhead, so the whole turn is slower (159 vs 140 ms). On vLLM 0.30 and 0.31 with --prefix-match-unit 16 it is a small win (next table).

vLLM 0.30 / 0.31 on an RTX 6000 Ada 48 GB (same harness, main; this card has about half the 5090's memory bandwidth, so ms/token is higher than above; KV cache fixed at 10 GiB; 0.30.0 and 0.31.0 run back to back on 2026-10-07; TTFT = median of 54 requests, warm turn = median of the second and third pass, prefix already cached)

Table with columns: Recipe, vLLM, TTFT, Warm turn, ms/token, Reply tokens (median), MTP accepted / step
Recipe	vLLM	TTFT	Warm turn	ms/token	Reply tokens (median)	MTP accepted / step
Quick start without `--prefix-match-unit`	0.31.0	50	137	4.2	23	-
Quick start (with `--prefix-match-unit 16`)	0.30.0 / 0.31.0	28 / 25

Reply lengths differ between rows because greedy replies are not bit-stable across restarts: two restarts of the same vLLM version gave a different reply on 9 to 12 of the 18 prompts on this MoE (kernels are picked by timing at startup), so compare per token: for a 24-token reply the MTP recipe comes to about 114 ms against 120 ms without it on 0.31.0. 0.31.0 is equal to or slightly faster than 0.30.0 in every row. MoE weights run on vLLM's Marlin WNA16 MoE backend on both versions.

Prefix caching on a hybrid GDN MoE reuses whole blocks only — 1056 tokens with the fp32 recurrent state, 544 with --mamba-ssm-cache-dtype bfloat16 — so pad a fixed system prompt to a block multiple if TTFT matters (on a 5090 the difference is small, 53–63 ms either way; on slower cards it is not). On vLLM 0.26 and newer --prefix-match-unit 16 replaces the padding.

Fidelity vs the BF16 base (same harness, n = 18)

Reference = greedy answers of the BF16 base to the same 18 prompts. NLL = mean per-token negative log-likelihood the quantized model assigns to the BF16 answer (lower = closer). Exact / first-sentence = greedy output identical to the BF16 answer / up to the first sentence break. Counts out of 18 have wide intervals (5 / 18 is 0.10 to 0.54 at 95 %), so read them as a sanity check, not a ranking.

Table with columns: Checkpoint, NLL mean / median ↓, exact / first-sent, long-form distinct-2, vision, format probes
Checkpoint	NLL mean / median ↓	exact / first-sent	long-form distinct-2	vision	format probes
anchor (experts-only INT4, data-free)	0.408 / 0.426	6 / 7	0.975	✓	2/3
experts-only INT4, our GPTQ + Chinese-chat calibration (structure control)	0.419 / 0.470	8 / 9	0.914	✓	2/3

All checkpoints sit in one NLL band (0.40–0.43). With 18 prompts the smallest difference this harness can resolve is about 0.025 nats, larger than the gaps inside the band, so read the table as a pass/fail check against the BF16 base, not as a ranking. Persona tag compliance 54/54, empty replies 0, no repetition loops, for every checkpoint. Its greedy replies are a little longer than the anchor's (median 27 vs 20 tokens; BF16 base median 21.5) because on two or three prompts it takes a longer branch where the base stops after one sentence — same NLL either way, i.e. quantization noise flipping a near-tie.

What is quantized, what is not

Table with columns: Module, Precision, Why
Module	Precision	Why
Routed experts `mlp.experts.*.{gate,up,down}_proj`, shared expert, GDN `in_proj_qkv / in_proj_z / out_proj`, attention `q/k/v/o_proj`, incl. layer 0	INT4 sym g128 (GPTQ)	everything a decode step reads; the community builds keep all but the routed experts in BF16
GDN `in_proj_a`, `in_proj_b`; MoE router `mlp.gate`, `shared_expert_gate`; norms, `conv1d`, embeddings	BF16	tiny gates / routing decisions — 4-bit hurts, saves nothing

Format: compressed-tensors pack-quantized, group_0 = INT4 g128 sym actorder=weight on Linear, group_1 = INT8 g128 on re:.*lm_head$; per-expert tensors (experts.N.gate_proj.weight_packed, vLLM CompressedTensorsWNA16MarlinMoEMethod). 21.5 GB on disk (BF16 base 71.9 GB). Until 2026-10-07 the ignore list in config.json also carried 60 entries that matched no tensor, layers.N.linear_attn and layers.N.linear_attn.norm: the compressed-tensors 0.18 exporter treated any class whose name contains Gate as a Linear layer, which caught GatedDeltaNet and RMSNormGated (fixed upstream in ). They were no-ops, since the GDN projections listed above are INT4 and vLLM never looks those names up, and they have been removed from ; weights are unchanged.

Calibration & reproduce

384 sequences ≥ 2048 tokens, rendered through the model's own chat template with enable_thinking=False: ~60 % multi-turn spoken-dialogue transcripts (a ≈1.4 k-token assistant-persona system prompt + 10–20 short colloquial turns, predominantly Chinese, real speech incl. disfluencies), ~20 % the same turns without a system prompt, ~20 % ShareGPT-GPT4 / UltraChat (zh/en). No benchmark test sets; the transcripts are not distributed and nothing from them is embedded in the weights. moe_calibrate_all_experts=True so rarely-routed experts still get a usable Hessian.

llm-compressor 0.13.0 · transformers 5.14.1 · compressed-tensors 0.18.0 · torch 2.13; one RTX 5090 + 120 GB RAM, 3 h wall-clock (GPTQ's per-module loop over 256 experts × 3 × 40 layers; independent of sample count).

python

import torchfrom datasets import Datasetfrom llmcompressor import oneshotfrom llmcompressor.modeling.moe.linearize import load_quantizable_moefrom llmcompressor.modifiers.quantization import GPTQModifierfrom transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration MODEL = "Qwen/Qwen3.6-35B-A3B"with load_quantizable_moe(Qwen3_5MoeForConditionalGeneration):          # fused 3-D experts -> per-expert nn.Linear    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL, dtype=torch.bfloat16)tok = AutoTokenizer.from_pretrained(MODEL)ds = Dataset.from_list([{"text": t} for t in pre_rendered_texts])       # 384 x >=2048 tokens, chat template applied, thinking offignore = ["re:.*visual.*", "lm_head", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$",          r"re:.*mlp\.gate$", r"re:.*shared_expert_gate$"]recipe = GPTQModifier(ignore=ignore, scheme="W4A16", targets=["Linear"], dampening_frac=0.01, actorder="weight")oneshot(model=model, processor=tok, dataset=ds, recipe=recipe, max_seq_length=2048,        num_calibration_samples=384, moe_calibrate_all_experts=True)model.save_pretrained("out", save_compressed=True, max_shard_size="4GB"); tok.save_pretrained("out")

Three things the exporter does not do for you: (1) copy the BF16 mtp.* tensors from the base checkpoint into the export and add re:.*mtp.* to quantization_config.ignore — without both, vLLM's drafter loads packed weights it cannot use and MTP runs at 0 %; (2) apply the INT8 lm_head post-hoc (compressed-tensors packer, group_1, target re:.*lm_head$); (3) copy the upstream tokenizer / config files back verbatim (transformers 5.14 re-serialises tokenizer.json). Full scripts (quantization, export fix-up, INT8 head, evaluation harness) are in our repo, llm_quant/qwen36_35b_zhchat/.

Limitations

Built and measured for short-reply, thinking-off, single/few-stream spoken chat. Long chain-of-thought, tool use, > 12 k context and academic benchmarks were not evaluated; fidelity numbers are relative to the BF16 base on 18 conversational prompts.
Measured on one RTX 5090 (vLLM 0.24.0 / 0.27.1) and one RTX 6000 Ada (vLLM 0.30.0 / 0.31.0); the Marlin path is the same on Ampere and Ada, absolute numbers follow memory bandwidth.
vLLM 0.27.1 on Python 3.10 needs from __future__ import annotations at the top of flashinfer/comm/fd_exchange.py; fresh machines need g++ and python3-dev for the JIT kernels.

Acknowledgements

Qwen team for Qwen3.6; the vLLM / llm-compressor / compressed-tensors maintainers.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Twu31

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality