Twu31

Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency

README

License: apache-2.0

Quick start (vLLM ≥ 0.17; measured on 0.24.0 / 0.27.1)

bash
vllm serve Twu31/Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency \
  --served-model-name qwen36 --max-model-len 12288 --max-num-seqs 6 \
  --enable-prefix-caching --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

--mamba-ssm-cache-dtype bfloat16 halves the hybrid model's prefix-cache block (1056 → 544 tokens); outputs were identical with and without it on our harness.
Leave MTP off for chat — it loads and accepts (~53 % at k=2) but the turn gets slower (below). If you want it: --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.
Vision tower is BF16, images work out of the box (--mm-processor-kwargs '{"min_pixels":50176,"max_pixels":200704}' to bound image tokens). Fits a 24 GB card for text; on 32 GB at --gpu-memory-utilization 0.85 vLLM reports 298 k KV tokens. SGLang: untested.

Latency (measured)

Same harness for every row: a ~1.4 k-token fixed persona system prompt + 2 turns of history + 18 real spoken-Chinese user turns × 3, greedy, streaming, thinking off; medians of 54 requests; replies 20–27 tokens. One RTX 5090 32 GB, vLLM 0.27.1, flags as above. The anchor is the most-downloaded Qwen3.6-35B-A3B INT4 checkpoint (data-free RTN, experts only, AWQ format).

Table with columns: Checkpoint, Weights in vLLM, fp32 state, 1478-tok prompt — TTFT / turn / ms·tok, bf16 state, block-aligned 1693-tok prompt, + MTP k=2 (acceptance)
Checkpoint	Weights in vLLM	fp32 state, 1478-tok prompt — TTFT / turn / ms·tok	bf16 state, block-aligned 1693-tok prompt	+ MTP k=2 (acceptance)
anchor (experts-only INT4, data-free)	22.2 GiB	53 / 142 / 4.6	56 / 141 / 4.5	53 / 149 / 4.1 (56.9 %)
this build, INT4 body only	19.0 GiB	58 / 148 / 3.6	63 / 153 / 3.5	57 / 168 / 3.5 (54.5 %)
this build (`main`, + INT8 lm_head)

vLLM 0.24.0 on the same card: this build 3.2–3.3 ms/token, anchor 4.7. Five concurrent streams (MTP off): this build 329 ms/turn, 9.4 ms/token, 424 tok/s aggregate vs anchor 348 / 12.2 / 321.

MTP does not pay here: the draft head is a full BF16 MoE decoder layer, the INT4 main body is cheap, replies are 20–30 tokens and acceptance is 53–57 % — k=2 buys ~10 % on ms/token and gives it back in overhead, so the whole turn is slower (159 vs 140 ms). Re-measure if your replies are long.

Prefix caching on a hybrid GDN MoE reuses whole blocks only — 1056 tokens with the fp32 recurrent state, 544 with --mamba-ssm-cache-dtype bfloat16 — so pad a fixed system prompt to a block multiple if TTFT matters (on a 5090 the difference is small, 53–63 ms either way; on slower cards it is not).

Fidelity vs the BF16 base (same harness, n = 18)

Reference = greedy answers of the BF16 base to the same 18 prompts. NLL = mean per-token negative log-likelihood the quantized model assigns to the BF16 answer (lower = closer). Exact / first-sentence = greedy output identical to the BF16 answer / up to the first sentence break. Treat ±1–2 counts as noise.

Table with columns: Checkpoint, NLL mean / median ↓, exact / first-sent, long-form distinct-2, vision, format probes
Checkpoint	NLL mean / median ↓	exact / first-sent	long-form distinct-2	vision	format probes
anchor (experts-only INT4, data-free)	0.408 / 0.426	6 / 7	0.975	✓	2/3
experts-only INT4, our GPTQ + Chinese-chat calibration (structure control)	0.419 / 0.470	8 / 9	0.914	✓	2/3

All checkpoints sit in one NLL band (0.40–0.43); this build has the lowest. Persona tag compliance 54/54, empty replies 0, no repetition loops, for every checkpoint. Its greedy replies are a little longer than the anchor's (median 27 vs 20 tokens; BF16 base median 21.5) because on two or three prompts it takes a longer branch where the base stops after one sentence — same NLL either way, i.e. quantization noise flipping a near-tie.

What is quantized, what is not

Table with columns: Module, Precision, Why
Module	Precision	Why
Routed experts `mlp.experts.*.{gate,up,down}_proj`, shared expert, GDN `in_proj_qkv / in_proj_z / out_proj`, attention `q/k/v/o_proj`, incl. layer 0	INT4 sym g128 (GPTQ)	everything a decode step reads; the community builds keep all but the routed experts in BF16
GDN `in_proj_a`, `in_proj_b`; MoE router `mlp.gate`, `shared_expert_gate`; norms, `conv1d`, embeddings	BF16	tiny gates / routing decisions — 4-bit hurts, saves nothing

Format: compressed-tensors pack-quantized, group_0 = INT4 g128 sym actorder=weight on Linear, group_1 = INT8 g128 on re:.*lm_head$; per-expert tensors (experts.N.gate_proj.weight_packed, vLLM CompressedTensorsWNA16MarlinMoEMethod). 21.5 GB on disk (BF16 base 71.9 GB).

Calibration & reproduce

384 sequences ≥ 2048 tokens, rendered through the model's own chat template with enable_thinking=False: ~60 % multi-turn spoken-dialogue transcripts (a ≈1.4 k-token assistant-persona system prompt + 10–20 short colloquial turns, predominantly Chinese, real speech incl. disfluencies), ~20 % the same turns without a system prompt, ~20 % ShareGPT-GPT4 / UltraChat (zh/en). No benchmark test sets; the transcripts are not distributed and nothing from them is embedded in the weights. moe_calibrate_all_experts=True so rarely-routed experts still get a usable Hessian.

llm-compressor 0.13.0 · transformers 5.14.1 · compressed-tensors 0.18.0 · torch 2.13; one RTX 5090 + 120 GB RAM, 3 h wall-clock (GPTQ's per-module loop over 256 experts × 3 × 40 layers; independent of sample count).

python
import torch
from datasets import Dataset
from llmcompressor import oneshot
from llmcompressor.modeling.moe.linearize import load_quantizable_moe
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration

MODEL = "Qwen/Qwen3.6-35B-A3B"
with load_quantizable_moe(Qwen3_5MoeForConditionalGeneration):          # fused 3-D experts -> per-expert nn.Linear
    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL, dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(MODEL)
ds = Dataset.from_list([{"text": t} for t in pre_rendered_texts])       # 384 x >=2048 tokens, chat template applied, thinking off
ignore = ["re:.*visual.*", "lm_head", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$",
          r"re:.*mlp\.gate$", r"re:.*shared_expert_gate$"]
recipe = GPTQModifier(ignore=ignore, scheme="W4A16", targets=["Linear"], dampening_frac=0.01, actorder="weight")
oneshot(model=model, processor=tok, dataset=ds, recipe=recipe, max_seq_length=2048,
        num_calibration_samples=384, moe_calibrate_all_experts=True)
model.save_pretrained("out", save_compressed=True, max_shard_size="4GB"); tok.save_pretrained("out")

Three things the exporter does not do for you: (1) copy the BF16 mtp.* tensors from the base checkpoint into the export and add re:.*mtp.* to quantization_config.ignore — without both, vLLM's drafter loads packed weights it cannot use and MTP runs at 0 %; (2) apply the INT8 lm_head post-hoc (compressed-tensors packer, group_1, target re:.*lm_head$); (3) copy the upstream tokenizer / config files back verbatim (transformers 5.14 re-serialises tokenizer.json). Full scripts (quantization, export fix-up, INT8 head, evaluation harness) are in our repo, llm_quant/qwen36_35b_zhchat/.

Limitations

Built and measured for short-reply, thinking-off, single/few-stream spoken chat. Long chain-of-thought, tool use, > 12 k context and academic benchmarks were not evaluated; fidelity numbers are relative to the BF16 base on 18 conversational prompts.
Measured on one RTX 5090 only; the Marlin path is the same on Ampere/Ada but absolute numbers will differ.
vLLM 0.27.1 on Python 3.10 needs from __future__ import annotations at the top of flashinfer/comm/fd_exchange.py; fresh machines need g++ and python3-dev for the JIT kernels.

Acknowledgements

Qwen team for Qwen3.6; the vLLM / llm-compressor / compressed-tensors maintainers.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Twu31

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality