Twu31

Qwen3.8-27B-AWQ-INT4-ZhChat-MTP

README

License: apache-2.0

Highlights

Table with columns: This build
	This build
Weights	INT4 symmetric, group_size 128, `compressed-tensors` `pack-quantized` → vLLM Marlin W4A16 kernel on SM 8.0+
Size	19.45 GB (BF16 base 55.6 GB) — single 24 GB card for text-only serving with modest context, comfortable on 48 GB+
Kept in BF16	vision tower (`model.visual.`), MTP draft head (`mtp.`, 15 tensors)**, `lm_head`, embeddings, all norms, Gated-DeltaNet per-head gates `linear_attn.in_proj_a` / `in_proj_b`
Calibration	530 × ≥2048-token sequences: 60 % production-style Chinese spoken dialogue (≈1.4 k-token persona system prompt + 10–20 colloquial turns + `[emotion:X]` tail, thinking off), 20 % system-less colloquial multi-turn, 20 % general zh/en chat (ShareGPT-GPT4, UltraChat)
Method	AWQ-class W4A16 via Intel AutoRound 0.14.2, iters 200, nsamples 384, seqlen 2048, batch 4 (1 h 14 min on one RTX 6000 Ada, peak 28 GB VRAM)
MTP	draft head shipped and verified in vLLM — `--speculative-config '{"method":"mtp","num_speculative_tokens":2}'`: acceptance 73.1 %, decode 20.4 → 13.7 ms/token (1.49×) on Chinese chat
Multimodal	image/video input works (vision tower untouched; verified through vLLM)
License	Apache-2.0 (inherited)

Quick start (vLLM ≥ 0.17; measured on 0.24.0 and 0.27.1)

Recipe A — shortest turn latency for chat with a long fixed system prompt (recommended for voice agents):

bash
vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-ZhChat-MTP \
  --served-model-name qwen38 \
  --max-model-len 12288 --max-num-seqs 8 \
  --enable-prefix-caching \
  --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}'

and size your fixed system prompt to a multiple of the cache block (see Prefix caching on a hybrid GDN model) — with a 1.6 k-token prompt this build reaches TTFT 67 ms, 415 ms per 18-token turn on one RTX 6000 Ada.

Recipe B — longer replies (≳ 25 tokens) or short prompts: add MTP

bash
vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-ZhChat-MTP \
  --served-model-name qwen38 \
  --max-model-len 12288 --max-num-seqs 8 \
  --enable-prefix-caching \
  --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="qwen38",
    messages=[
        {"role": "system", "content": "你是一个陪人聊天的仿生人，说话自然简短，每句结尾加 [emotion:happy|neutral|curious|sad|shy|surprised|angry]。"},
        {"role": "user", "content": "今天累死了。"},
    ],
    max_tokens=128, temperature=0.8,
)
print(r.choices[0].message.content)

Notes

No --quantization flag needed — quantization_config in config.json is auto-detected.
Do not pass --calculate-kv-scales on hybrid GDN+attention models (vllm#37554); FP8 KV cache is unnecessary here (only 16 of 64 layers keep a KV cache).
Tensor parallelism: the GDN gate projections (in_proj_a/b, 48×5120) are BF16 precisely so TP ≥ 2 does not hit the Marlin minimum-N constraint.
Transformers ≥ 5.8 + compressed-tensors loads it for inference; use vLLM for serving.

Benchmarks

Single-stream, real conversation prompts (≈1.4 k-token Chinese persona system prompt + 2 turns of history + a real user utterance), greedy, thinking off, 18 prompts × 3 runs, streamed over HTTP, same harness for every row. GPU: one NVIDIA RTX 6000 Ada (48 GB, 960 GB/s), vLLM 0.24.0. Median values. "ms/token" is inter-token latency during decode.

1. Serving recipe ladder (MTP off) — the latency actually comes from here

Table with columns: Configuration, Prompt tokens, Prefix-cache hit, TTFT, Total / turn, ms/token, [emotion:X]
Configuration	Prompt tokens	Prefix-cache hit	TTFT	Total / turn	ms/token	`[emotion:X]`
default (`--enable-prefix-caching`)	1449	51 %	272 ms	634 ms	20.4	54/54
+ `--mamba-ssm-cache-dtype bfloat16`	1449

Same weights, same quality (identical outputs / tag compliance across rows) — the 634 → 415 ms comes purely from making the hybrid model's prefix cache actually hit.

2. MTP speculative decoding on Chinese chat

Table with columns: K (draft tokens), Acceptance, Accepted / draft step, ms/token, Decode speed-up
K (draft tokens)	Acceptance	Accepted / draft step	ms/token	Decode speed-up
0 (off)	—	—	20.4	1.00×
1	79.4 %	0.79	15.9	1.28×
2	73.1 %	1.46	13.7

MTP interacts with prefix caching on this architecture (see below). Full-turn numbers with MTP K=2, --mamba-ssm-cache-dtype bfloat16 and a block-aligned 1.6 k prompt: TTFT 205 ms, 455 ms / 18-token turn, 14.3 ms/token, 72 % cache hit — it overtakes Recipe A once replies exceed ~25 tokens.

3. Why calibration matters — same recipe, different data (both AutoRound W4A16 sym g128, identical BF16 MTP head, MTP K=2)

Table with columns: Build, Calibration, Draft acceptance, Accepted / step, ms/token
Build	Calibration	Draft acceptance	Accepted / step	ms/token
Community build as published (no `mtp` in `ignore`)	pile-10k (English)	0.0 % — drafter loads mismatched weights, every draft rejected	0.00	30.0 (1.5× slower than MTP off)
Same community build, `ignore` patched	pile-10k (English)	57.9 %	1.16	14.2

Draft acceptance measures how closely the quantized target agrees with the BF16 draft head on this traffic — a direct, task-relevant fidelity signal. All three rows produce correctly-tagged replies (54/54); the difference shows up in speed, not in surface correctness.

Prefix caching on a hybrid GDN model

Qwen3.8-27B is 16 full-attention + 48 Gated-DeltaNet layers. vLLM keeps attention KV pages and GDN recurrent-state pages in one pool with equal page size, so the attention block size is forced to ceil(GDN state bytes / KV bytes per token): with the default fp32 SSM state that is 784 tokens for this model, and the prefix cache can only reuse whole blocks. Consequences and fixes, all measured above:

A 1449-token prompt reuses only 784 tokens (51 % hit) → TTFT 272 ms. --mamba-ssm-cache-dtype bfloat16 halves the state → 400-token blocks → 79 % hit, TTFT 120 ms; no measurable output change on this workload.
Size the fixed part of your prompt to just past a block multiple (e.g. ≥ 1568 tokens for 784-blocks) so it is fully cached and only the per-turn tail is recomputed → TTFT 67–75 ms.
MTP + prefix cache: Qwen3.5-family models only support --mamba-cache-mode align (both 0.24 and 0.27.1 raise on all), which caches GDN state only when a scheduler step ends exactly on a block boundary. With speculative decoding that rarely happens for a short prompt: hit rate drops to 0 % on a 1449-token prompt (TTFT 272 → 600 ms), recovers to 46 % with a block-aligned prompt and to 72 % with block-aligned prompt + bf16 state. So: for MTP, always combine it with both tricks; for very short replies with a long prompt, Recipe A (MTP off) is still the fastest.

What is quantized, what is not

Table with columns: Module, Precision, Why
Module	Precision	Why
Text decoder `self_attn.{q,k,v,o}_proj` (16 full-attention layers)	INT4 sym g128	—
Text decoder `mlp.{gate,up,down}_proj` (64 layers)	INT4 sym g128	—
Gated-DeltaNet `linear_attn.in_proj_qkv / in_proj_z / out_proj` (48 layers)	INT4 sym g128	large projections, quantize cleanly
Gated-DeltaNet `linear_attn.in_proj_a`, `in_proj_b`

Group size 128 rather than 32/64 is deliberate: for short-reply serving the decode loop is memory-bandwidth-bound; g128 reads ~9 % fewer bytes per token than g32 and lands on the most battle-tested Marlin path.

Calibration

The quantizer only ever sees activations; calibration data decides where rounding error is spent, it is not memorised. 530 sequences of ≥ 2048 tokens were rendered through the model's own chat template with enable_thinking=False (empty think block — exactly what a thinking-off deployment sees):

Table with columns: Slice, Share, Content
Slice	Share	Content
Production-style dialogue	~60 %	a ≈1.4 k-token Chinese persona-style system prompt + 10–20 turns of colloquial spoken Chinese (real ASR transcripts incl. disfluencies) with short assistant replies ending in an `[emotion:X]` tag
System-less dialogue	~20 %	the same turn material without a system prompt (diversifies token positions)
General chat	~20 %	ShareGPT-GPT4 (zh/en) and UltraChat (en) multi-turn, same rendering

No benchmark test sets were used for calibration. The private dialogue transcripts are not distributed with this model; nothing from them is embedded in the weights.

Reproduce

bash
pip install "auto-round==0.14.2" "transformers>=5.15" compressed-tensors flash-linear-attention
auto-round --model Qwen/Qwen3.8-27B \
  --scheme W4A16 --bits 4 --group_size 128 \
  --iters 200 --nsamples 384 --seqlen 2048 --batch_size 4 \
  --dataset "/path/to/your_dialogue_calib.jsonl:apply_chat_template=false" \
  --ignore_layers "model.visual,linear_attn.in_proj_a,linear_attn.in_proj_b,re:.*mtp.*" \
  --format llm_compressor --enable_torch_compile --output_dir out/

Two things AutoRound 0.14.2 does not do for you and which decide whether MTP works: (1) local .jsonl calibration data must be passed with the :apply_chat_template=false suffix on this multimodal architecture, and every sample must be ≥ seqlen tokens (shorter ones are dropped silently); (2) after export, add re:.*mtp.* to quantization_config.ignore in config.json (the 15 BF16 mtp.* tensors are copied for you into model_extra_tensors.safetensors, but without the ignore entry vLLM's drafter expects packed INT4 weights, loads nothing useful, and speculative decoding runs at 0 % acceptance — slower than off).

Intended use & limitations

Built for real-time voice/text agents: short replies, fixed persona prompts, thinking disabled, single or few concurrent streams. It is a drop-in for any Qwen3.8-27B use, but calibration effort went into that regime; long chain-of-thought and > 32 k-context behaviour were not evaluated by us beyond the base model's own claims.
Emotion tags: no tag format is required; the calibration set happens to use [emotion:X] and a short-reply style, so those behave stably under 4-bit. Use whatever contract you like.
Not evaluated on academic benchmarks by us; independent reports of AutoRound W4A16 g128 on this model family show near-BF16 MMLU/GSM8K-class scores, but verify for your workload.

Acknowledgements

Qwen team for Qwen3.8; Intel for AutoRound; the vLLM / llm-compressor / compressed-tensors maintainers for the serving stack.

中文简介

面向极低延迟中文口语交流的 Qwen3.8-27B AWQ 级 INT4(W4A16,AutoRound 算法、compressed-tensors 格式、vLLM Marlin 内核)量化,和 Hub 上其它 4-bit 版本的区别不在"多少 bit",而在三件被量出来的事:

校准料是中文口语对话、thinking 关、带情绪标签的线上格式,不是英文网页/代码/长推理链。同配方、同一颗 bf16 草稿头下,MTP 草稿接受率 73.1 % vs 57.9 %(pile-10k 校准的社区版)—— 接受率就是"量化后模型在这个域上离 bf16 有多近"。
MTP 草稿头完整保留且在 vLLM 里实测可用(ignore 里带 re:.*mtp.*,少了这一条草稿头会悄悄失效、比不开还慢 1.5×)。K=2:decode 20.4 → 13.7 ms/token。
给出了让混合 GDN 架构前缀缓存真正命中的部署配方:--mamba-ssm-cache-dtype bfloat16 + 把固定 system 段凑到缓存块的整数倍 → 同一份权重、同样的输出,单轮 634 ms → 415 ms(TTFT 67 ms),在 RTX 6000 Ada 上追平 Qwen3.6-35B-A3B(MoE)的端到端延迟。

视觉塔、GDN 门控、lm_head 全 BF16,图像/视频输入照常;Apache-2.0。适合仿生人/语音助手/陪伴对话等"说话短、要快、要稳"的场景。

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Twu31

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality