Twu31

Qwen3.8-27B-AWQ-INT4-MTP-LowLatency

README

License: apache-2.0

Quick start (vLLM ≥ 0.17)

Recipe A — shortest turn latency for chat with a long fixed system prompt

bash
vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency \
  --served-model-name qwen38 --max-model-len 12288 --max-num-seqs 8 \
  --enable-prefix-caching --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

Size the fixed part of your prompt to a multiple of the cache block (see Prefix caching) → TTFT 67 ms, 415 ms per 18-token turn on one RTX 6000 Ada.

Recipe B — longer replies (≳ 25 tokens) or short prompts: add MTP

bash
vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency \
  --served-model-name qwen38 --max-model-len 12288 --max-num-seqs 8 \
  --enable-prefix-caching --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(model="qwen38", max_tokens=128, temperature=0.7,
    messages=[{"role": "system", "content": "You are a concise, friendly assistant."},
              {"role": "user", "content": "I've had a long day. Any quick tip to unwind?"}])
print(r.choices[0].message.content)

No --quantization flag needed; vLLM logs Using MarlinLinearKernel for CompressedTensorsWNA16.
Check MTP is really accepting: curl :8000/metrics | grep spec_decode → accepted/draft ≈ 0.6–0.75 on chat. A drafter that accepts 0 % is slower than MTP off.
Don't pass --calculate-kv-scales on hybrid GDN models (vllm#37554). SGLang: untested.
Tokenizer and config files are byte-identical to upstream Qwen/Qwen3.8-27B.

Latency (measured)

Conversational prompts (≈1.6 k-token system prompt + 2 turns of history + a user utterance), greedy, thinking off, 18 prompts × 3 runs, streamed over HTTP, medians; "ms/tok" = inter-token latency. The headline numbers are for ~20-token replies; from the measured TTFT and per-token rates, a 40–60-token reply lands at roughly 0.8–1.1 s on the RTX 6000 Ada and 0.45–0.6 s on the RTX 5090 with MTP on.

Serving-recipe ladder — RTX 6000 Ada 48 GB, vLLM 0.24.0, MTP off

Table with columns: Configuration, Prompt tok, Cache hit, TTFT, Turn, ms/tok
Configuration	Prompt tok	Cache hit	TTFT	Turn	ms/tok
default (`--enable-prefix-caching`)	1449	51 %	272 ms	634 ms	20.4
+ `--mamba-ssm-cache-dtype bfloat16`	1449	79 %	120 ms	468 ms	20.1

Same weights and identical outputs in every row — the 634 → 415 ms is purely cache behaviour.

MTP — RTX 6000 Ada, vLLM 0.24.0: K=0 20.4 ms/tok → K=1 15.9 (79 % acceptance) → K=2 13.7 (73 %, 1.49×) → K=3 13.5 (61 %). With K=2 + bf16 state + block-aligned prompt: TTFT 205 ms, 455 ms/turn, 72 % cache hit — MTP overtakes Recipe A once replies exceed ~25 tokens. Calibration data matters here: the same recipe calibrated on pile-10k reaches 57.9 % acceptance, on conversational transcripts 73.1 %; a community export without mtp in ignore loads a broken drafter and runs at 0 % acceptance, 1.5× slower than MTP off.

Head precision, fidelity and concurrency — RTX 5090 32 GB, vLLM 0.27.1 (NLL = per-token negative log-likelihood of the BF16 base's own greedy answers under the quantized model; lower = closer to BF16)

Table with columns: Variant, MTP off TTFT / turn / ms·tok, MTP K=2 TTFT / turn / ms·tok, K=2 accept, NLL, Repetition / vision / format probes
Variant	MTP off TTFT / turn / ms·tok	MTP K=2 TTFT / turn / ms·tok	K=2 accept	NLL	Repetition / vision / format probes
INT4 body, BF16 lm_head	80 / 300 / 11.6	145 / 299 / 8.7	64.6 %	0.327	all pass
INT4 + INT8 lm_head — `main`	78 / 281 / 10.8	140 / 274 / 7.6	64.4 %	0.326

The levers that move decode latency cut bytes per token: INT8 lm_head −13 %, INT4 lm_head −20 %, INT4 MTP head −8 % (MTP on). Concurrency (5 streams, MTP off): 394 ms/turn, 253 tok/s aggregate; with MTP K=2: 750 ms/turn — MTP is a single-stream optimisation, turn it off for multi-user endpoints.

Prefix caching on a hybrid GDN model

Qwen3.8-27B has 16 attention + 48 Gated-DeltaNet layers. vLLM forces the attention block size to ceil(GDN state bytes / KV bytes per token) — 784 tokens with the default fp32 state — and the prefix cache only reuses whole blocks:

--mamba-ssm-cache-dtype bfloat16 halves the state → 400-token blocks (51 % → 79 % hit, no output change on this workload);
size the fixed prompt just past a block multiple so only the per-turn tail is recomputed (→ 67–75 ms TTFT);
MTP + prefix cache: Qwen3.5-family only supports --mamba-cache-mode align, which stores GDN state only when a step ends on a block boundary — with speculation the hit rate can fall to 0 % unless you use both tricks above (then 72 %).

Which algorithm

Same base model, same calibration pool, same harness, same box (RTX 5090, vLLM 0.27.1); all rows with INT8 lm_head:

Table with columns: Algorithm (tool), Calibration, MTP off TTFT / turn / ms·tok, K=2 accept, NLL mean / median, Verbatim / first-sentence match with BF16 (of 18)
Algorithm (tool)	Calibration	MTP off TTFT / turn / ms·tok	K=2 accept	NLL mean / median	Verbatim / first-sentence match with BF16 (of 18)
AutoRound 0.14.2, 200 iters — `main`	384 × 2048	112 / 293 / 10.6	66.6 %	0.325 / 0.320	8 / 11
AWQ (llm-compressor 0.13.0) — `awq-llmcompressor`	256 × 2048	114 / 307 / 10.6	64.7 %

Serving speed is identical (same tensors, same kernel). Fidelity to BF16 is best for AutoRound, GPTQ close on NLL, AWQ slightly behind (a few prompts raise its mean; its median is in line; llm-compressor's AWQ mappings leave o_proj / linear_attn.out_proj unscaled). Treat the ordering as indicative — 18 prompts, coarse verbatim counts, flat MTP acceptance — not as a benchmark. (This box reads ~30 ms higher TTFT than the one used in the table above for the same main weights; compare within one table.)

What is quantized, what is not

Table with columns: Module, Precision, Why
Module	Precision	Why
`self_attn.{q,k,v,o}_proj` (16 layers), `mlp.{gate,up,down}_proj` (64), `linear_attn.in_proj_qkv / in_proj_z / out_proj` (48)	INT4 sym g128	bulk of the bytes
`linear_attn.in_proj_a`, `in_proj_b` (48-wide gates)	BF16	4-bit hurts the recurrence, saves nothing, breaks Marlin under TP
Vision tower `model.visual.*`	BF16	keeps image/video quality

Sizes: main 18.2 GB · lm-head-int4 17.6 GB · mtp-int4-autoround 19.0 GB · branches 18.2 GB. A 24 GB card serves text with a modest KV budget; 32 GB+ is comfortable at 12 k context.

Calibration & reproduce

530 sequences ≥ 2048 tokens rendered through the model's chat template with thinking off: ~60 % multi-turn conversational transcripts (assistant-persona system prompt + short colloquial turns, predominantly Chinese), ~20 % the same turns without a system prompt, ~20 % ShareGPT-GPT4 / UltraChat (zh/en). No benchmark test sets; the transcripts are not distributed and nothing from them is embedded in the weights. If your traffic differs (code, long reasoning), re-run the recipe on your own data — that is what moved MTP acceptance from 58 % to 73 % here.

bash
# AutoRound (main)
auto-round --model Qwen/Qwen3.8-27B --scheme W4A16 --bits 4 --group_size 128 \
  --iters 200 --nsamples 384 --seqlen 2048 --batch_size 4 \
  --dataset "/path/calib.jsonl:apply_chat_template=false" \
  --ignore_layers "model.visual,linear_attn.in_proj_a,linear_attn.in_proj_b,re:.*mtp.*" \
  --format llm_compressor --enable_torch_compile --output_dir out/      # 1 h 14 min on one RTX 6000 Ada

python
# AWQ / GPTQ (llm-compressor 0.13.0)
ignore = ["re:.*visual.*", "lm_head", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$"]
recipe = [AWQModifier(duo_scaling=True), QuantizationModifier(ignore=ignore, scheme="W4A16", targets=["Linear"])]   # AWQ
# recipe = [GPTQModifier(ignore=ignore, scheme="W4A16", targets=["Linear"], dampening_frac=0.01, actorder="weight")]  # GPTQ
oneshot(model=model, processor=tokenizer, dataset=ds, recipe=recipe, max_seq_length=2048, num_calibration_samples=256)
model.save_pretrained(out, save_compressed=True, max_shard_size="4GB")

Three things the exporters do not do for you: (1) keep the 15 BF16 mtp.* tensors in the checkpoint and add re:.*mtp.* to quantization_config.ignore — without both, vLLM's drafter loads garbage and MTP runs at 0 %; (2) copy the upstream tokenizer files back verbatim — save_pretrained in transformers 5.14/5.15 re-serialises tokenizer.json with an older pre-tokenizer regex (drops \p{M}, changes tokenization of scripts with combining marks); (3) the INT8 lm_head is applied post-hoc (RTN g128 via compressed_tensors, a group_1 scheme targeting re:.*lm_head$) because AutoRound's --quant_lm_head needs the 248 k-vocab logits in memory.

Revisions

Table with columns: Revision, Content, Load
Revision	Content	Load
`main`	INT4 body (AutoRound) + INT8 lm_head, MTP head BF16	compressed-tensors, zero-config (vLLM 0.24.0 / 0.27.1 verified)
`lm-head-int4`	INT4 body (AutoRound) + INT4 lm_head (fastest decode)	compressed-tensors
`mtp-int4-autoround`	INT4 body (AutoRound) + INT4 MTP head (`mtp.fc` BF16), lm_head BF16	`auto_round:auto_gptq` packing, vLLM `inc` backend, no flag

vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency --revision lm-head-int4 .... All revisions carry upstream-identical tokenizer/config files (earlier uploads had a transformers-resaved tokenizer.json).

Limitations

Built and measured for short-reply, thinking-off, single/few-stream interactive serving; long chain-of-thought and > 32 k-context behaviour were not evaluated beyond the base model's claims. No academic benchmarks were run; the fidelity numbers above are relative to the BF16 base on conversational prompts.
MTP roughly doubles per-turn latency at 5 concurrent streams; FP8 KV cache is not worth it here (only 16 of 64 layers keep KV).
vLLM 0.27.1 on Python 3.10 needs from __future__ import annotations at the top of flashinfer/comm/fd_exchange.py; fresh machines need g++ and python3-dev for the JIT kernels.

Acknowledgements

Qwen team for Qwen3.8; Intel for AutoRound; the vLLM / llm-compressor / compressed-tensors maintainers.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Twu31

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality