antonyMox

Qwen3.6-27B-AutoRound-INT4-wide-MTP

README

License: apache-2.0

Highlights

Benchmarks — the wide trick pays off

Same clean base Qwen/Qwen3.6-27B, one harness — only the quant recipe changes. Keeping the big SSM projections in BF16 lifts hard +8 pp over the standard narrow quant:

Table with columns: Quant of Qwen3.6-27B (our harness), Recipe, hard, tools, Size
Quant of Qwen3.6-27B (our harness)	Recipe	hard	tools	Size
Lorbus narrow INT4 (reference)	narrow	70	83	18.5 GB
abliterated sister	narrow + Heretic	70	87	19 GB
➡️ wide (this)	wide	78 🏆	83	26 GB
35B-A3B (coming soon)	MoE narrow	48	83	22 GB

+8 pp hard for +7.5 GB over narrow — and it beats Qwen3-Coder-Next 80B (hard 58), three times its size.

[!NOTE] Numbers are % of tasks passed (40 hard coding + 30 tool tasks; raw pass/total in benchmark-results/). Conditions: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 · seed 42 · non-thinking · vLLM 0.19 · RTX 4090 (48 GB) · MTP n=3. The abliterated & wide rows share the exact 40 hard tasks the narrow quant scored 70 on → clean apples-to-apples. The Lorbus row is our own measurement of their public quant, not their claim. Reproduce it, don't trust it.

🔬 The "wide" trick — why it scores higher

Qwen3.6 is a hybrid model: most layers are linear-attention (GDN/Mamba), where long-range logic flows through the large projections in_proj_qkv, in_proj_z, out_proj. Standard "narrow" INT4 quants compress these to 4-bit too — and that quietly costs reasoning. We keep them in BF16:

Table with columns: Kept in BF16 (not quantized), Tensors, Why
Kept in BF16 (not quantized)	Tensors	Why
🔬 Big SSM projections (the wide trick)	`in_proj_qkv`, `in_proj_z`, `out_proj`	carry long-range logic — INT4 here costs ~8 pp hard
🌊 Small SSM control	`A_log`, `conv1d`, `dt_bias`, `in_proj_a/b`, `norm`	SSM recurrence stability
🧬 MTP head

Everything else (full-attention q/k/v/o, MLP gate/up/down) is INT4. The recipe was validated on Qwen3.5 first (+6 pp) and confirmed here on 3.6 (+8 pp).

⚡ Speed — MTP speculative decoding

Honest 768-token × 3-run measurement on a single RTX 4090 (48 GB), vLLM 0.19:

Table with columns: MTP n, short tok/s, 28k-ctx tok/s, acceptance @28k
MTP `n`	short tok/s	28k-ctx tok/s	acceptance @28k
1	55	33	79%
2	71	39	64%
3 ⭐	83	40	52%
4	92	42

Use num_speculative_tokens: 3 — long-context throughput is already on its plateau (~40 tok/s) while acceptance stays highest (52%), so the draft head misfires least. Spec-decode is lossless — n changes speed only, never output.

🖼️ Multimodal — sees images & video

The full vision-language model. The entire vision tower (333 tensors) is kept in full BF16 — quantization never touched it — so image/video understanding is identical to the pristine base. Serve with --limit-mm-per-prompt '{"image":4,"video":1}'. Published benchmarks cover text/coding/tool-use; vision quality is inherited from the base, not separately re-scored.

🚀 Quickstart — this is a big-card model

Weights are 26 GB, so this is a big-card model. A 48 GB card is its home; a 32 GB card runs it but with only a small context window; 24 GB can't load it at all.

Table with columns: GPU, VRAM, Script, max-model-len, gpu-mem-util, Status
GPU	VRAM	Script	`max-model-len`	`gpu-mem-util`	Status
RTX 4090 48G / A6000 / L40S / RTX 6000	48 GB	`run_48gb.sh`	150 000	0.72	✅ tested (benchmark rig)
RTX 5090	32 GB

[!NOTE] 32 GB is tight. The 26 GB weights leave only ~1.7 GB for KV even at mem-util 0.95, so the safe window is ~16k tokens (measured by emulating a 32 GB budget on a 48 GB card: concurrency ~1.5x at 16k; 24k dropped to 1.03x = the hang zone). mem 0.95 assumes a dedicated/headless card — lower it if a display is attached. Want long context on ≤ 32 GB? Use the 19 GB abliterated sister (fits 24 GB with 40k ctx, or 32 GB with much more).

bash
vllm serve antonyMox/Qwen3.6-27B-AutoRound-INT4-wide-MTP \
  --quantization auto-round \
  --max-model-len 150000 \
  --gpu-memory-utilization 0.72 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --trust-remote-code

Long context is cheap: Qwen3.6 is a hybrid (Mamba/GDN + attention), KV costs ~37 MB/1k tokens at fp8. The model's native maximum is 256k (262,144) tokens — how much fits on 48 GB depends on mem-util:

Context vs mem-util (48 GB card, measured)

Table with columns: gpu-mem-util, context that fits, notes
`gpu-mem-util`	context that fits	notes
0.72 ⭐ (default preset)	~150k (conc 1.15x)	balanced; leaves VRAM free
0.80	256k (conc 1.11x)	full native context

Raise --gpu-memory-utilization toward 0.80 for the full 256k; the 0.72 default keeps more VRAM free (this variant's 26 GB weights leave less room than the 19 GB abliterated sister).

[!WARNING] Never set --max-model-len at exact KV capacity. On vLLM 0.19 hybrid models, requests in the last ~3 % of the window pass validation then hang forever (no error, GPU idle) — the scheduler can't allocate the final KV block. Keep the window a couple of blocks below capacity: after launch, the log line Maximum concurrency for N tokens: Yx must be ≥ ~1.10x (ours is 1.15x at 150k), never 1.00x. Also, the GPU KV cache size: X tokens log line understates real capacity ~3× on hybrids — trust only the Maximum concurrency line.

🧬 The family

Clean quants of the same base — pick by hardware:

Table with columns: Model, Size, hard, tools, Best for
Model	Size	hard	tools	Best for
➡️ wide (this)	26 GB	78 🏆	83	48 GB+ cards — maximum reasoning
abliterated	19 GB	70	87	24 GB cards · uncensored
35B-A3B (coming soon)	22 GB	48

The 27B sisters: AutoRound INT4, MTP preserved, multimodal. All benchmarked against the community reference Lorbus narrow INT4 (hard 70) — see the table above.

🍳 Recipe (reproducible)

Base — pristine Qwen/Qwen3.6-27B BF16.
Quant — AutoRound W4A16, group 128, 200 iters, v2 calibration.
Wide fp_layers (kept BF16) — mtp.*, visual.*, small SSM (in_proj_a/b, A_log, conv1d, dt_bias, linear_attn.norm) + the wide trick: in_proj_qkv, in_proj_z, out_proj.
MTP — 15 mtp.* tensors preserved (BF16), n=3 recommended.

🔢 Why HuggingFace shows "~11.5B params"

Not a small model — it's how the widget reads a partially-packed INT4 quant. The INT4 weight tensors are packed 8× int4 into each int32 (HF counts fewer slots), while the large wide projections, vision, and MTP stay full BF16 (counted fully). The true model is Qwen3.6-27B, 26 GB on disk. Verified breakdown: language_model 9.35 B · other 1.27 B · vision 0.46 B (333 t, BF16) · mtp 0.42 B (15 t, BF16). (The wide sister shows more BF16 than a narrow quant precisely because it keeps those big projections full-precision — that's the whole point.)

📜 License & attribution

Apache-2.0, same as the base. Quantized from Qwen/Qwen3.6-27B with a wide AutoRound INT4 recipe. This is the clean base (creator safety alignment intact). Community build; benchmark numbers are from our own harness — JSON included, methodology open. Provided "as is", without warranty.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

antonyMox

Model Tree

Base

Qwen/Qwen3.6-27B

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities