Minachist

Qwen3.6-27B-INT8-Autoround-V2

README

License: apache-2.0

Quantization details

Some layers are not quantized as quantizing them causes a heavy downgrade, or AutoRound simply doesn't support quantizing them.

Table with columns: Precision, Layer Count, Target Layers
Precision	Layer Count	Target Layers
INT8 (group_size=-1, symmetric)	410	`gate/up/down_proj`, `q/k/out_proj`, `in_proj_qkv`, `in_proj_z`, `mtp`
FP16 (unquantized)	98	`in_proj_a` / `in_proj_b` of linear_attn, `embed_tokens`, `visual`

AutoRound doesn't support quantizing in_proj_a and in_proj_b.
Calibration configuration: iters=250, nsamples=1024, batch_size=4, with gradient_accumulate_steps=2.

You need to change a line in utils.py of the AutoRound library

from:

layer_config[match] = val

to: layer_config[match] = copy.deepcopy(val)

in order to quantize linear_attn.

To whoever wants to open a pull request upstream: if you believe this is actually a bug, I encourage you to do so.

Evaluation Results

1. KL Divergence (KLD)

The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.

Table with columns: Metric @ctx 512, Value
Metric @ctx 512	Value
Mean KLD	0.00182 ± 0.00014
Median KLD	0.000672
P90 / P95 / P99 KLD	0.00232 / 0.00349 / 0.00931
P99.9 / Max KLD	0.0668 / 9.156
Top-1 Agreement	98.32%
PPL (Base → Quantized)	7.7415 → 7.7442
ln(PPL Ratio)	0.000343

Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats.

I heard quantizing linear_attn impacts the performance quite badly, but that's not the case here. Maybe not quantizing in_proj_a / in_proj_b is the reason.

2. Benchmarks

Measured via lm-evaluation-harness (0.4.12) using the vLLM OpenAI-compatible API.

Settings: 0-shot, temperature=0.6, top_p=0.95, max generation tokens 65,536.
Comparison baseline: Official metrics from the Qwen3.6-27B model card.

Table with columns: Benchmark, This Model, Base Model (Official), Difference, n
Benchmark	This Model	Base Model (Official)	Difference	n
MMLU-Redux	91.9% ± 1.6	93.5	-1.6	285
MMLU-Pro	83.8% ± 1.8	86.2	-2.4	420

Performance drops in knowledge-focused benchmarks (MMLU-Redux, MMLU-Pro) are minor and lie mostly within the standard error. I wanted to run more benchmarks, but it's quite hot here and AC doesn't seem to work well. You have to do it yourself.

How to Use

This model has been tested using the latest vllm-openai:nightly.

On a 2x RTX 3090 setup, you can get 204,800 context without kv-cache quantization.

You can append --kv-cache-dtype fp8 to optimize memory further.

Acknowledgements

Alibaba / Qwen team for the base Qwen3.6-27B model.
Intel AutoRound team for the quantization framework.
vLLM project for the inference engine and Qwen3.x MTP support.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Minachist

Model Tree

Base

Qwen/Qwen3.6-27B

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities