Minachist

Qwen3.6-27B-INT8-AutoRound

README

License: apache-2.0

Available versions

There are two versions.
Main branch one is a little bit smaller by quantizing self_attn, and disabling group_size at the cost of the model's intelligence.
For users with 48GB VRAM, just using Main branch is recommended. If you have more than that, gs128 branch might be better. The performance difference in practical use is minimal.

Quantization details

Table with columns: Field, Main branch, gs128 branch
Field	Main branch	gs128 branch
Base	`Qwen/Qwen3.6-27B`	`Qwen/Qwen3.6-27B`
Method	AutoRound (`intel/auto-round`), custom recipe	AutoRound (`intel/auto-round`), default recipe
Scheme	W8A16	W8A16
Bits	8	8
Group size	-1	128
Symmetric	yes	yes
Unquantized layers	`visual`, `mtp`, `linear_attn`, `embed_tokens`, `lm_head`	`visual`, `mtp`, self_attn, `linear_attn`, `embed_tokens`, `lm_head`
Calibration samples	128	128
Iterations	1000	200
Batch size	8	8
torch.compile	enabled	enabled
Size	36.8GB	38.8GB
GPU used for quant	2× RTX 3090	2× RTX 3090

For more information, please check quantize.py.

Evaluation Results (KLD)

~~This model's KLD was measured using vLLM because Transformers couldn't load this model at the time. It's good enough for a sanity check, but it's not precise.~~ It's now fixed and re-evaluated.

Lower values indicate less degradation caused by quantization. The main branch is used for the evaluation.

The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.

Table with columns: Metric @ctx 512, Value
Metric @ctx 512	Value
Mean KLD	0.00167 ± 0.00022
Median KLD	0.000567
P90 / P95 / P99 KLD	0.00207 / 0.00297 / 0.00781
P99.9 / Max KLD	0.0490 / 18.72
Top-1 Agreement	98.51%
PPL (Base → Quantized)	7.7415 → 7.7410
ln(PPL Ratio)	-0.000076

Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats.

Quantization log

Please check log.txt.

How to use

This model is tested on latest docker.io/vllm/vllm-openai:cu130-nightly.
vLLM is recommended.
Example args (For 2x 3090 Users) :

text

vllm serve ./Qwen3.6-27B-INT8-AutoRound \  --tensor-parallel-size 2 \  --attention-backend FLASHINFER \  --performance-mode interactivity \  --max-model-len auto \  --max-num-batched-tokens 2048 \  --max-num-seqs 1 \  --gpu-memory-utilization 0.932 \  --compilation-config '{"mode":"VLLM_COMPILE","cudagraph_capture_sizes":[3]}' \  -O3 \  --async-scheduling \  --language-model-only \  --tool-call-parser qwen3_coder \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \  --default-chat-template-kwargs.preserve_thinking true \  --mamba-cache-mode all \  --mamba-block-size 8 \  --enable-prefix-caching \  --enable-chunked-prefill

With these settings, you get around 129k context. You can also add --kv-cache-dtype fp8_e4m3 ~~--calculate-kv-scales~~ args to get about 252k tokens. Edit: --calculate-kv-scales silently corrupts kv cache, so please don't use it for now. For more information, please see this huggingface discussion.
You can add --enforce-eager (you might need to remove --compilation-config) or set the PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False environment variable (requires --disable-custom-all-reduce) to allocate more VRAM to the KV cache, but the tk/s will be noticeably lower.
Remove --speculative-config if you really want more context, but I highly recommend keeping it.
Note: This information is based on my current understanding and testing. Optimal configurations may vary depending on your specific hardware setup. For further details, please refer to the official vLLM documentation.

Acknowledgements

Lorbus for the README.md format
Alibaba / Qwen team for the base Qwen3.6-27B model
Intel AutoRound team for the quantization framework
vLLM project for the inference engine and Qwen3_5 MTP support

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Minachist

Model Tree

Base

Qwen/Qwen3.6-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality