Minachist

Qwen3.6-27B-Mixed-AutoRound

README

License: apache-2.0

Quantization details

This model uses a mixed-precision quantization to balance performance and model size.
The self_attn layers are quantized to 8-bit.
The MLP layers are generally quantized to 4-bit, but the first 3 and last 3 layers are kept at 8-bit.
The lm_head, linear_attn, visual, mtp.fc layers are kept unquantized in FP16.

Table with columns: Field, Custom Mixed Recipe
Field	Custom Mixed Recipe
Base	`Qwen/Qwen3.6-27B`
Method	AutoRound (`intel/auto-round`), custom recipe
Scheme	Mixed (W4A16 / W8A16)
Bits	4 & 8
Group size	128
Symmetric	yes
Unquantized layers	`lm_head`, `linear_attn`, `visual`, `mtp.fc`
Calibration dataset	`NeelNanda/pile-10k`
Calibration samples	512
Sequence length	2048
Iterations	1000
Batch size	8
torch.compile	enabled

For more information, please check quantize.py.

KLD Metrics

KLD metrics are updated since I managed to run KLD test more precise.

Table with columns: Metric, Value, Description
Metric	Value	Description
Median KLD	0.005331	Median divergence
P90 KLD	0.031100	Divergence at the 90th percentile
Mean KLD	0.030175	Average divergence
Mean Coverage	1.000000	Full-vocabulary reference (exact KLD)

Evaluation Configuration

Table with columns: Parameter, Value
Parameter	Value
Evaluation Dataset	wikitext-2-raw-v1 (test)
Sequence Length	2048
Num Samples	60
Total Positions	122,820
Top-K Reference	Full vocabulary (248,320)

Additional Metrics (optional)

Table with columns: Metric, Value
Metric	Value
Mean KLD (±SE)	0.030175 ± 0.001174
P95 / P99 KLD	0.057433 / 0.238007
Top-1 Agreement	94.77%
PPL (BF16 → quant)	6.4060 → 6.4432
ln(PPL ratio)	0.005794
RMS Δp (actual token)	0.044628

How to use

This model is tested on the latest docker.io/vllm/vllm-openai:nightly, using podman.
vLLM is recommended.
Example args (For 2x 3090 Users) :

bash
vllm serve ./Qwen3.6-27B-mixed-autoround \
  --tensor-parallel-size 2 \
  --performance-mode interactivity \
  --max-model-len auto \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.97 \
  -O3 \
  --async-scheduling \
  --language-model-only \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ # 3 should be slightly faster but uses more VRAM
  --default-chat-template-kwargs.preserve_thinking true \
  --mamba-cache-mode all \
  --enable-prefix-caching \
  --enable-chunked-prefill

With these settings, you get full context.

If you're having quality issues while using MTP, it's likely a vLLM issue since MTP shouldn't degrade the output at all. Try disabling --speculative-config or --enable-prefix-caching for quick fix.

For more information, please see: this reddit thread, and this vLLM pull

Note: This information is based on current understanding and testing. Optimal configurations may vary depending on your specific hardware setup. For further details, please refer to the official vLLM documentation.

Acknowledgements

Lorbus for the README.md format
Alibaba / Qwen team for the base Qwen3.6-27B model
Intel AutoRound team for the quantization framework
vLLM project for the inference engine and Qwen3_5 MTP support

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Minachist

Model Tree

Base

Qwen/Qwen3.6-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality