webhie

Qwen3.6-27B-int4-AutoRound-Code

README

License: apache-2.0

TL;DR

Base: Qwen3.6-27B (27B dense VLM)
Quant: INT4 W4A16, group_size 128, symmetric
Tool: auto-round-best (1000 iters, 512 samples, torch.compile)
Calibration dataset: nvidia/OpenCodeInstruct (coding domain)
Size: ~18 GB (down from ~54 GB BF16) — 3× reduction
MTP: Native Multi-Token Prediction head preserved in BF16 — enables native speculative decoding in vLLM (~85–90% draft acceptance, ~2× throughput)
Vision: Image inputs work via the MoonViT encoder (weights kept at original BF16/FP16 precision)

Why code calibration?

AutoRound's algorithm optimizes weight rounding by minimizing the difference between the quantized model's outputs and the full-precision model's outputs on a set of calibration samples. The calibration dataset therefore shapes which activations and weight patterns are prioritized during optimization.

Using a normalized and sampled subset of nvidia/OpenCodeInstruct — a large, execution-verified dataset of coding problems and solutions — means the rounding decisions are tuned for code-style token distributions: identifiers, keywords, indentation patterns, and structured reasoning. In practice this tends to:

Better preserve accuracy on code generation benchmarks relative to a pile-calibrated quant
Improve instruction following for programming tasks (function signatures, docstrings, tool calls)
Retain structured output quality (JSON, markdown code blocks, structured diffs)

If your primary use-case is code generation or an AI coding assistant, this variant is the recommended choice. For general-purpose or multimodal usage, see the standard Qwen3.6-27B-int4-AutoRound quant.

Quick inference with vLLM (with MTP speculative decoding)

Requires vLLM v0.19.1+ with Qwen3_5 MTP support. Set the following environment variables before starting:

bash

export VLLM_USE_FLASHINFER_SAMPLER=1export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1export VLLM_FLOAT32_MATMUL_PRECISION=highexport PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:512"export VLLM_NO_USAGE_STATS=1export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1export VLLM_MARLIN_USE_ATOMIC_ADD=1export OMP_NUM_THREADS=1export CUDA_DEVICE_MAX_CONNECTIONS=8export NCCL_CUMEM_ENABLE=0export NCCL_P2P_DISABLE=1

bash

vllm serve webhie/Qwen3.6-27B-int4-AutoRound-Code \  --served-model-name qwen3.6-27b \  --host 0.0.0.0 --port 11434 \  --trust-remote-code \  --dtype auto \  --quantization auto_round \  --max-model-len 200704 \  --gpu-memory-utilization 0.92 \  --max-num-seqs 4 \  --kv-cache-dtype fp8_e4m3 \  --attention-backend flashinfer \  --performance-mode throughput \  --max-num-batched-tokens 2048 \  --enable-chunked-prefill \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --reasoning-parser qwen3 \  --default-chat-template-kwargs '{"preserve_thinking":true}' \  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \  --enable-prompt-tokens-details \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Remove --speculative-config to disable MTP speculative decoding. See the vllm-blackwell-guide repo for a full Docker Compose setup with all env vars pre-configured.

OpenAI-compatible request

python

from openai import OpenAIclient = OpenAI(base_url="http://localhost:11434/v1", api_key="EMPTY")r = client.chat.completions.create(    model="qwen3.6-27b",    messages=[{"role": "user", "content": "Write a quicksort in Python."}],    max_tokens=512,)print(r.choices[0].message.content)

Transformers (no spec decoding)

python

from transformers import AutoModelForCausalLM, AutoTokenizerm = AutoModelForCausalLM.from_pretrained(    "webhie/Qwen3.6-27B-int4-AutoRound-Code",    trust_remote_code=True,    device_map="auto",)tok = AutoTokenizer.from_pretrained("webhie/Qwen3.6-27B-int4-AutoRound-Code")msg = [{"role": "user", "content": "Write a binary search in Python."}]ids = tok.apply_chat_template(msg, add_generation_prompt=True, return_tensors="pt").to(m.device)print(tok.decode(m.generate(ids, max_new_tokens=256)[0]))

Quantization details

Table with columns: Field, Value
Field	Value
Base	`Qwen/Qwen3.6-27B`
Method	AutoRound (`intel/auto-round`), best recipe
Scheme	W4A16 (4-bit weights, FP16 activations)
Bits	4
Group size	128
Symmetric	yes
Packing format	`auto_round:auto_gptq`

Unquantized layers — why

linear_attn.in_proj_a/b: low-rank projections in Qwen3.6's Gated DeltaNet whose shapes aren't divisible by 32 (group_size), so AutoRound skips them automatically. Tiny fraction of total parameters.
Norms, routers: precision-sensitive and very small — kept at full precision.

Performance

Benchmarked on 1× RTX 5090 (32 GB) with vLLM + FP8 KV cache + MTP n=3:

Table with columns: Config, Throughput
Config	Throughput
vLLM + MTP n=3	~150 tok/s
vLLM (MTP disabled)	~70 tok/s

The ~2× speedup comes from ~85–90% draft acceptance via MTP speculative decoding with num_speculative_tokens: 3.

Reproduction

bash

pip install auto-round # The calibration data was first normalized and sampled from nvidia/OpenCodeInstruct# (formatting cleaned, deduplicated, balanced across domains) and exported as a# local JSON file before quantization. Pass your own prepared subset with:# --dataset ./subset_10k.json auto-round-best \  --model Qwen/Qwen3.6-27B \  --scheme W4A16 \  --format auto_round \  --output_dir Qwen3.6-27B-int4-AutoRound-Code \  --enable_torch_compile \  --low_gpu_mem_usage \  --device_map 0

No post-processing needed — MTP and image inputs work out of the box.

Acknowledgements

Alibaba / Qwen team for the base Qwen3.6-27B model
Intel AutoRound team for the quantization framework and the auto-round-best recipe
NVIDIA for the OpenCodeInstruct calibration dataset — ~5 M execution-verified coding samples used to domain-adapt this quant
Lorbus for the original AutoRound quant of this model that inspired this release
@eugr for the spark-vllm-docker fork and TurboQuant KV cache work

License

Apache 2.0 — same as Qwen3.6-27B base.

Citation

If you use this quant, please cite the original Qwen3.6 release (see base model card), the AutoRound paper, and the OpenCodeInstruct dataset:

bibtex

@article{cheng2023autoround,  title   = {Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},  author  = {Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},  journal = {arXiv preprint arXiv:2309.05516},  year    = {2023}} @misc{nvidia2025opencode,  title  = {OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs},  author = {NVIDIA},  year   = {2025},  url    = {https://huggingface.co/datasets/nvidia/OpenCodeInstruct}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

webhie

Model Tree

Base

Qwen/Qwen3.6-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

TL;DR

Base: Qwen3.6-27B (27B dense VLM)
Quant: INT4 W4A16, group_size 128, symmetric
Tool: auto-round-best (1000 iters, 512 samples, torch.compile)
Calibration dataset: nvidia/OpenCodeInstruct (coding domain)
Size: ~18 GB (down from ~54 GB BF16) — 3× reduction
MTP: Native Multi-Token Prediction head preserved in BF16 — enables native speculative decoding in vLLM (~85–90% draft acceptance, ~2× throughput)
Vision: Image inputs work via the MoonViT encoder (weights kept at original BF16/FP16 precision)

Why code calibration?

Better preserve accuracy on code generation benchmarks relative to a pile-calibrated quant
Improve instruction following for programming tasks (function signatures, docstrings, tool calls)
Retain structured output quality (JSON, markdown code blocks, structured diffs)

Quick inference with vLLM (with MTP speculative decoding)

Requires vLLM v0.19.1+ with Qwen3_5 MTP support. Set the following environment variables before starting:

bash

export VLLM_USE_FLASHINFER_SAMPLER=1export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1export VLLM_FLOAT32_MATMUL_PRECISION=highexport PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:512"export VLLM_NO_USAGE_STATS=1export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1export VLLM_MARLIN_USE_ATOMIC_ADD=1export OMP_NUM_THREADS=1export CUDA_DEVICE_MAX_CONNECTIONS=8export NCCL_CUMEM_ENABLE=0export NCCL_P2P_DISABLE=1

bash

vllm serve webhie/Qwen3.6-27B-int4-AutoRound-Code \  --served-model-name qwen3.6-27b \  --host 0.0.0.0 --port 11434 \  --trust-remote-code \  --dtype auto \  --quantization auto_round \  --max-model-len 200704 \  --gpu-memory-utilization 0.92 \  --max-num-seqs 4 \  --kv-cache-dtype fp8_e4m3 \  --attention-backend flashinfer \  --performance-mode throughput \  --max-num-batched-tokens 2048 \  --enable-chunked-prefill \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --reasoning-parser qwen3 \  --default-chat-template-kwargs '{"preserve_thinking":true}' \  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \  --enable-prompt-tokens-details \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Remove --speculative-config to disable MTP speculative decoding. See the vllm-blackwell-guide repo for a full Docker Compose setup with all env vars pre-configured.

OpenAI-compatible request

python

from openai import OpenAIclient = OpenAI(base_url="http://localhost:11434/v1", api_key="EMPTY")r = client.chat.completions.create(    model="qwen3.6-27b",    messages=[{"role": "user", "content": "Write a quicksort in Python."}],    max_tokens=512,)print(r.choices[0].message.content)

Transformers (no spec decoding)

python

from transformers import AutoModelForCausalLM, AutoTokenizerm = AutoModelForCausalLM.from_pretrained(    "webhie/Qwen3.6-27B-int4-AutoRound-Code",    trust_remote_code=True,    device_map="auto",)tok = AutoTokenizer.from_pretrained("webhie/Qwen3.6-27B-int4-AutoRound-Code")msg = [{"role": "user", "content": "Write a binary search in Python."}]ids = tok.apply_chat_template(msg, add_generation_prompt=True, return_tensors="pt").to(m.device)print(tok.decode(m.generate(ids, max_new_tokens=256)[0]))

Quantization details

Table with columns: Field, Value
Field	Value
Base	`Qwen/Qwen3.6-27B`
Method	AutoRound (`intel/auto-round`), best recipe
Scheme	W4A16 (4-bit weights, FP16 activations)
Bits	4
Group size	128
Symmetric	yes
Packing format	`auto_round:auto_gptq`

Unquantized layers — why

linear_attn.in_proj_a/b: low-rank projections in Qwen3.6's Gated DeltaNet whose shapes aren't divisible by 32 (group_size), so AutoRound skips them automatically. Tiny fraction of total parameters.
Norms, routers: precision-sensitive and very small — kept at full precision.

Performance

Benchmarked on 1× RTX 5090 (32 GB) with vLLM + FP8 KV cache + MTP n=3:

Table with columns: Config, Throughput
Config	Throughput
vLLM + MTP n=3	~150 tok/s
vLLM (MTP disabled)	~70 tok/s

The ~2× speedup comes from ~85–90% draft acceptance via MTP speculative decoding with num_speculative_tokens: 3.

Reproduction

bash

pip install auto-round # The calibration data was first normalized and sampled from nvidia/OpenCodeInstruct# (formatting cleaned, deduplicated, balanced across domains) and exported as a# local JSON file before quantization. Pass your own prepared subset with:# --dataset ./subset_10k.json auto-round-best \  --model Qwen/Qwen3.6-27B \  --scheme W4A16 \  --format auto_round \  --output_dir Qwen3.6-27B-int4-AutoRound-Code \  --enable_torch_compile \  --low_gpu_mem_usage \  --device_map 0

No post-processing needed — MTP and image inputs work out of the box.

Acknowledgements

Alibaba / Qwen team for the base Qwen3.6-27B model
Intel AutoRound team for the quantization framework and the auto-round-best recipe
NVIDIA for the OpenCodeInstruct calibration dataset — ~5 M execution-verified coding samples used to domain-adapt this quant
Lorbus for the original AutoRound quant of this model that inspired this release
@eugr for the spark-vllm-docker fork and TurboQuant KV cache work

License

Apache 2.0 — same as Qwen3.6-27B base.

Citation

If you use this quant, please cite the original Qwen3.6 release (see base model card), the AutoRound paper, and the OpenCodeInstruct dataset:

bibtex

@article{cheng2023autoround,  title   = {Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},  author  = {Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},  journal = {arXiv preprint arXiv:2309.05516},  year    = {2023}} @misc{nvidia2025opencode,  title  = {OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs},  author = {NVIDIA},  year   = {2025},  url    = {https://huggingface.co/datasets/nvidia/OpenCodeInstruct}}