Quantization Details

Layers kept in BF16

Certain layers are excluded from quantization to preserve model quality and ensure compatibility with tensor-parallel inference:

The quantization_config.json includes both checkpoint-level (in_proj_a, in_proj_b) and vLLM fused-module (in_proj_ba) FP override entries so that vLLM correctly identifies these layers as unquantized under tensor parallelism.

Quantization summary

• Quantized: 30,850 / 31,181 linear layers
• Block-wise loss (representative): layer 0 loss 0.000011→0.000001, layer 39 loss 0.002693→0.001283

Model size

How to use

With transformers + gptqmodel

python
Copy code
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "nchapman/Qwen3.6-35B-A3B-int4-AutoRound",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("nchapman/Qwen3.6-35B-A3B-int4-AutoRound")

messages = [{"role": "user", "content": "Explain mixture-of-experts architectures briefly."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Note: gptqmodel must be installed for the W4 dequantization kernels. Without it, transformers will silently produce garbage output.

With vLLM

bash
Copy code
vllm serve nchapman/Qwen3.6-35B-A3B-int4-AutoRound \
    --dtype bfloat16 \
    --quantization auto_round \
    --num-speculative-tokens 1 \
    --speculative-model [draft_model]

The MTP head is preserved in BF16, enabling vLLM's speculative decoding with the native MTP draft. Tensor parallelism (TP>=2) works correctly thanks to the in_proj_ba FP overrides.

Reproducing this quantization

Using AutoRound's Python API:

python
Copy code
from auto_round import AutoRound
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.6-35B-A3B",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-35B-A3B")

autoround = AutoRound(
    model,
    tokenizer,
    bits=4,
    group_size=128,
    sym=True,
    iters=400,
    nsamples=256,
    seqlen=2048,
    batch_size=8,
    dataset="NeelNanda/pile-10k",
    ignore_layers=[
        "mtp.fc",
        "linear_attn.in_proj_a",
        "linear_attn.in_proj_b",
        "shared_expert",
    ],
)
autoround.quantize()
autoround.save_quantized("Qwen3.6-35B-A3B-int4-AutoRound", format="auto_round")

The ignore_layers patterns keep in_proj_a and in_proj_b in BF16. AutoRound's saved quantization_config.json will contain the checkpoint-level overrides; the vLLM fused-module entries (*.linear_attn.in_proj_ba) need to be added as a post-save step — our quantization script does this automatically, but if using the raw API you'll need to add {"bits": 16, "data_type": "fp"} entries for each *.linear_attn.in_proj_ba to extra_config in both quantization_config.json and config.json for TP>=2 compatibility.

Acknowledgments

• Quantization method: Intel AutoRound — SignRound: optimization-based weight rounding for post-training quantization
• MTP preservation approach informed by Lorbus/Qwen3.6-27B-int4-AutoRound
• Shared-expert BF16 recipe from Intel/Qwen3.5-122B-A10B-int4-AutoRound
• GDN TP fix validated against vLLM's Marlin kernel constraints