Quantization details
Table with columns: Property, Value| Property | Value |
|---|
| Method | RTN (round-to-nearest), data-free (llmcompressor.model_free_ptq) |
| Precision | W4A16 (4-bit integer weights, 16-bit activations) |
| Strategy | group, group_size = 32 |
| Symmetric | true |
| Format | pack-quantized (compressed-tensors) |
| Quantized modules | routed MoE experts only (*.mlp.experts.*.{gate,up,down}_proj), incl. the MTP module |
| Left in BF16 | attention (self_attn + linear_attn/Gated DeltaNet), shared expert, router gates, vision tower, lm_head, embeddings, norms |
This mirrors Kimi's config: num_bits=4, type=int, strategy=group, group_size=32, symmetric=true, format=pack-quantized,
quantizing the MoE components while keeping attention and the shared expert at full precision.
Result: ~72 GB (BF16) → ~23 GB on disk.
Usage (vLLM)
vllm serve casperhansen/Qwen3.6-35B-A3B-INT4-RTN \
--tensor-parallel-size 1 \
--max-model-len 8192 \
--reasoning-parser qwen3
# add --language-model-only for text-only serving
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="casperhansen/Qwen3.6-35B-A3B-INT4-RTN",
messages=[{"role": "user", "content": "Give me a fun fact about the ocean."}],
max_tokens=256, temperature=0.7, top_p=0.8,
)
print(resp.choices[0].message.content)
Verified to load and generate on a single NVIDIA H200 with vLLM (INT4 WNA16 MoE Marlin path).
Reproduction
from compressed_tensors.config import CompressionFormat
from compressed_tensors.quantization import (
QuantizationArgs, QuantizationScheme, QuantizationStrategy, QuantizationType,
)
from llmcompressor import model_free_ptq
scheme = QuantizationScheme(
targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
weights=QuantizationArgs(
num_bits=4, type=QuantizationType.INT, strategy=QuantizationStrategy.GROUP,
group_size=32, symmetric=True, observer="minmax", dynamic=False,
),
format=CompressionFormat.pack_quantized.value,
)
model_free_ptq(
model_stub="Qwen/Qwen3.6-35B-A3B",
save_directory="Qwen3.6-35B-A3B-INT4-RTN",
scheme=scheme,
max_workers=8,
device="cuda:0",
)
License
Apache-2.0, inherited from the base model Qwen/Qwen3.6-35B-A3B.
