Quantization scheme
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Method | GPTQ with an AWQ pre-pass (vendor recipe) |
| Precision | W4A16 — 4-bit weights, 16-bit activations |
| Group size | 32 |
| Symmetry | asymmetric (with zero-point) |
| Ordering | actorder=static |
| AWQ | duo_scaling: both, n_grid: 10 |
| Serialization | pack-quantized (compressed-tensors) |
| Ignored layers | lm_head, vision tower (model.visual.*) |
| Kernel in vLLM | Marlin (MarlinLinearKernel) |
The exact recipe is stored next to the weights in recipe.yaml; the effective
quantization config is embedded in config.json (quantization_config).
What was done, step by step
- Base model. Started from the Claude-Opus-reasoning distill of Qwen3.5-4B
(
Qwen3_5ForConditionalGeneration, a hybrid model with 8 of 32 layers in full
attention and the rest in gated linear attention, GDN). Loaded in float16.
- Environment.
llm-compressor with transformers >= 5 (the Qwen3.5
recipes require the 5.x branch). The GPU was freed from other services first —
a 4.5B model does not fit alongside a running inference stack on a 16 GB card.
- Calibration set. 256 child chunks (400–4000 characters) sampled from the
target production corpus rather than a generic dataset, so that the GPTQ/AWQ
statistics match the documents the model actually sees.
- AWQ pre-pass. Channel-wise scaling factors are searched per layer
(
AWQModifier, duo_scaling="both", n_grid=10), balancing the weight
groups before quantization. This step is the expensive one — roughly 4.6
minutes per layer, about 2 h 23 min for the whole model.
- GPTQ. Weights are quantized to INT4 with group size 32, asymmetric
(zero-point) scales, and
actorder=static, so that the most salient input
channels are quantized first. Applied to every Linear layer except lm_head
and the vision tower.
- Save and check. The result is a single ~4.0 GB
model.safetensors plus
config.json in compressed-tensors format. Note that the MTP head
(mtp.* tensors present in the base model) is not preserved by this recipe.
- Acceptance on our own set. The checkpoint was loaded by vLLM, and a
116-question golden set of real user questions was run against it, followed by
an assertion check on the answers.
Results
- Size: ~4.0 GB on disk (one
model.safetensors) against ~5.3 GB in float16;
weights in VRAM ~3.33 GiB against ~4.48 GiB.
- Speed: on a 16 GB card, decode grew from 120 to 152–159 tokens/s; the KV
cache pool grew by about a third.
- Quality: 28 of 32 assertions on our golden set — the same score as the
plain GPTQ arm. Three failures are stable and unrelated to quantization (date
extraction from one specific document, and one total-sum case); a fourth
fluctuates between two cases and is vLLM decode nondeterminism.
- Note on linear attention. Unlike the vendor convention (which keeps the
whole
linear_attn block in 16-bit), this export quantizes it too: 1.88 GiB →
0.55 GiB, which is where most of the speed-up comes from. It is a deliberate
deviation, taken on measurements, not on the vendor's ignore list.
Usage
Serve with vLLM (the checkpoint is already in the format vLLM expects):
vllm serve <this-repo> \ --served-model-name qwen3.5-4b-distill-w4a16 \ --max-model-len 16384 \ --gpu-memory-utilization 0.85
The model needs a GPU with compute capability ≥ 75: the Marlin kernel used for
W4A16 does not support older architectures (V100 and below).
Files
Table with columns: File, Purpose| File | Purpose |
|---|
model.safetensors | Quantized weights (~4.0 GB) |
config.json | Model config with the embedded quantization_config |
recipe.yaml | Exact llm-compressor recipe used for this export |
tokenizer.json, tokenizer_config.json, processor_config.json | Tokenizer and processor |
chat_template.jinja | Chat template |
generation_config.json | Default generation parameters |
License
Apache 2.0, inherited from the base model. Please refer to the base model card
for the original terms and for the provenance of the distillation data.