bumbuk

Qwen3.5-4B-Distill-w4a16-g32-asym-awq-gptq-actorder

README

License: apache-2.0

Quantization scheme

Table with columns: Parameter, Value
Parameter	Value
Method	GPTQ with an AWQ pre-pass (vendor recipe)
Precision	W4A16 — 4-bit weights, 16-bit activations
Group size	32
Symmetry	asymmetric (with zero-point)
Ordering	`actorder=static`
AWQ	`duo_scaling: both`, `n_grid: 10`
Serialization	`pack-quantized` (compressed-tensors)
Ignored layers	`lm_head`, vision tower (`model.visual.*`)
Kernel in vLLM	Marlin (`MarlinLinearKernel`)

The exact recipe is stored next to the weights in recipe.yaml; the effective quantization config is embedded in config.json (quantization_config).

What was done, step by step

Base model. Started from the Claude-Opus-reasoning distill of Qwen3.5-4B (Qwen3_5ForConditionalGeneration, a hybrid model with 8 of 32 layers in full attention and the rest in gated linear attention, GDN). Loaded in float16.
Environment. llm-compressor with transformers >= 5 (the Qwen3.5 recipes require the 5.x branch). The GPU was freed from other services first — a 4.5B model does not fit alongside a running inference stack on a 16 GB card.
Calibration set. 256 child chunks (400–4000 characters) sampled from the target production corpus rather than a generic dataset, so that the GPTQ/AWQ statistics match the documents the model actually sees.
AWQ pre-pass. Channel-wise scaling factors are searched per layer (AWQModifier, duo_scaling="both", n_grid=10), balancing the weight groups before quantization. This step is the expensive one — roughly 4.6 minutes per layer, about 2 h 23 min for the whole model.
GPTQ. Weights are quantized to INT4 with group size 32, asymmetric (zero-point) scales, and , so that the most salient input channels are quantized first. Applied to every layer except and the vision tower.

Results

Size: ~4.0 GB on disk (one model.safetensors) against ~5.3 GB in float16; weights in VRAM ~3.33 GiB against ~4.48 GiB.
Speed: on a 16 GB card, decode grew from 120 to 152–159 tokens/s; the KV cache pool grew by about a third.
Quality: 28 of 32 assertions on our golden set — the same score as the plain GPTQ arm. Three failures are stable and unrelated to quantization (date extraction from one specific document, and one total-sum case); a fourth fluctuates between two cases and is vLLM decode nondeterminism.
Note on linear attention. Unlike the vendor convention (which keeps the whole linear_attn block in 16-bit), this export quantizes it too: 1.88 GiB → 0.55 GiB, which is where most of the speed-up comes from. It is a deliberate deviation, taken on measurements, not on the vendor's ignore list.

Usage

Serve with vLLM (the checkpoint is already in the format vLLM expects):

bash

vllm serve <this-repo> \  --served-model-name qwen3.5-4b-distill-w4a16 \  --max-model-len 16384 \  --gpu-memory-utilization 0.85

The model needs a GPU with compute capability ≥ 75: the Marlin kernel used for W4A16 does not support older architectures (V100 and below).

Files

Table with columns: File, Purpose
File	Purpose
`model.safetensors`	Quantized weights (~4.0 GB)
`config.json`	Model config with the embedded `quantization_config`
`recipe.yaml`	Exact llm-compressor recipe used for this export
`tokenizer.json`, `tokenizer_config.json`, `processor_config.json`	Tokenizer and processor

License

Apache 2.0, inherited from the base model. Please refer to the base model card for the original terms and for the provenance of the distillation data.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

bumbuk

Model Tree

Base

Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality