lazybrick

Qwen3.5-4B-Kiln-Glaze-W4A16

README

License: apache-2.0

Quantization

Glaze quantizes from the BF16 weights in five steps:

In-domain calibration data. 4,419 prompts from public training sets (MATH, ARC-Challenge, SciQ, UltraChat), answered by the BF16 model itself. Prompts that share a 13-gram with a MATH-500 or MMLU-Pro test question are removed. The quantized model is trained toward BF16's own answers, never toward gold answers.
Fisher pass. Sampled-label gradients of the BF16 model approximate how much each layer's error costs in next-token loss.
Byte-neutral mixed precision. The vision encoder's Linears are stored at 8 bits instead of BF16. The bytes this frees go to the language model's most sensitive layers, chosen by a knapsack over predicted loss savings: 8-bit for 40 of 152 fused units, 4-bit with group size 32 or 64 for 81, and 4-bit with group size 128 for the rest.
Learned rounding and clipping. Layer by layer, each weight's rounding and each group's clipping are learned against a Fisher-weighted reconstruction loss, with early stopping on validation data.
Export to compressed-tensors, served by stock vLLM with its Marlin kernels.

Table

Weights	INT4, symmetric, group size 128, 64 or 32 per layer; INT8, group size 128, for the most sensitive layers and the vision encoder
Activations	BF16 (not quantized)
Not quantized	`lm_head`, token embeddings
Format	`compressed-tensors` (safetensors), one config group per precision
Toolkit	`torch` 2.10.0, `transformers` 5.12.1, `compressed-tensors` 0.18.0
Checkpoint size	3.80 GB (BF16: 9.32 GB; 2.46× smaller), 5 MB smaller than the AutoRound W4A16 g128 checkpoint

Evaluation

Every model is scored with the same protocol, so the differences between columns measure the effect of quantization alone. AutoRound W4A16 g128 is the strongest uniform 4-bit baseline at the same size (Qwen3.5-4B-Kiln-AutoRound-W4A16-g128).

Table with columns: Task, Metric, BF16 (reference), AutoRound W4A16 g128, Glaze W4A16
Task	Metric	BF16 (reference)	AutoRound W4A16 g128	Glaze W4A16
MMLU-Pro	exact match	74.6	72.8	73.7
GSM8K	exact match, flexible extract	83.2	82.3	82.8
MATH-500	math_verify	83.4	73.2

Benchmark scores of GPTQ, AWQ, AutoRound and Glaze, each axis relative to BF16

Paired comparisons on the same questions (bootstrap, 95% intervals), Glaze minus AutoRound: MATH-500 +10.0 points [+6.6, +14.0]; IFEval +2.2 [−0.6, +5.0]; MMLU-Pro (fixed 1,001-question subset) −0.3 [−2.4, +2.0]. Glaze minus BF16 on MATH-500: −0.2 [−3.4, +2.8].

Token-level agreement with BF16. Replaying the BF16 model's own 1,501 MATH-500 and MMLU-Pro answers (2.3M tokens), Glaze picks a different top token than BF16 on 3.29% of positions (AutoRound 4.96%, INT8 W8A8 2.42%), and its excess loss over BF16 is 0.014 nats per token (AutoRound 0.042, INT8 W8A8 0.011).

Protocol. Instruct mode (enable_thinking=False), greedy decoding, up to 8,192 generated tokens. Text tasks use lm-evaluation-harness 0.4.13 (0-shot, chat template) on vLLM 0.29.0. Vision tasks use VLMEvalKit (revision 34a64e6) against a vLLM server. TextVQA, OCRBench and DocVQA are scored by VLMEvalKit's rules. For MMBench, MMMU and MathVista, a fixed answer extractor (gpt-4o-mini) reads the chosen option from each answer. It is the same for every model and never the model under test. Base model revision: 851bf6e.

Notes.

The calibration domains (math, multiple-choice science) overlap with the benchmark domains by design. ARC-Challenge training questions are used for calibration; they are not checked against the ARC-Challenge test split, on which Glaze scores below both BF16 and AutoRound.
Repeated evaluations of one model on different GPUs differ by about 1–2 points.

These figures are not comparable with those on the Qwen3.5-4B model card, which reports thinking mode with sampling, a 32,768–81,920-token budget and benchmark-specific answer prompts.

Per-sample outputs, predictions and score files: lazybrick/kiln-evals.

Usage

The weights load directly in vLLM, which selects the matching quantized kernels for each layer:

bash

vllm serve lazybrick/Qwen3.5-4B-Kiln-Glaze-W4A16 --max-model-len 32768

To get direct answers without a thinking trace, pass chat_template_kwargs={"enable_thinking": false} in the request. See the base model card for the recommended sampling settings.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B. This is a derivative of the Qwen3.5-4B weights; please cite the Qwen team's work:

bibtex

@misc{qwen3.5,    title  = {{Qwen3.5}: Towards Native Multimodal Agents},    author = {{Qwen Team}},    month  = {February},    year   = {2026},    url    = {https://qwen.ai/blog?id=qwen3.5}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

lazybrick

Model Tree

Base

Qwen/Qwen3.5-4B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality