lazybrick

Qwen3.5-4B-Kiln-AutoRound-W4A16-g128

README

License: apache-2.0

Quantization

Table

Method	AutoRound: each block's rounding and clipping tuned by signed gradient descent (200 steps, batch 8)
Weights	INT4, symmetric, group size 128
Activations	BF16 (not quantized)
Not quantized	`lm_head`, token embeddings, vision encoder (`re:.visual.`)
Format	`compressed-tensors` (safetensors)
Toolkit	`llm-compressor` 0.13.0, `compressed-tensors` 0.18.0, `auto-round` 0.14.2, `flash-linear-attention` 0.5.2
Checkpoint size	3.80 GB (BF16: 9.32 GB; 2.45× smaller)

Recipe:

python

AutoRoundModifier(scheme="W4A16", group_size=128, iters=200, ignore=[...])

Calibration data: the same 512 conversations as the other Kiln variants, from HuggingFaceH4/ultrachat_200k (train_sft, revision 8049631, seed 42, chat template applied), joined in order and cut into 2,048-token blocks. AutoRound uses the first 128 blocks, its own default sample count.

Evaluation

The variant is scored against the BF16 base model with the same protocol, so the difference between the two columns measures the effect of quantization alone.

Table with columns: Task, Metric, BF16 (reference), AutoRound W4A16, Δ
Task	Metric	BF16 (reference)	AutoRound W4A16	Δ
MMLU-Pro	exact match	74.6	72.8	-1.8
GSM8K	exact match, flexible extract	83.2	82.3	-0.9
MATH-500	math_verify	83.4	73.2	-10.2

Protocol. Instruct mode (enable_thinking=False), greedy decoding, up to 8,192 generated tokens. Text tasks use lm-evaluation-harness 0.4.13 (0-shot, chat template) on vLLM 0.29.0. Vision tasks use VLMEvalKit (revision 34a64e6) against a vLLM server. TextVQA, OCRBench and DocVQA are scored by VLMEvalKit's rules. For MMBench, MMMU and MathVista, a fixed answer extractor (gpt-4o-mini) reads the chosen option from each answer. It is the same for every model and never the model under test. Base model revision: 851bf6e.

These figures are not comparable with those on the Qwen3.5-4B model card, which reports thinking mode with sampling, a 32,768–81,920-token budget and benchmark-specific answer prompts.

Per-sample outputs, predictions and score files: lazybrick/kiln-evals.

Usage

The weights load directly in vLLM, which selects the matching quantized kernels:

bash

vllm serve lazybrick/Qwen3.5-4B-Kiln-AutoRound-W4A16-g128 --max-model-len 32768

To get direct answers without a thinking trace, pass chat_template_kwargs={"enable_thinking": false} in the request. See the base model card for the recommended sampling settings.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B. This is a derivative of the Qwen3.5-4B weights; please cite the Qwen team's work:

bibtex

@misc{qwen3.5,    title  = {{Qwen3.5}: Towards Native Multimodal Agents},    author = {{Qwen Team}},    month  = {February},    year   = {2026},    url    = {https://qwen.ai/blog?id=qwen3.5}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

lazybrick

Model Tree

Base

Qwen/Qwen3.5-4B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality