lazybrick

Qwen3.5-4B-Kiln-INT8-W8A8

README

License: apache-2.0

Quantization

Table

Method	SmoothQuant (smoothing strength 0.8), then GPTQ
Weights	INT8, symmetric, per output channel
Activations	INT8, symmetric, dynamic per token
Not quantized	`lm_head`, token embeddings, vision encoder (`re:.visual.`)
Format	`compressed-tensors` (safetensors)
Toolkit	`llm-compressor` 0.13.0, `compressed-tensors` 0.18.0
Checkpoint size	5.51 GB (BF16: 9.32 GB; 1.69× smaller)

Recipe:

python

SmoothQuantModifier(smoothing_strength=0.8) + GPTQModifier(scheme="W8A8", ignore=[...])

Calibration data: 512 conversations from HuggingFaceH4/ultrachat_200k (train_sft, revision 8049631), sampled with seed 42, chat template applied, truncated to 2,048 tokens. All Kiln variants use the same calibration set.

Evaluation

The variant is scored against the BF16 base model with the same protocol, so the difference between the two columns measures the effect of quantization alone.

Table with columns: Task, Metric, BF16 (reference), INT8 W8A8, Δ
Task	Metric	BF16 (reference)	INT8 W8A8	Δ
MMLU-Pro	exact match	74.6	74.3	-0.3
GSM8K	exact match, flexible extract	83.2	83.5	+0.3
MATH-500	math_verify	83.4	82.6	-0.8

Protocol. Instruct mode (enable_thinking=False), greedy decoding, up to 8,192 generated tokens. Text tasks use lm-evaluation-harness 0.4.13 (0-shot, chat template) on vLLM 0.29.0. Vision tasks use VLMEvalKit (revision 34a64e6) against a vLLM server. TextVQA, OCRBench and DocVQA are scored by VLMEvalKit's rules. For MMBench, MMMU and MathVista, a fixed answer extractor (gpt-4o-mini) reads the chosen option from each answer. It is the same for every model and never the model under test. Base model revision: 851bf6e.

These figures are not comparable with those on the Qwen3.5-4B model card, which reports thinking mode with sampling, a 32,768–81,920-token budget and benchmark-specific answer prompts.

Per-sample outputs, predictions and score files: lazybrick/kiln-evals.

Usage

The weights load directly in vLLM, which selects the matching quantized kernels:

bash

vllm serve lazybrick/Qwen3.5-4B-Kiln-INT8-W8A8 --max-model-len 32768

To get direct answers without a thinking trace, pass chat_template_kwargs={"enable_thinking": false} in the request. See the base model card for the recommended sampling settings.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B. This is a derivative of the Qwen3.5-4B weights; please cite the Qwen team's work:

bibtex

@misc{qwen3.5,    title  = {{Qwen3.5}: Towards Native Multimodal Agents},    author = {{Qwen Team}},    month  = {February},    year   = {2026},    url    = {https://qwen.ai/blog?id=qwen3.5}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

lazybrick

Model Tree

Base

Qwen/Qwen3.5-4B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality