abhishekchohan

Qwen3.8-27B-GPTQ-INT4

README

License: apache-2.0

TL;DR

Table with columns: value
	value
Format	INT4, asymmetric (zero-point), `group_size=32`, weight-only
Method	GPTQ (Hessian error compensation) + imatrix-mse observer, `actorder=static`, block 128, `dampening_frac: 0.01`
Calibration	8 sequences × 262,144 tokens (the model's native context length), packed from Nemotron-Post-Training-v3 SFT datasets (instruction-following chat, math, science, agentic code, multilingual)
Disk size	~27 GB (vs ~55.6 GB BF16)
Accuracy vs BF16 base	No degradation beyond sampling noise on 5 benchmarks + long-context retrieval

What was quantized (and what was protected)

Table with columns: Component, Quantized, Protected (kept BF16)
Component	Quantized	Protected (kept BF16)
`language_model` (64 hybrid-attention text layers)	block linears — attention `self_attn.{q_proj,k_proj,v_proj,o_proj}` and FFN `mlp.{gate_proj,up_proj,down_proj}` (INT4, asymmetric, `group_size=32`)	layer norms, `embed_tokens`, `norm`, `lm_head`
Linear-attention (DeltaNet) projections	—	all `.linear_attn.` modules
(frozen ViT encoder)

Quantization recipe

GPTQ (Hessian error compensation) with weight-only INT4: asymmetric (zero-point, stored as int8), group_size=32, imatrix-mse weight observer, actorder=static, block size 128, dampening_frac: 0.01. Targets all Linear layers except lm_head, embeddings, DeltaNet (.*linear_attn.*), vision (model.visual.*), and the MTP predictor. The full resolved recipe — including defaults — ships as recipe.yaml in this repo.

imatrix importance statistics were collected by forward hooks over the 8 packed calibration sequences (llm-compressor imatrix-mse observer). The 8-sequence budget was chosen to calibrate at the model's native 262,144-token context within a single 80GB GPU — the token count (~2.1M) is conventional, the document count is the trade-off.

Evaluation

Identical harness and hardware for base and quantized (lm-eval-harness, hf-multimodal backend, greedy). Generative tasks use the chat template with thinking enabled; ARC-Easy is loglikelihood.

Table with columns: task, protocol, n, base (BF16), this, Δ
task	protocol	n	base (BF16)	this	Δ
GSM8K	thinking, generative (250-sample subset)	250	95.60	94.80	−0.8 (within noise)
GPQA Diamond	thinking, generative	198	64.65	64.65	0.0
RealWorldQA	thinking, generative

¹ Letter-multiple-choice protocol applied identically to both models; absolutes are not comparable to the official boxed-answer protocol — the Δ is the meaningful comparison. ² acc_norm < acc inverts the usual ARC-Easy ordering for both models under this harness/tokenizer; the effect is identical for base and quantized.

Long-context needle-in-a-haystack (greedy, 3 needle depths per length; both models retrieve exactly):

Table with columns: Context (tokens), Base, This model
Context (tokens)	Base	This model
8,192	3/3	3/3
32,768	3/3	3/3
131,072	3/3	3/3
262,144 (native)	3/3	3/3

Retrieval validated at the model's native 262,144-token context — the same length the model was calibrated at.

Not evaluated: agentic/tool-calling behavior; vision coverage beyond the two small-n tasks above.

Serving

bash
vllm serve abhishekchohan/Qwen3.8-27B-GPTQ-INT4

Requires a recent vLLM with qwen3_5 support (per the base model card, use the latest version). The checkpoint is compressed-tensors pack-quantized int4 — int4 block linears plus BF16 DeltaNet/vision/embeddings/lm_head, so int4 kernels (Marlin/compressed-tensors) engage for the quantized linears only. ~27 GB of weights: fits a 32GB+ GPU at moderate context (KV ≈2 GB at 32k). Full 262,144-token context needs ≈17 GB of KV on top — plan for ~48GB total, or cap --max-model-len on smaller cards. KV ≈65 KB/token (hybrid: only global-attention layers accumulate KV; DeltaNet layers carry fixed-size state). MTP predictor weights are included intact; speculative-decoding wiring depends on engine support for this architecture.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

abhishekchohan

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality