darioooooo0o

Qwen3.8-27B-GPTQ-Int4-asym

README

License: apache-2.0

Quality vs BF16

KL divergence over the top-20 logits on held-out 2048-token windows (lower is better):

Table with columns: code, technical prose, chat (private samples), all, top-1 agree
	code	technical prose	chat (private samples)	all	top-1 agree
SergiioB's GPTQ int4 (sym g128, GPTQModel)	0.087	0.103	0.050	0.078	91.3%
this quant	0.037	0.057	0.025	0.039	93.8%
this quant + fp8 lm_head (as served in the cookbook)	0.038	0.058	0.025	0.039	93.5%

In-head arithmetic, greedy, thinking off, same server:

the B70 cookbook's 6-question canary: this quant 4/6, SergiioB's 6/6 (6 near-tie questions, so mostly noise)
100 seeded questions of the same kind: this quant 48/100, SergiioB's 49/100

Running it

Built for and tested on Intel Arc Pro B70 + vLLM XPU v0.31.0 with the launcher and patches in riceharvest/b70-qwen38-cookbook. That setup gives 262K context with vision, MTP k=4, and fp8 KV on one card.

bash

hf download darioooooo0o/Qwen3.8-27B-GPTQ-Int4-asym --local-dir qwen38-b70MODEL_DIR=$PWD/qwen38-b70 ./serve.sh     # from the cookbook repo

On a B70 at 275 W, one request at a time:

Table

Decode, short prompts	66-119 tok/s greedy depending on content (35 ms per MTP verify step)
Decode at 65K / 100K context	~72 / 65-67 tok/s greedy, 73-77 / 66-69 sampled (43 / 46 ms per step)
Prefill	2042 tok/s at 5.7K, 1422 at 74K, 824 at 237K
Context	262,144 tokens with vision; 237K-token needle retrieved

Other runtimes: the format is standard asymmetric GPTQ v1, so GPTQModel / transformers / vLLM on CUDA should load it. I only tested on Intel XPU. Stock vLLM XPU v0.31 does not load asym GPTQ correctly without the cookbook's patch_asym_gptq.py.

How it was made

Quantizer: b70quant.py from the cookbook. GPTQ (damp 0.01, act-order off), asym uint4 with per-group scale + zero, and a clip search over 11 ratios minimizing |err|^2.4. One layer at a time on the B70, ~1 h.
Calibration: 128 x 2048 tokens: 40 reasoning/agentic samples from my own datasets (not released), 44 code windows (Python source), 44 technical prose windows (Linux docs). Disjoint from the eval windows.
Not quantized: lm_head, MTP layers, vision tower, the GDN in_proj_a/in_proj_b.
Export: GPTQ v1 (qzeros store z-1). The two groups with z=0 were set to z=1 (two weights changed). It was checked against the raw quant using GPTQModel's dequant semantics.

Credits: Qwen team (base model), SergiioB (B70 vLLM cookbook this builds on).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

darioooooo0o

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality