darioooooo0o

Qwen3.8-27B-Uncensored-GPTQ-Int4-asym

README

License: apache-2.0

Quality vs the BF16 base

KL divergence over the top-20 logits on held-out 2048-token windows (lower is better):

Table with columns: private chat, code, technical prose, all, top-1 agree
	private chat	code	technical prose	all	top-1 agree
symmetric GPTQ int4 g128 (usual recipe)	0.038	0.078	0.096	0.069	91.9%
this quant	0.018	0.040	0.060	0.038	94.1%

The served version (fp8 lm_head, see the cookbook) measures KL 0.039 end to end. On the arithmetic canary from the B70 cookbook it scores 5/6 vs 4/6 for symmetric GPTQ (miss: sum(i*i for i in range(4)) -> 30). That canary is only 6 greedy near-tie questions, so treat it as noise. On the stock model a 100-question version showed no real difference between quants.

Running it

Built for and tested on Intel Arc Pro B70 + vLLM XPU v0.31.0 with the launcher and patches in riceharvest/b70-qwen38-cookbook. That setup gives 262K context with vision, MTP k=4, and fp8 KV on one card.

bash

hf download darioooooo0o/Qwen3.8-27B-Uncensored-GPTQ-Int4-asym --local-dir qwen38-uncensored-b70MODEL_DIR=$PWD/qwen38-uncensored-b70 ./serve.sh     # from the cookbook repo

On a B70 at 275 W, one request at a time. These were measured with a draft vocab tuned on my own traffic; the cookbook's public vocab gives the same ms/step and slightly different tokens/step:

Table

Decode, short prompts (greedy)	64-128 tok/s depending on content (35 ms per MTP verify step)
Decode at 65K / 100K context	~79 / 65-70 tok/s greedy, ~64 sampled (43 / 46 ms per step)
Prefill	~2050 tok/s at 6K, ~1450 at 74K
Context	262,144 tokens, ~320K-token KV pool

Other runtimes: the format is standard asymmetric GPTQ v1, so GPTQModel / transformers / vLLM on CUDA should load it. I only tested on Intel XPU. Stock vLLM XPU v0.31 does not load asym GPTQ correctly without the cookbook's patch_asym_gptq.py.

How it was made

Quantizer: b70quant.py from the cookbook. GPTQ (damp 0.01, act-order off), asym uint4 with per-group scale + zero, and a clip search over 11 ratios minimizing |err|^2.4. One layer at a time on the B70, ~1 h.
Calibration: 128 x 2048 tokens. A mix of private chat samples (not released), reasoning/agentic text, code and technical prose, disjoint from the eval windows.
Not quantized: lm_head, MTP layers, vision tower, the GDN in_proj_a/in_proj_b.
Export: GPTQ v1 (qzeros store z-1). The single group with z=0 was set to z=1 (one weight changed). It was checked against the raw quant using GPTQModel's dequant semantics.

Credits: Qwen team (base model), OrcaRouter (finetune), SergiioB (B70 vLLM cookbook this builds on).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

darioooooo0o

Model Tree

Base

orcarouter/Qwen3.8-27B-Uncensored

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality