Quality vs the BF16 base
KL divergence over the top-20 logits on held-out 2048-token windows (lower is better):
Table with columns: private chat, code, technical prose, all, top-1 agree | private chat | code | technical prose | all | top-1 agree |
|---|
| symmetric GPTQ int4 g128 (usual recipe) | 0.038 | 0.078 | 0.096 | 0.069 | 91.9% |
| this quant | 0.018 | 0.040 | 0.060 | 0.038 | 94.1% |
The served version (fp8 lm_head, see the cookbook) measures KL 0.039 end to end. On the arithmetic canary from the B70
cookbook it scores 5/6 vs 4/6 for symmetric GPTQ (miss: sum(i*i for i in range(4)) -> 30). That canary is only 6
greedy near-tie questions, so treat it as noise. On the stock model a 100-question version showed no real difference
between quants.
Running it
Built for and tested on Intel Arc Pro B70 + vLLM XPU v0.31.0 and v0.32.0 with the launcher and B70 vLLM changes in
riceharvest/b70-qwen38-cookbook. That setup gives 262K
context with vision, MTP k=4, and fp8 KV on one card.
hf download darioooooo0o/Qwen3.8-27B-Uncensored-GPTQ-Int4-asym --local-dir qwen38-uncensored-b70MODEL_DIR=$PWD/qwen38-uncensored-b70 ./serve.sh # from the cookbook repo
On a B70 at 275 W, one request at a time. These were measured with a draft vocab tuned on my own traffic; the cookbook's
public vocab gives the same ms/step and slightly different tokens/step:
Table | |
|---|
| Decode, short prompts (greedy) | 64-128 tok/s depending on content (35 ms per MTP verify step) |
| Decode at 65K / 100K context | ~79 / 65-70 tok/s greedy, ~64 sampled (43 / 46 ms per step) |
| Prefill | ~2050 tok/s at 6K, ~1450 at 74K |
| Context | 262,144 tokens, ~320K-token KV pool |
Prebuilt image: ghcr.io/riceharvest/b70-vllm:v0.32.0-b70.7 (= :latest, vLLM v0.32.0) has the B70 vLLM changes and kernels
baked in:
docker run -d --device /dev/dri -v /dev/dri:/dev/dri:ro --shm-size 8g -p 8000:8000 -v ~/models:/models -e MODEL=darioooooo0o/Qwen3.8-27B-Uncensored-GPTQ-Int4-asym ghcr.io/riceharvest/b70-vllm:latest
.
Many concurrent requests: add --mamba-ssm-cache-dtype float16 (after the image name, or B70_GDN_STATE=fp16 ./serve.sh).
Qwen3.8 stores its linear-attention (GDN) state in fp32 by default, and with many requests that state is a large part of what
the GPU reads per step. Measured on Qwen3.6-35B-A3B, which has the same GDN layers: 64 concurrent users went from 49 to 42-45
ms per token, with decode quality at the fp32 noise floor and gsm8k 98/100 vs 94/100
(details). Not measured on this model yet.
Other runtimes: the format is standard asymmetric GPTQ v1, so GPTQModel / transformers / vLLM on CUDA should
load it. I only tested on Intel XPU. Stock vLLM XPU (v0.31, v0.32) does not load asym GPTQ correctly without the
cookbook's asym GPTQ change (upstream PR vllm#61731).
How it was made
- Quantizer:
b70quant.py from the cookbook. GPTQ (damp 0.01, act-order off), asym uint4 with per-group
scale + zero, and a clip search over 11 ratios minimizing |err|^2.4. One layer at a time on the B70, ~1 h.
- Calibration: 128 x 2048 tokens. A mix of private chat samples (not released), reasoning/agentic text, code and
technical prose, disjoint from the eval windows.
- Not quantized: lm_head, MTP layers, vision tower, the GDN
in_proj_a/in_proj_b.
- Export: GPTQ v1 (qzeros store z-1). The single group with z=0 was set to z=1 (one weight changed). It was checked
against the raw quant using GPTQModel's dequant semantics.
Credits: Qwen team (base model), OrcaRouter (finetune),
SergiioB (B70 vLLM cookbook this builds on).
Questions or issues: reply to @imdariotoo on X or open a discussion on this repo.