davetha

Huihui-Qwen3.8-27B-abliterated-AutoRound-Int4-baked-embed-int8

README

License: apache-2.0

What is in the files

Table with columns: part, form, how
part	form	how
400 text body linears	INT4, symmetric, group 128 (GPTQ-compatible packing)	AutoRound / SignRound from the bf16 Huihui weights
`lm_head`	INT4 GPTQ, group 128	Hessian GPTQ on this model's own activations
MTP block (`fc`, q/k/v/o, gate/up/down)	INT4 GPTQ, group 128	same
token embedding	per-row symmetric INT8 + FP16 scales (`model-embed-int8.safetensors`)	absmax/127
vision tower, MTP norms, GDN `in_proj_a` / `in_proj_b`, norms	BF16 (unchanged)	—

Weights resident on the GPU: 14.75 GiB (vs 18.28 GiB for a GPTQ build that keeps lm_head / MTP dense), which leaves room for the full 262,144-token context on one 32 GB card (vLLM reports a 319K-token KV pool with fp8 KV and 400K with the K8/V4 int8-K / int4-V KV cache).

The dense embedding is still in the shards, so stock loaders work; the INT8 embedding only takes effect in a runtime that reads embed_tokens_quant (the patched image below).

Quality

WikiText-2 test perplexity, 40 chunks x 2,048 tokens (81,880 scored tokens), the same token chunks for both, computed from vLLM prompt_logprobs on the B70 (fp8 KV; with the K8/V4 KV cache it measures 6.276):

Table with columns: build, perplexity
build	perplexity
this repo (AutoRound body from bf16 + baked head/MTP + INT8 embedding)	6.266
zrlu's GPTQ-Int4 Huihui export + the same head/MTP/embedding bake	6.346

Head / MTP bake on held-out activations (relative output L2 error; GPTQ vs round-to-nearest):

Table with columns: linear, GPTQ, round-to-nearest
linear	GPTQ	round-to-nearest
lm_head	3.39%	7.61%
mtp.fc	5.67%	11.86%
MTP q / k / v / o	2.11% / 4.99% / 4.06% / 5.04%	4.42% / 10.46% / 8.33% / 12.12%
MTP gate / up / down	2.93% / 4.59% / 5.74%	6.46% / 9.75% / 11.30%
embedding (INT8, weight L2)	0.90%	—

These are matrix and perplexity checks, not a full benchmark suite; upstream evaluation scores have not been re-established for this derivative. A quick functional check (arithmetic, thinking on/off, tool-call template, an uncensored prompt) passed.

Speed on one Arc Pro B70

Numbers are BetterBench median decode tokens/s (temperature 0.7, top-p 0.95, thinking on) and prefill tokens/s, all on one 32 GB card.

Recommended: K8/V4 KV cache + the built-in MTP head (3 speculative tokens), 262,144-token context. The KV cache is int8-K / int4-V (k8v4-xpu, ported to one GPU in davetha/k8v4-xpu tp1-k8v4); every attention layer, including MTP, uses the same cache. The KV pool holds 400,705 tokens (1.5 full-length requests); WikiText-2 perplexity 6.276 (6.266 with fp8 KV).

Table with columns: category, K8/V4 + MTP3 @256K, fp8 + MTP3 @256K, K8/V4 + DFlash2 @256K, fp8 + DFlash2 @192K
category	K8/V4 + MTP3 @256K	fp8 + MTP3 @256K	K8/V4 + DFlash2 @256K	fp8 + DFlash2 @192K
code	73.9	73.6	71.3	82.2
reasoning	67.8	62.8	54.6	61.6
prose	58.2	64.3	65.5	61.4

The K8/V4 + MTP3 column is a short run (5 passes per category, the others 20), so differences under ~5% are noise.

Table with columns: prompt length, K8/V4 + MTP3, K8/V4 + DFlash2, fp8 + DFlash2, fp8 + MTP3
prompt length	K8/V4 + MTP3	K8/V4 + DFlash2	fp8 + DFlash2	fp8 + MTP3
~2K	1,850	1,757	1,841	1,848
~8K	1,922	1,813	1,897	1,888
~16K	1,885	1,808	1,790	1,772

Time to first token with K8/V4 + MTP3: 0.8 s at 2K, 6.3 s at 16K, 30 s at 64K. Greedy single runs: prose ~73, JSON ~99, code ~93 tok/s; 72K-token needle retrieval passes.

Why not the DFlash2 drafter? With the z-lab DFlash2 drafter decode is 7-15% faster, but at 256K it only fits next to K8/V4 by running the drafter's own layers on fp8 (a vLLM patch, vllm-patches/), i.e. two KV formats sharing one block pool. That setup produced one degenerate repetition loop in real use (a single token repeated from the first reasoning token; draft acceptance dropped to zero) that could not be reproduced in ~700 test requests, so it is not recommended. fp8 KV + DFlash2 has no such mixing and was fine, but tops out near 192K context.

Serving (vLLM XPU, one Arc Pro B70)

Build the image from davetha/k8v4-xpu tp1-k8v4 (TP1-B70.md: Dockerfile.tp1, then k8v4_v030/Dockerfile.tp1-kv; the vLLM skip patch is not needed). The single-library version of the same port is proposed upstream in Wrapzii/k8v4-xpu#1.

bash

docker run -d --name qwen27-b70 \  --device /dev/dri/card1 --device /dev/dri/renderD128 --group-add render \  --ipc=host --shm-size=8g -p 8200:8000 \  -v /path/to/this/repo:/model:ro \  -e VLLM_TARGET_DEVICE=xpu -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \  -e XE2_KV_S2_NSG_DRAFT=32 -e XE2_KV_S2_NSG_VERIFY=8 -e K8V4_PREFILL_GATHER=triton -e K8V4_SDPA_ASYNC=0 \  vllm-openai-xpu:v0.30.0-k8v4-tp1-kv /model \  --host 0.0.0.0 --port 8000 --served-model-name qwen27-huihui-unc \  --gpu-memory-utilization 0.92 --dtype bfloat16 \  --max-model-len 262144 --kv-cache-dtype int8_k_int4_v --tensor-parallel-size 1 \  --max-num-seqs 4 --max-num-batched-tokens 4224 \  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \  --enable-prefix-caching --trust-remote-code \  --limit-mm-per-prompt '{"image":4,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}' \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[4,8,12,16]}'

For fp8 KV instead, use the vllm-openai-xpu:v0.30.0-k8v4-tp1 image with --kv-cache-dtype fp8 (319K-token pool). A stock vLLM XPU image loads the checkpoint too, with the embedding dense.

Thinking: chat_template_kwargs: {"enable_thinking": false} turns it off; reasoning_effort takes low / medium / xhigh (the bundled chat_template_low.jinja defaults to low). Tool calls use the qwen3 XML format.

How it was made

Body — AutoRound 0.16.0 on the bf16 checkpoint, on the B70 (2.5 h, 27.5 GB peak VRAM):

bash

auto-round --model huihui-ai/Huihui-Qwen3.8-27B-abliterated --scheme W4A16 --bits 4 --group_size 128 \  --iters 200 --batch_size 8 --nsamples 256 --seqlen 2048 --seed 42 \  --dataset HuggingFaceH4/ultrachat_200k \  --ignore_layers "lm_head,.*visual.*,.*mtp.*,.*in_proj_a,.*in_proj_b" \  --format auto_round --low_gpu_mem_usage

AutoRound writes the 15 MTP tensors (dropped by the Transformers loader) back from the source into model_extra_tensors.safetensors.

GPTQ compatibility — tools/prepare_swift_autoround.py from k8v4-xpu: verifies the 400 packed linears and zero points, adds deterministic g_idx, switches the config to gptq. No weight values change.
Calibration — tools/calibrate_swift_bake.py (run at TP1 with fp8 KV on the single card): 128 WikiText-2 validation prompts of 384 tokens, 64 generated tokens each, MTP on; Hessians for lm_head and the MTP inputs, every 64th row held out.
Bake — tools/bake_gptq_resume.py: symmetric GPTQ INT4, group 128, 1% damping, no activation ordering.
— : rewritten shards without the dense replaced tensors, the 36 GPTQ arrays in , the INT8 embedding side file. Settings and per-linear errors are in .

Credits and licenses

Base model: Qwen/Qwen3.8-27B (Apache-2.0); abliteration: huihui-ai (Apache-2.0, LICENSE).
Body quantization recipe: ukisai/Swift-1.5-Qwen3.8-27b-W4A16-AutoRound; tool: Intel AutoRound.
Head/MTP bake, INT8 embedding and XPU serving stack: Wrapzii/k8v4-xpu; its GPTQ bake tooling is adapted from launch80/B65 (MIT, B65-LICENSE).
This derivative: weights requantized only — no fine-tuning, no merging.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

davetha

Model Tree

Base

huihui-ai/Huihui-Qwen3.8-27B-abliterated

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What is in the files

Table with columns: part, form, how
part	form	how
400 text body linears	INT4, symmetric, group 128 (GPTQ-compatible packing)	AutoRound / SignRound from the bf16 Huihui weights
`lm_head`	INT4 GPTQ, group 128	Hessian GPTQ on this model's own activations
MTP block (`fc`, q/k/v/o, gate/up/down)	INT4 GPTQ, group 128	same
token embedding	per-row symmetric INT8 + FP16 scales (`model-embed-int8.safetensors`)	absmax/127
vision tower, MTP norms, GDN `in_proj_a` / `in_proj_b`, norms	BF16 (unchanged)	—

The dense embedding is still in the shards, so stock loaders work; the INT8 embedding only takes effect in a runtime that reads embed_tokens_quant (the patched image below).

Quality

Table with columns: build, perplexity
build	perplexity
this repo (AutoRound body from bf16 + baked head/MTP + INT8 embedding)	6.266
zrlu's GPTQ-Int4 Huihui export + the same head/MTP/embedding bake	6.346

Head / MTP bake on held-out activations (relative output L2 error; GPTQ vs round-to-nearest):

Table with columns: linear, GPTQ, round-to-nearest
linear	GPTQ	round-to-nearest
lm_head	3.39%	7.61%
mtp.fc	5.67%	11.86%
MTP q / k / v / o	2.11% / 4.99% / 4.06% / 5.04%	4.42% / 10.46% / 8.33% / 12.12%
MTP gate / up / down	2.93% / 4.59% / 5.74%	6.46% / 9.75% / 11.30%
embedding (INT8, weight L2)	0.90%	—

Speed on one Arc Pro B70

Numbers are BetterBench median decode tokens/s (temperature 0.7, top-p 0.95, thinking on) and prefill tokens/s, all on one 32 GB card.

Table with columns: category, K8/V4 + MTP3 @256K, fp8 + MTP3 @256K, K8/V4 + DFlash2 @256K, fp8 + DFlash2 @192K
category	K8/V4 + MTP3 @256K	fp8 + MTP3 @256K	K8/V4 + DFlash2 @256K	fp8 + DFlash2 @192K
code	73.9	73.6	71.3	82.2
reasoning	67.8	62.8	54.6	61.6
prose	58.2	64.3	65.5	61.4

The K8/V4 + MTP3 column is a short run (5 passes per category, the others 20), so differences under ~5% are noise.

Table with columns: prompt length, K8/V4 + MTP3, K8/V4 + DFlash2, fp8 + DFlash2, fp8 + MTP3
prompt length	K8/V4 + MTP3	K8/V4 + DFlash2	fp8 + DFlash2	fp8 + MTP3
~2K	1,850	1,757	1,841	1,848
~8K	1,922	1,813	1,897	1,888
~16K	1,885	1,808	1,790	1,772

Time to first token with K8/V4 + MTP3: 0.8 s at 2K, 6.3 s at 16K, 30 s at 64K. Greedy single runs: prose ~73, JSON ~99, code ~93 tok/s; 72K-token needle retrieval passes.

Serving (vLLM XPU, one Arc Pro B70)

bash

docker run -d --name qwen27-b70 \  --device /dev/dri/card1 --device /dev/dri/renderD128 --group-add render \  --ipc=host --shm-size=8g -p 8200:8000 \  -v /path/to/this/repo:/model:ro \  -e VLLM_TARGET_DEVICE=xpu -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \  -e XE2_KV_S2_NSG_DRAFT=32 -e XE2_KV_S2_NSG_VERIFY=8 -e K8V4_PREFILL_GATHER=triton -e K8V4_SDPA_ASYNC=0 \  vllm-openai-xpu:v0.30.0-k8v4-tp1-kv /model \  --host 0.0.0.0 --port 8000 --served-model-name qwen27-huihui-unc \  --gpu-memory-utilization 0.92 --dtype bfloat16 \  --max-model-len 262144 --kv-cache-dtype int8_k_int4_v --tensor-parallel-size 1 \  --max-num-seqs 4 --max-num-batched-tokens 4224 \  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \  --enable-prefix-caching --trust-remote-code \  --limit-mm-per-prompt '{"image":4,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}' \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[4,8,12,16]}'

For fp8 KV instead, use the vllm-openai-xpu:v0.30.0-k8v4-tp1 image with --kv-cache-dtype fp8 (319K-token pool). A stock vLLM XPU image loads the checkpoint too, with the embedding dense.

How it was made

Body — AutoRound 0.16.0 on the bf16 checkpoint, on the B70 (2.5 h, 27.5 GB peak VRAM):

bash

auto-round --model huihui-ai/Huihui-Qwen3.8-27B-abliterated --scheme W4A16 --bits 4 --group_size 128 \  --iters 200 --batch_size 8 --nsamples 256 --seqlen 2048 --seed 42 \  --dataset HuggingFaceH4/ultrachat_200k \  --ignore_layers "lm_head,.*visual.*,.*mtp.*,.*in_proj_a,.*in_proj_b" \  --format auto_round --low_gpu_mem_usage

AutoRound writes the 15 MTP tensors (dropped by the Transformers loader) back from the source into model_extra_tensors.safetensors.

GPTQ compatibility — tools/prepare_swift_autoround.py from k8v4-xpu: verifies the 400 packed linears and zero points, adds deterministic g_idx, switches the config to gptq. No weight values change.
Calibration — tools/calibrate_swift_bake.py (run at TP1 with fp8 KV on the single card): 128 WikiText-2 validation prompts of 384 tokens, 64 generated tokens each, MTP on; Hessians for lm_head and the MTP inputs, every 64th row held out.
Bake — tools/bake_gptq_resume.py: symmetric GPTQ INT4, group 128, 1% damping, no activation ordering.
— : rewritten shards without the dense replaced tensors, the 36 GPTQ arrays in , the INT8 embedding side file. Settings and per-linear errors are in .

Credits and licenses

Base model: Qwen/Qwen3.8-27B (Apache-2.0); abliteration: huihui-ai (Apache-2.0, LICENSE).
Body quantization recipe: ukisai/Swift-1.5-Qwen3.8-27b-W4A16-AutoRound; tool: Intel AutoRound.
Head/MTP bake, INT8 embedding and XPU serving stack: Wrapzii/k8v4-xpu; its GPTQ bake tooling is adapted from launch80/B65 (MIT, B65-LICENSE).
This derivative: weights requantized only — no fine-tuning, no merging.