davetha

Qwen3.8-27B-ABLITERATED-W8A8-gdnint8

README

License: apache-2.0

What is different from a normal W8A8 of this model

This checkpoint additionally quantizes the three large gated-delta-net (GDN) projections — in_proj_qkv, in_proj_z, out_proj — across all 48 linear_attention layers. That is a deliberate departure from the usual recipe for this architecture, which protects the whole linear-attention path as quantization-sensitive. Read the evaluation section before using it; a plain W8A8 build is the conservative choice.

It buys, on one MI210:

Table with columns: W8A8 (GDN bf16), this checkpoint
	W8A8 (GDN bf16)	this checkpoint
decode, MTP N=1	40.5 tok/s	50.0 tok/s (+23.5%)
per forward step	43.68 ms	35.68 ms
KV cache capacity	298,381 tok	374,711 tok (+26%)
on disk	35 GB	30 GB

The GDN projections were 5.54 B parameters sitting in BF16 — 11.07 GB read every decode step, against 19.3 GB for everything already quantized. On a bandwidth-bound decode that was ~30% of kernel time in one unquantized tensor family.

What is quantized

Everything the parent W8A8 recipe quantizes (MLPs on all 64 layers, self_attn projections on the 16 full_attention layers), plus:

Table with columns: added here, count
added here	count
`linear_attn.in_proj_qkv`	48	10240x5120
`linear_attn.in_proj_z`	48	6144x5120
`linear_attn.out_proj`	48	5120x6144

Still BF16, deliberately:

Table with columns: protected, why
protected	why
`linear_attn.in_proj_a`, `linear_attn.in_proj_b`	48x5120 each — 0.42% of the block's bytes, but they feed the recurrent gating (`a` → softplus → exp → decay, `b` → sigmoid → beta). Error there compounds along the sequence for no meaningful saving.
`linear_attn.conv1d`, all `norm`	small, sensitive
`re:.visual.`	vision tower
`mtp.*`	MTP draft head, kept BF16 for speculative decoding

Scheme

Symmetric per-output-channel INT8 weights, matching the parent checkpoint exactly:

markdown
scale = max|W| / 127        # computed in fp32, then STORED AS BF16
q     = round(W / scale.to(bf16)).clamp(-128, 127)

Casting the scale to BF16 before dividing is load-bearing — it is what reproduces the parent's statistics (row maxima usually 127 but often 120–126, and -128 present). Dividing by an fp32 scale and casting afterwards yields subtly different weights.

Activations are unchanged: 8-bit per-token dynamic, computed at inference.

No calibration data is used or needed — this step is a pure weight transform. quantize_gdn_int8.py in this repo reproduces it from the parent checkpoint exactly. Max relative weight error introduced: 0.39%.

Evaluation — and its limits

Measured against the same checkpoint with GDN projections left in BF16, same harness, same hardware.

Perplexity: +2.3% (token-weighted over 4,437 tokens)

Table with columns: corpus, tokens, W8A8, this
corpus	tokens	W8A8	this
prose	317	4.6624	4.3646
source A	2,347	2.3573	2.4441
source B	1,773	1.9029	1.9409

Long-context recall: 9/9, identical to the unquantized-GDN control

Table with columns: prompt tokens, depths 10/50/90%, W8A8, this
prompt tokens	W8A8	this
14,147	3/3	3/3
56,788	3/3	3/3
107,337	3/3	3/3

The needle test was run against both checkpoints on purpose: this hardware declines the fast paged-attention kernel, so a long-context failure could have been the attention path rather than the weights. Running both makes the result attributable.

What was NOT evaluated. No standard benchmark suite (MMLU, GSM8K, HumanEval, etc). No multi-turn or long-generation testing. No coding or math evaluation. The perplexity probe is three passages, two of them Python source from one project. The needle test is a single fact. Given that this deliberately quantizes a path the usual recipe protects, and that GDN is recurrent so errors can compound in ways short probes do not reveal, treat these numbers as a smoke test, not a quality claim. If you evaluate it properly, please open a discussion with what you find.

Running it

vLLM on ROCm, single MI210:

bash
vllm serve <this-model> \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.90 \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

with VLLM_ROCM_USE_AITER=1. Notes for gfx90a specifically:

VLLM_ROCM_USE_AITER=1 needs an image whose AITER carries gfx90a code objects, or the INT8 GEMM silently falls back to a generic Triton kernel at roughly a third of the rate. Confirm with grep 'Selected.*ScaledMMLinearKernel' in the server log — you want AiterInt8ScaledMMLinearKernel, not TritonInt8ScaledMMLinearKernel.
On a host with GPUs of more than one architecture, pass only the gfx90a render nodes (--device=/dev/dri/renderD<N>, not all of /dev/dri). vLLM resolves the architecture once at import from amdsmi's physical device 0 and ignores HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES, so a foreign card can switch every gfx9 path off silently. That alone was worth 2.4x here.
cudagraph_mode: FULL_DECODE_ONLY matters — vLLM otherwise drops to PIECEWISE under speculative decoding.
num_speculative_tokens: 1; N=2 measured on this part.

Tooling for gfx90a: davetha/mi210-vllm.

Provenance

Qwen/Qwen3.8-27B → Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (abliteration) → W8A8 INT8 (llm-compressor, weights RTN + dynamic per-token activations) → this (GDN projections additionally INT8, quantize_gdn_int8.py).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

davetha

Model Tree

Base

Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality