akumaburn

Swift-Qwen3.8-27b-heretic-W8A8-DFlash2

README

License: apache-2.0

Variants

Table with columns: build, size, served KL, first-token KL, DFlash2, best for
build	size	served KL	first-token KL	DFlash2	best for
BF16 source	52 GB	—	—	n/a	research, re-quantization
W8A8 + QuaRot/SmoothQuant	30 GB	0.0117	0.0244	❌ 1.00	maximum fidelity; MTP speculation
this — W8A8 unrotated	30 GB	0.0337	0.0417	✅ 4.14	general purpose: DFlash2 + full throughput
W4A16 unrotated	19 GB	0.0354	0.1146	✅ 4.35	smallest; single-user decode; low VRAM

served KL = KL(source ‖ build) under vLLM (real serving numerics), WikiText-2, top-512, 12,264 token positions. first-token KL = full-vocabulary at the first generated token over 100 harmless instruction prompts, under transformers.

[!NOTE] The two metrics disagree in an informative way. On served natural text the 4-bit and unrotated-8-bit builds are effectively tied (0.0354 vs 0.0337); the 4-bit penalty is concentrated at high-entropy decision points, where first-token KL puts it 2.7× worse. Conversely the rotation is worth 2.9× on the served metric but only 1.7× first-token — expected, since rotation exists to tame activation outliers for INT8 activation quantization, which only the served measurement exercises.

Performance

vllm bench serve, one CMP 170HX (64 GB), TP=1, fp8_e4m3 KV, async scheduling, random 1,024-token inputs. Prefill = c × input_len ÷ mean TTFT (instantaneous rate during prefill, not end-to-end input throughput).

Table with columns: c, prefill, no spec, prefill, DFlash2 k=7, decode, no spec, decode, DFlash2 k=7
c	prefill, no spec	prefill, DFlash2 k=7	decode, no spec	decode, DFlash2 k=7
1	3,695	2,878	39.9	110.7
8	9,072	7,755	233.3	379.1
16	13,820	5,400	363.8	369.5

[!TIP] Match the speculative depth to your concurrency. DFlash2 k=7 gives up to 2.8× decode at c=1 and still wins at c=8, but at c≥16 it costs throughput — at c=32 it cuts prefill from 18,942 to 2,768. This is speculative overhead on an already-saturated GPU (rejected drafts are wasted compute), not a property of the quantization: with speculation off, this checkpoint reaches 18,942 tok/s prefill, matching the rotated sibling.

Rule of thumb: DFlash2 for c ≤ 8, disable it for c ≥ 16.

Full DFlash2 sweep at 1,024-token inputs (k=7): prefill 2,878 / 4,134 / 6,209 / 7,755 / 5,400 / 2,768 and decode 110.7 / 159.8 / 232.7 / 379.1 / 369.5 / 396.9 for c = 1 / 2 / 4 / 8 / 16 / 32; acceptance length stays in the 3.3–4.2 band throughout (that sweep used randomized inputs; see Speculative acceptance for the per-domain figures).

Speculative acceptance

Greedy (temperature 0), k=7, concurrency 1, 48 prompts per set, DFlash2 drafter, identical harness for both checkpoints:

Table with columns: prompt set, mean chars, this (W8A8 unrot.), W4A16
prompt set	mean chars	this (W8A8 unrot.)	W4A16
code tasks (`harmless48`)	358	4.138	4.354
abliterated-content prose (`harmful_behaviors`)	73	3.110	3.206

Acceptance is governed by prompt domain, not by the checkpoint or by the abliteration. Structured code drafts far better than short open-ended prose (−32 % accepted tokens per draft, significant). A benign-prose control scores 3.267 against 3.206 for abliterated-content prose — a −2.7 % difference with a 95 % CI of ±0.16, i.e. not significant: abliteration does not degrade speculation.

The 4-bit sibling's acceptance edge over this build is small — +6.5 % on code (marginally significant) and not significant on prose. Its advantage is wall-clock decode (+36 % at c=1), traded against this build's much faster prefill and lower quantization error.

The drafter's block_size is 8, so k=7 is the architectural maximum.

[!NOTE] An earlier version of this card reported 3.36 / 33.8 %, measured on a 4-prompt file, alongside 5.12 for the 4-bit sibling. Both came from that too-small sample and overstated the gap between the builds; the 48-prompt figures above supersede them.

Quantization fidelity

KL(source ‖ build), measured two ways:

Table with columns: build, served top-512, first-token full-vocab
build	served top-512	first-token full-vocab
W8A8 + QuaRot/SmoothQuant	0.0117	0.0244
this (plain GPTQ, unrotated)	0.0337	0.0417
W4A16 unrotated	0.0354	0.1146

Dropping the rotation costs 2.9× on the served metric and 1.7× first-token. The served figure is the one to weigh: it is measured under vLLM with real INT8 activations, which is precisely what the rotation exists to protect. The first-token measurement runs in transformers with fp16 activations and so understates the rotation's value — the earlier "~1.7× cost" figure on this card was based on that metric alone and understated the trade.

That is the price of DFlash2 compatibility, and it is a real one. If you do not need the speculative drafter, the rotated sibling is meaningfully more faithful at identical size and speed.

Against the 4-bit build this checkpoint is ~2.7× closer on first-token KL but essentially tied on served KL (0.0337 vs 0.0354). Its advantages over W4A16 are prefill throughput (native CUTLASS INT8 vs Marlin dequant, ~2× at short prompts) and high-entropy fidelity — not general output quality on ordinary text. It costs 11 GB more VRAM for that.

Refusal behaviour is inherited unchanged from the source: 0/100 hard refusals, with the same ~20–24 % chain-of-thought deflection band described on the source card.

Serving

Choosing a drafter

The drafter is a separate checkpoint from this model. Two work, and which one you can use depends on your vLLM build:

Table with columns: drafter, size, needs a patch?, decode (c=1)
drafter	size	needs a patch?	decode (c=1)
`incoai/Qwen3.8-27B-DFlash2` (BF16)	3.6 GB	no — works on stock vLLM 0.29.0	122.4 tok/s
`syvai/Qwen3.8-27B-DFlash2-W4A16`	1.2 GB	yes	127.1 tok/s

Stock vLLM 0.29.0 already ships DFlash2, but _build_fused_kv_buffers reads qkv_proj.weight directly. A pack-quantized (W4A16) drafter has no such attribute, so it dies at startup with

text

AttributeError: 'QKVParallelLinear' object has no attribute 'weight'  vllm/model_executor/models/qwen3_dflash.py:472

A BF16 drafter takes the dense fast path and needs no patch at all — that is the simplest way to run this model. The one-function fix for the quantized drafter is at github.com/akumaburn/vllm-dflash2.

[!CAUTION] Serving past 262,144 tokens kills the engine. Both drafters declare max_position_embeddings: 262144. At a longer position the drafter's rope table is indexed out of bounds and the CUDA assert index out of bounds: 0 <= ... < 262144 takes down the whole vLLM engine (EngineDeadError), not just the request. This is reachable whenever the target is YaRN-extended past its native 262,144 context.

Fix: raise max_position_embeddings in the drafter's config.json to your served length, or use the ready-made akumaburn/Qwen3.8-27B-DFlash2-W4A16-longctx (identical weights, that one field set to 393216). The weights are untouched, and it is safe because the drafter uses sliding-window (2048) attention — only relative positions inside the window matter. Measured with the enlarged table: acceptance 3.82 at 150k and 2.70 at 300k (degraded by extrapolation beyond the drafter's trained range, but working). Unfixed, 300k is a hard crash.

Measured on identical hardware and harness, the quantized drafter is worth +4 % decode and 2.4 GB of VRAM; acceptance between the two is statistically identical. At long context the VRAM is the bigger prize, since it becomes KV cache.

# single user / low concurrencyvllm serve akumaburn/Swift-Qwen3.8-27b-heretic-W8A8-DFlash2 \  --tensor-parallel-size 1 --max-model-len 32768 \  --kv-cache-dtype fp8_e4m3 --async-scheduling \  --speculative-config '{"method":"dflash","model":"<path>/Qwen3.8-27B-DFlash2-W4A16","num_speculative_tokens":7}' # batch serving (c >= 16): drop the speculative config entirely

All quantized matmul dimensions are multiples of 128 (no INT8 padding).

Safety

Refusal behaviour has been deliberately removed; this model produces content the source declines, including dangerous, illegal or unethical material, with no moderation. Intended for interpretability/safety research, red-teaming and evaluation by people who understand and accept those risks. Do not deploy it where it can reach people who have not consented to unfiltered output. You are responsible for your use and for compliance with all applicable laws.

Disclaimer

This model is provided for research purposes only, "AS IS", without warranty of any kind, express or implied. The author accepts no liability for any use of this model or any consequences arising from it. By downloading or using it, you accept sole responsibility for your use and for compliance with all applicable laws and regulations.

Licences: the Swift contribution is under the Swift Open License v1.0 (© UkisAI); the underlying Qwen3.8-27B is Apache-2.0 (© Alibaba Cloud). Both licence files are included. Abliteration via Heretic; quantization via llm-compressor.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

akumaburn

Model Tree

Base

akumaburn/Swift-Qwen3.8-27b-heretic

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality