akumaburn

Swift-Qwen3.8-27b-heretic-W4A16

README

License: apache-2.0

Variants

Table with columns: build, size, served KL, first-token KL, DFlash2, best for
build	size	served KL	first-token KL	DFlash2	best for
BF16 source	52 GB	—	—	n/a	research, re-quantization
W8A8 + QuaRot/SmoothQuant	30 GB	0.0117	0.0244	❌ 1.00	maximum fidelity; MTP speculation
W8A8 unrotated	30 GB	0.0337	0.0417	✅ 4.14	general purpose: DFlash2 + full throughput
W4A16 unrotated	19 GB	0.0354	0.1146	✅ 4.35	smallest; single-user decode; low VRAM

served KL = KL(source ‖ build) under vLLM (real serving numerics), WikiText-2, top-512, 12,264 token positions. first-token KL = full-vocabulary at the first generated token over 100 harmless instruction prompts, under transformers.

[!NOTE] The two metrics disagree in an informative way. On served natural text the 4-bit and unrotated-8-bit builds are effectively tied (0.0354 vs 0.0337); the 4-bit penalty is concentrated at high-entropy decision points, where first-token KL puts it 2.7× worse. Conversely the rotation is worth 2.9× on the served metric but only 1.7× first-token — expected, since rotation exists to tame activation outliers for INT8 activation quantization, which only the served measurement exercises.

DFlash2 column is acceptance length on harmless48 (48 code/prose tasks, greedy, k=7; 1.00 = every draft rejected). Acceptance is governed by prompt domain, not by the build: the same checkpoints score ~3.2 on conversational prose. See Speculative acceptance below.

[!TIP] Match speculative depth to concurrency. Measured on the unrotated W8A8 (same model and dataset, only the speculative config varying, 1,024-token inputs): DFlash2 k=7 gives 2.8× decode at c=1 and still wins at c=8, but at c≥16 it costs throughput — at c=32 prefill falls from 18,942 to 2,768 tok/s. That is speculative overhead on a saturated GPU, not a property of any checkpoint. Use DFlash2 for c ≤ 8; disable it for c ≥ 16.

Choosing between this and the 8-bit siblings

These are different operating points, not better/worse. Measured on one CMP 170HX (64 GB), TP=1, fp8_e4m3 KV, async scheduling, identical datasets. Two workloads are shown because the answer depends on prompt length — quoting one number for "prefill" is misleading.

Long prompts (~16k in, 16 out):

Table with columns: this (W4A16, DFlash2 k=7), W8A8 + QuaRot (MTP k=3)
	this (W4A16, DFlash2 k=7)	W8A8 + QuaRot (MTP k=3)
decode @ c=1	154 tok/s	73 tok/s
decode @ c=32	578 tok/s	840 tok/s
prefill (flat across c)	~1,750 tok/s	~3,350 tok/s

Short prompts (1,024 in, 256 out) — prefill = c × in_len ÷ mean TTFT:

Table with columns: c, this prefill, W8A8 prefill, this decode, W8A8 decode
c	this prefill	W8A8 prefill	this decode	W8A8 decode
1	1,868	3,349	103.4	69.1
4	4,028	7,539	199.2	171.9
32	1,804	18,621	255.6

Table with columns: this (W4A16), 8-bit siblings
	this (W4A16)	8-bit siblings
size	19 GB	30 GB
served KL	0.0354	0.0117 (rotated) / 0.0337 (unrotated)
first-token KL	0.1146	0.0244 (rotated) / 0.0417 (unrotated)

Interactive / single user (c ≤ 4): this build — up to 1.5× the decode throughput of the 8-bit builds at 1k-token inputs, and 11 GB of freed VRAM.
Anything larger, or prefill-heavy: an 8-bit sibling. Prefill here is a flat ~0.55× of W8A8 (Marlin int4→bf16 dequant vs native CUTLASS INT8), and decode saturates around 255 tok/s where W8A8 climbs past 500.
If you want DFlash2 and 8-bit fidelity: the unrotated W8A8 — same speculative capability, 2.7× closer to the source on first-token KL, though statistically tied with this build on served KL (0.0337 vs 0.0354). Choose it for prefill throughput and high-entropy fidelity, not for a blanket quality win.

Performance

vllm bench serve, TP=1, fp8_e4m3 KV, async scheduling, DFlash2 drafter with num_speculative_tokens=7.

Table with columns: c, Prefill (input tok/s), Decode (output tok/s), TTFT p50, TPOT p50, DFlash2 acceptance length
c	Prefill (input tok/s)	Decode (output tok/s)	TTFT p50	TPOT p50	DFlash2 acceptance length
1	1,762	154.1	10.5 s	5.7 ms	3.74
2	1,778	236.3	20.0 s	7.2 ms	3.71
4	1,760

Prefill point: distinct ~16k-token realistic prompts (concatenated code/math), 16 output tokens. Throughput is compute-saturated at ~1,750 tok/s for this workload and does not scale with c; TTFT grows linearly as long-prompt requests queue. (On synthetic random-token prompts the same server does scale with c — the saturation is workload-specific, not universal.) The flat ~0.52× versus the W8A8 sibling is the Marlin int4→bf16 dequant path versus native CUTLASS INT8 GEMMs.
Decode point: realistic code/math prompts (~140 input tokens), 2,048 output cap. Decode saturates above c≈16 (570 → 578) as the model stops being weight-bandwidth-bound; the 8-bit sibling keeps climbing to 840.
DFlash2 acceptance is healthy and stable (3.7–3.9) across the whole range.

Speculative acceptance

Greedy (temperature 0), k=7, concurrency 1, 48 prompts per set, DFlash2 drafter, identical harness for both checkpoints:

Table with columns: prompt set, mean chars, this (W4A16), W8A8 unrotated
prompt set	mean chars	this (W4A16)	W8A8 unrotated
code tasks (`harmless48`)	358	4.354	4.138
benign prose (`harmless_alpaca`)	63	3.267	—
abliterated-content prose (`harmful_behaviors`)	73

Acceptance is governed by prompt domain, not by the checkpoint or by the abliteration. Structured code — boilerplate, rigid syntax, predictable continuations — drafts far better than short open-ended prose (−32 % accepted tokens per draft, 95 % CI ±0.17, significant). Whether that prose is benign or is exactly the content the model was decensored for makes no measurable difference (−2.7 %, 95 % CI ±0.16 — not significant). Abliteration does not degrade speculation.

Against the 8-bit sibling the acceptance edge is small: +6.5 % on code (95 % CI ±0.18, marginally significant) and not significant on prose. This build's real speculative advantage is wall-clock rather than acceptance: +36 % decode throughput at c=1 on both sets, because 4-bit weights are cheaper to stream.

The drafter's block_size is 8, so k=7 is the architectural maximum — there is no headroom from drafting deeper.

[!NOTE] An earlier version of this card reported 5.12 / 58.9 %, measured on a 4-prompt file. That sample was far too small and optimistic; the 48-prompt figures above supersede it. The 8-bit sibling's previously published 3.36 was measured on that same 4-prompt file and is likewise superseded by 4.138 — the two builds are much closer than those numbers implied.

Quantization fidelity

KL(source ‖ this), measured two ways — they disagree, and the disagreement is the most useful thing on this card:

Table with columns: metric, this (W4A16), W8A8 rotated, W8A8 unrotated
metric	this (W4A16)	W8A8 rotated	W8A8 unrotated
served top-512, WikiText-2, 12,264 positions, under vLLM	0.0354	0.0117	0.0337
first-token full-vocab, 100 harmless prompts, `transformers`	0.1146	0.0244	0.0417

On the first-token metric 4-bit weights cost ~4.7× the divergence of the rotated 8-bit build — and it exceeds the abliteration's own divergence from the unmodified Swift model (0.0763), meaning quantization perturbs this checkpoint more than the uncensoring did.

On the served metric that gap almost vanishes: 0.0354 here versus 0.0337 for the unrotated 8-bit build at the same activation precision story — a 5 % difference, not a 2.7× one. Averaged over 12,264 positions of ordinary text the two are effectively equivalent.

Both numbers are real; they measure different things. Natural-language continuation is dominated by low-entropy, highly predictable positions where 4-bit weights are adequate. The first-token metric samples exactly the opposite regime — the high-entropy moment where an instruction's response is chosen among many plausible openings — and there the coarser weight grid costs real accuracy. Expect this build to feel close to 8-bit on flowing prose and continuation, and to diverge most at decision points: the first token of a reply, branch points in reasoning, structured-format choices.

The rotated W8A8 leads on both metrics, and by more on served (2.9×) than first-token (1.7×) — consistent with the rotation's actual job, which is taming activation outliers for INT8 activation quantization. Only the served measurement exercises INT8 activations at all; the transformers first-token measurement runs them in fp16 and therefore understates what rotation buys.

Refusal behaviour is inherited unchanged from the source: 0/100 hard refusals, with the same ~20–24 % chain-of-thought deflection band described on the source card.

Serving

Choosing a drafter

The drafter is a separate checkpoint from this model. Two work, and which one you can use depends on your vLLM build:

Table with columns: drafter, size, needs a patch?, decode (c=1)
drafter	size	needs a patch?	decode (c=1)
`incoai/Qwen3.8-27B-DFlash2` (BF16)	3.6 GB	no — works on stock vLLM 0.29.0	122.4 tok/s
`syvai/Qwen3.8-27B-DFlash2-W4A16`	1.2 GB	yes	127.1 tok/s

Stock vLLM 0.29.0 already ships DFlash2, but _build_fused_kv_buffers reads qkv_proj.weight directly. A pack-quantized (W4A16) drafter has no such attribute, so it dies at startup with

markdown
AttributeError: 'QKVParallelLinear' object has no attribute 'weight'
  vllm/model_executor/models/qwen3_dflash.py:472

A BF16 drafter takes the dense fast path and needs no patch at all — that is the simplest way to run this model. The one-function fix for the quantized drafter is at github.com/akumaburn/vllm-dflash2.

[!CAUTION] Serving past 262,144 tokens kills the engine. Both drafters declare max_position_embeddings: 262144. At a longer position the drafter's rope table is indexed out of bounds and the CUDA assert index out of bounds: 0 <= ... < 262144 takes down the whole vLLM engine (EngineDeadError), not just the request. This is reachable whenever the target is YaRN-extended past its native 262,144 context.

Fix: raise max_position_embeddings in the drafter's config.json to your served length, or use the ready-made akumaburn/Qwen3.8-27B-DFlash2-W4A16-longctx (identical weights, that one field set to 393216). The weights are untouched, and it is safe because the drafter uses sliding-window (2048) attention — only relative positions inside the window matter. Measured with the enlarged table: acceptance 3.82 at 150k and 2.70 at 300k (degraded by extrapolation beyond the drafter's trained range, but working). Unfixed, 300k is a hard crash.

Measured on identical hardware and harness, the quantized drafter is worth +4 % decode and 2.4 GB of VRAM; acceptance between the two is statistically identical. At long context the VRAM is the bigger prize, since it becomes KV cache.

sh
vllm serve akumaburn/Swift-Qwen3.8-27b-heretic-W4A16 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --async-scheduling \
  --speculative-config '{"method":"dflash","model":"<path-to>/Qwen3.8-27B-DFlash2-W4A16","num_speculative_tokens":7}'

Without a drafter it serves as a plain W4A16 checkpoint on the Marlin kernel. Optional: VLLM_MARLIN_INPUT_DTYPE=int8 makes Marlin consume INT8 activations on these same weights (an effective "W4A8" without a second checkpoint); on patched builds VLLM_MARLIN_INT8_INCLUDE_RE=mlp applies it selectively. Neither was used for the numbers above.

All quantized matmul dimensions are multiples of 128 (no INT4 padding).

Safety

Refusal behaviour has been deliberately removed; this model produces content the source declines, including dangerous, illegal or unethical material, with no moderation. Intended for interpretability/safety research, red-teaming and evaluation by people who understand and accept those risks. Do not deploy it where it can reach people who have not consented to unfiltered output. You are responsible for your use and for compliance with all applicable laws.

Disclaimer

This model is provided for research purposes only, "AS IS", without warranty of any kind, express or implied. The author accepts no liability for any use of this model or any consequences arising from it. By downloading or using it, you accept sole responsibility for your use and for compliance with all applicable laws and regulations.

Licences: the Swift contribution is licensed under the Swift Open License v1.0 (© UkisAI); the underlying Qwen3.8-27B is Apache-2.0 (© Alibaba Cloud). Both licence files are included. Abliteration via Heretic; quantization via llm-compressor / compressed-tensors.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

akumaburn

Model Tree

Base

akumaburn/Swift-Qwen3.8-27b-heretic

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality