debackerl

Qwen3.8-27B-Uncensored-FP8-NO-MTP

README

License: apache-2.0

Why this build

Surveying 148 abliterated/uncensored Qwen3.8-27B repositories in August 2026 turned up exactly one FP8 build, and it was access-gated. Everything else was GGUF, NVFP4 (Blackwell-only), AWQ, or unquantized BF16. On Hopper hardware — where FP8 has native tensor-core support — that left no ungated option.

Only a handful of builds in any format preserve the MTP head, and getting it to actually work under vLLM required the checkpoint to be in the native fp8 format rather than compressed-tensors (see below).

Measured against the official build

Same hardware (H100 47GB vGPU), same vLLM 0.27.1, same flags.

Table with columns: Qwen/Qwen3.8-27B-FP8, this build
	`Qwen/Qwen3.8-27B-FP8`	this build
Thai extraction, exact codepoint (temp 0, 4 runs)	16/16	16/16
Coding tasks — generated code executed against assertions	4/5	4/5
Vision: Thai table, activity names read exactly	4/4	4/4
Vision: numeric cells	correct	correct
MTP acceptance rate	58%	52%
tok/s @ concurrency 1	131	132
tok/s @ concurrency 8	816	813
tok/s @ concurrency 16	—	1,373

The one coding task both models fail is the same one, for the same reason: the model spends its whole token budget reasoning and never emits an answer.

Hardware requirements

FP8 needs tensor-core support to be worth anything. On architectures without it, vLLM either refuses the checkpoint or falls back to a dequantize-then-multiply path that is slower than a 4-bit AWQ build.

Table with columns: Architecture, Examples, Compute cap., FP8 tensor cores, Runs this model
Architecture	Examples	Compute cap.	FP8 tensor cores	Runs this model
Volta	V100	7.0	✗	no — also lacks BF16 tensor cores, and vLLM has dropped Volta
Turing	RTX 20xx, T4	7.5	✗	no
Ampere	RTX 30xx, A100, A40, A6000	8.0 / 8.6	✗	see note below

On Ampere (A40, A100, A6000). These have the VRAM — an A40 has 48 GB, the same as the H100 47C this was tested on — but no FP8 tensor cores. vLLM can fall back to Fp8MarlinLinearMethod, which keeps weights in FP8 and dequantizes inside the kernel, so memory still works out. Whether that path accepts block-wise 128×128 scales (as opposed to per-tensor or per-channel) is untested here — we had no Ampere card to try it on. If you are on Ampere, an INT8 W8A16 or AWQ build of the same base is the safer choice.

VRAM

Weights occupy ~30 GB. The rest goes to KV cache, which decides how much context you can actually serve.

Table with columns: VRAM, Verdict, KV pool at --kv-cache-dtype fp8
VRAM	Verdict	KV pool at `--kv-cache-dtype fp8`
24 GB (RTX 4090)	will not load on one card	—
32 GB (V100 32GB)	no — wrong architecture entirely	—
48 GB (A40)	VRAM is fine; FP8 path is not — see note above	—
47–48 GB (H100 47C vGPU, L40S)	works	~350k tokens — measured on H100 47C
80 GB (H100/H200 80GB)	comfortable	~600k+ tokens

At 47 GB the whole 262,144-token context fits for a single request with room for roughly one more — Maximum concurrency for 262,144 tokens per request: 1.3x. Shorter contexts scale proportionally: ~20 concurrent requests at 16k each.

Without --kv-cache-dtype fp8, KV for 262k context needs 16.17 GiB instead of ~8.5 GiB and a 47 GB card will refuse to start.

Serving

bash
vllm serve pramoths/Qwen3.8-27B-Uncensored-FP8-MTP \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.94 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 16 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

--kv-cache-dtype fp8 is required for 262k context on a 47GB card — fp16 KV needs 16.17 GiB and will refuse to start. --max-num-seqs must stay below the available Mamba state block count; the default of 256 fails to boot.

Verify MTP is actually working — it can fail silently:

bash
curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total

A Mean acceptance length of 1.00 means every draft is being rejected. You are paying the drafting cost for nothing.

What was and was not quantized

Follows Qwen/Qwen3.8-27B-FP8 exactly — the distinction is matmul versus control parameter, not which component of the model:

FP8: q/k/v/o_proj and gate/up/down_proj, in the language layers and in the MTP head.
BF16: LayerNorms, gates, A_log, dt_bias, conv1d, and the gated-DeltaNet in_proj_a / in_proj_b — these govern recurrent state, where quantization error accumulates rather than cancelling.
BF16: the entire vision tower, lm_head, embed_tokens, mtp.fc.

Reproducing

Scripts, benchmarks, and a writeup of the five silent failure modes encountered along the way: https://github.com/pramoth/qwen38-fp8-forge

Quantization needs no calibration data and no GPU — roughly two minutes of CPU.

Attribution

Qwen/Qwen3.8-27B — original model, Apache-2.0
JonathanColetti/Qwen3.8-27B-Uncensored — abliterated base
Qwen/Qwen3.8-27B-FP8 — the quantization recipe this build copies

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

debackerl

Model Tree

Base

JonathanColetti/Qwen3.8-27B-Uncensored

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality