Scheme
Same scheme and the same module coverage as the official
Qwen/Qwen3.8-27B-FP8 release, so it runs on
the same well-trodden vLLM kernel path.
Table | |
|---|
| Weights | FP8 e4m3, static, 128×128 blocks |
| Activations | FP8 e4m3, dynamic, per-token groups of 128 |
| Format | compressed-tensors (float-quantized) |
| Calibration | none (data-free) |
400 Linear modules are quantized:
q_proj/k_proj/v_proj/o_proj of the 16 full-attention layers
in_proj_qkv/in_proj_z/out_proj of the 48 gated-delta-net (linear-attention) layers
gate_proj/up_proj/down_proj of all 64 MLPs
Kept in BF16: in_proj_a/in_proj_b, conv1d, all norms, embed_tokens, lm_head, the
whole vision tower, and the MTP head (mtp.*, so MTP speculative decoding works).
Every non-quantized tensor is byte-identical to upstream (all 799 of them, checked
tensor by tensor), and so are the tokenizer, preprocessor configs, generation config and chat
template. The chat template is in turn byte-identical to the stock Qwen3.8-27B one.
Serving with vLLM
vllm serve batsclamp/Huihui-Qwen3.8-27B-abliterated-FP8-v2 \ --max-model-len 262144 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20}'
The shipped template uses Qwen's XML tool-call format, so the parser must be qwen3_coder
(not hermes). Sampling defaults follow the upstream thinking-mode recommendation.
The chat template accepts only reasoning_effort values xhigh, medium and low, and
raises on anything else (high, max, minimal), which an OpenAI-compatible server turns
into HTTP 400. If your client sends other values, map them in a small Jinja prelude before
the template.
Measured on one DGX Spark (GB10)
vLLM 0.32.0rc1 (nightly of 2026-10-16), --max-model-len 262144, MTP k=3,
--gpu-memory-utilization 0.55, kv-cache-dtype auto, no other GPU process resident:
Table | |
|---|
| Weights resident | 29.5 GiB |
| KV cache | 28.4 GiB, ~412k tokens (1.57× concurrency at 262k) |
| Decode, single stream, no thinking | 14.3 tok/s |
| TTFT, short prompt | 189 ms |
| MTP mean acceptance length | 2.5-3.9 |
| Kernel | CutlassFp8BlockScaledMMKernel |
Measured earlier on the same box and image: the previous build decoded at 13.4 tok/s, and
stock Qwen/Qwen3.8-27B-FP8 at 15.0 tok/s. Being 5 GiB heavier, the previous build also
left 5 GiB less for the KV cache at the same --gpu-memory-utilization.
Smoke-tested on vLLM: every reasoning_effort level and enable_thinking on/off return a
non-empty answer, tool calls (single, parallel, multi-turn with a tool result, and Python
code inside a streamed tool argument, 15/15) parse cleanly, JSON-schema structured output
and image input work.
Caveats
- This is an abliterated model: refusal behaviour has been removed upstream. Safety
properties are not those of the original Qwen release.
- Quantization was verified structurally (tensor by tensor against the source) and by
generation and tool-calling tests on vLLM; no perplexity or benchmark-suite numbers are
claimed.
- With
reasoning_effort: xhigh this model family can spend a very large output budget
inside the reasoning block. If you cap max_tokens below what it needs you get
finish_reason: length and an empty content. This is not specific to this
quantization or to abliteration; stock Qwen/Qwen3.8-27B-FP8 behaves the same way.