GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
munekazu
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
munekazu
Model Tree
huihui-ai/Huihui-Qwen3.8-27B-abliterated
Input Modalities
Output Modalities
Supported Functionality
Re-quantized from the updated base model
(commit 739e3c5,
uploaded 2026-08-24), in which huihui-ai narrowed the abliteration to a band
of middle layers instead of the whole upper stack. The narrower band keeps
more of the parent model's behaviour intact while still removing refusals.
Measured against the parent FP8 checkpoint, this revision modifies only the
residual-writing projections (mlp.down_proj, self_attn.o_proj /
linear_attn.out_proj):
| this revision | previous revision | |
|---|---|---|
mlp.down_proj changed in | layers 17–51 (35) | layers 15–63 (49) |
| attn out-projection changed in | layers 17–51 | layers 15–63 |
gate/up/q/k/v_proj, GDN input projections | untouched | untouched |
| MTP head, vision tower | untouched | untouched |
(huihui's model card describes the band as "layers 18 to 51"; the per-layer weight diff puts the lower edge one layer earlier, at 17.) For layers 17–51 the ablation transform is byte-identical to the previous revision — only the range changed, releasing layers 15–16 and 52–63 back to parent values.
The quantization recipe, module set and on-disk layout are unchanged from the previous revision; only the underlying weights are newer. The earlier revision of this repo is still reachable in the git history if you want the wider ablation.
The layout is identical to the official FP8 release of this model
(Qwen/Qwen3.8-27B-FP8):
| Method | quant_method: fp8, DeepSeek-style block-wise |
| Block size | weight_block_size: [128, 128] |
| Scales | amax / 448.0, stored as bf16 under <module>.weight_scale_inv |
| Activation | dynamic |
| Quantized modules | 407 (MLP gate/up/down_proj, attention q/k/v/o_proj, linear-attn in_proj_qkv / / , the MTP head's / projections) |
Unlike the earlier Qwen3.6 FP8 releases, the official Qwen3.8 FP8 checkpoint
also quantizes part of the MTP head (mtp.layers.0.self_attn.* and
mtp.layers.0.mlp.*) — this release matches that. The quantized-module set
is taken verbatim from the reference checkpoint rather than chosen by hand,
so it tracks whatever the official release does.
Deliberately left in BF16 — quantizing these breaks the model:
conv1d, in_proj_a, in_proj_b, A_log, dt_bias, norminput_layernorm / post_attention_layernorm / q_norm / k_normlm_head, embed_tokensmodel.visual.*)The quantizer was validated against Qwen/Qwen3.8-27B-FP8 (the FP8 release
of this model's unablated parent). Every module abliteration did not touch
quantizes to the reference FP8 bit-for-bit; a full per-layer sweep of the
residual-writing projections confirms the 17–51 band precisely:
markdown
layers 0–16, 52–63 + entire MTP head — bit-identical to referencelayers 17–51 (down_proj / out-proj) — differ, as expectedgate_proj / up_proj / q_proj (any layer) — bit-identical to reference
Tensor names, dtypes and shapes all match the reference (1199 BF16 + 407 FP8
= 1606 tensors; quantization_config identical to both the reference and the
previous revision).
bash
vllm serve munekazu/Huihui-Qwen3.8-27B-abliterated-FP8 \--quantization fp8 \--dtype bfloat16 \--tensor-parallel-size 2 \--max-model-len 262144 \--kv-cache-dtype fp8 \--trust-remote-code \--reasoning-parser qwen3 \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Tested on vLLM v0.24.0.
Uses the model's bundled chat_template.jinja, which is reasoning_effort-based
(low / medium / high / xhigh) rather than the older enable_thinking
boolean. Pass e.g. "chat_template_kwargs": {"reasoning_effort": "low"} in
requests, or set --default-chat-template-kwargs server-side.
⚠️ Known issue observed on the previous revision (vLLM v0.24.0): with
--reasoning-parser qwen3, reasoning_content comes back empty in both
streaming and non-streaming responses, even though the model is actually
thinking (verified via the raw /v1/completions endpoint, which shows a normal
<reasoning>...</think> generation). Feeding the exact same generated text to
the parser directly (outside the server) extracts reasoning/content correctly,
so the bug looks like it's in how vLLM v0.24.0's decode loop drives the parser
during generation, not in the parser's extraction logic itself. Not
root-caused further. Answer quality is unaffected either way; you just don't
get the reasoning trace back over the API.
Verified on 2× RTX 3090 (Ampere, sm_86) with TP=2, ~22 GB per card at
--gpu-memory-utilization 0.92 and 262K context with fp8 KV cache.
Ampere has no FP8 tensor cores, so vLLM falls back to the Marlin W8A16 kernel (weight-only decompression). You get the memory saving but no FP8 compute speedup — expect this warning, which is normal:
markdown
Your GPU does not have native support for FP8 computation but FP8 quantizationis being used. Weight-only FP8 compression will be used leveraging the Marlin kernel.
On Ada / Hopper / Blackwell (sm_89+) FP8 runs natively.
MTP speculative decoding works: measured mean acceptance length 2.22 on 2× 3090.
This is a quantization of an abliterated (uncensored) model. The safety filtering of the base model has been significantly reduced. The warnings from huihui-ai apply unchanged:
Quantization does not change the behaviour of the base model in this respect. Credit for the base model and its abliteration goes to huihui-ai; please support their work there.
in_proj_zout_projself_attnmlpmtp.fc