GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
mashima
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
mashima
Model Tree
huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated
Input Modalities
Output Modalities
Supported Functionality
The layout is identical to the official FP8 releases of this architecture
(Qwen/Qwen3.6-27B-FP8, bottlecapai/ThinkingCap-Qwen3.6-27B-FP8):
| Method | quant_method: fp8, DeepSeek-style block-wise |
| Block size | weight_block_size: [128, 128] |
| Scales | amax / 448.0, stored as bf16 under <module>.weight_scale_inv |
| Activation | dynamic |
| Quantized modules | 400 (MLP gate/up/down_proj, attention q/k/v/o_proj, linear-attn in_proj_qkv / in_proj_z / out_proj) |
Deliberately left in BF16 — quantizing these breaks the model:
conv1d, in_proj_a, in_proj_b, A_log, dt_bias, norminput_layernorm / post_attention_layernorm / q_norm / k_normlm_head, embed_tokensmodel.visual.*)The set of quantized modules was taken verbatim from the reference FP8 checkpoint rather than chosen by hand.
The quantizer was validated against bottlecapai/ThinkingCap-Qwen3.6-27B-FP8
(the FP8 release of this model's unablated parent). For modules that
abliteration does not touch, the output is bit-for-bit identical to the
reference checkpoint:
markdown
layers.3.mlp.gate_proj bit-identical: weight=True scale=Truelayers.3.mlp.up_proj bit-identical: weight=True scale=Truelayers.3.self_attn.q_proj bit-identical: weight=True scale=Truelayers.0.linear_attn.in_proj_qkv bit-identical: weight=True scale=Truelayers.0.linear_attn.in_proj_z bit-identical: weight=True scale=Truelayers.3.mlp.down_proj differs (abliteration target)layers.3.self_attn.o_proj differs (abliteration target)layers.0.linear_attn.out_proj differs (abliteration target)
Only the projections abliteration actually modifies differ — exactly as expected. Tensor names, dtypes and shapes all match the reference (1199 BF16 + 400 FP8 = 1599 tensors).
bash
vllm serve mashima/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-FP8 \--quantization fp8 \--dtype bfloat16 \--tensor-parallel-size 2 \--max-model-len 262144 \--kv-cache-dtype fp8 \--trust-remote-code \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Tested on vLLM v0.24.0.
Verified on 2× RTX 3090 (Ampere, sm_86) with TP=2, ~22 GB per card at
--gpu-memory-utilization 0.92, 262K context with fp8 KV cache.
Ampere has no FP8 tensor cores, so vLLM falls back to the Marlin W8A16 kernel (weight-only decompression). You get the memory saving but no FP8 compute speedup — expect this warning, which is normal:
markdown
Your GPU does not have native support for FP8 computation but FP8 quantizationis being used. Weight-only FP8 compression will be used leveraging the Marlin kernel.
On Ada / Hopper / Blackwell (sm_89+) FP8 runs natively.
MTP speculative decoding works: measured mean acceptance length 2.42 on 2× 3090.
This is a quantization of an abliterated (uncensored) model. The safety filtering of the base model has been significantly reduced. The warnings from huihui-ai apply unchanged:
Quantization does not change the behaviour of the base model in this respect. Credit for the base model and its abliteration goes to huihui-ai; please support their work there.
mtp.*--speculative-config