GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
davetha
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
davetha
Model Tree
Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16
Input Modalities
Output Modalities
Supported Functionality
Weights 8-bit, group size 128, symmetric. Activations are not quantized (pack-quantized), so this is weight-only int8.
Quantized with llm-compressor 0.12.1a20260701, QuantizationModifier
(round-to-nearest). No calibration data was used and none is needed: the scheme is weight-only, so scales come straight from the weights.
Qwen3.8-27B is a hybrid: layer_types is 48 linear_attention (GDN) layers
and 16 full_attention, full_attention_interval: 4. Only 16 of 64 layers hold
a KV cache, which is why its context is cheaper than the layer count suggests.
Of 985 modules, 256 are quantized and 729 are left at BF16:
| quantized | count |
|---|---|
model.language_model.layers.N.mlp.{gate,up,down}_proj | 192 (all 64 layers) |
model.language_model.layers.N.self_attn.{q,k,v,o}_proj | 64 (the 16 full-attention layers) |
| protected | count | why |
|---|---|---|
re:.*linear_attn.* | 384 | all 48 GDN layers, incl. conv1d / in_proj_* / out_proj |
re:.*visual.* | 167 | vision tower |
re:.*norm.* | 271 | norms |
re:^mtp\..* |
MLPs dominate parameter count, so quantizing them captures most of the benefit while the sensitive linear-attention path stays untouched.
The 15 mtp.* tensors are preserved at BF16. Note they live in the top-level
mtp. namespace, not under model., so naive ignore patterns miss them.
On Qwen3_5Config, AutoModelForCausalLM maps to Qwen3_5ForCausalLM — the
text-only class. It loads without error and silently drops all 54 vision
blocks. Use AutoModelForImageTextToText, which maps to
Qwen3_5ForConditionalGeneration.
bash
vllm serve davetha/Qwen3.8-27B-abliterated-W8A16 \--tensor-parallel-size 2 --max-model-len 131072 \--gpu-memory-utilization 0.85 --trust-remote-code \--enable-auto-tool-choice --tool-call-parser qwen3_xml \--reasoning-parser qwen3
qwen3_xml, not hermes: the chat template emits
<tool_call><function=name><parameter=x> XML, which the hermes parser does not
match.
On gfx90a note that head_dim = 256, which upstream vLLM's ROCm paged-attention
gate rejects (it accepts only 64 or 128).
0.27.2rc0+mi210.1, TP=2 on 2x MI21017*23 -> 391; 2:15pm–6:40pm -> 265 minutes;
string-reverse one-liner -> s = s[::-1]Not measured: throughput against the BF16 base, and no perplexity or benchmark suite has been run. Treat quality claims as unverified beyond the spot checks above.
The base model is abliterated, so the usual refusal behaviour is substantially removed. You are responsible for how you use it.
| 15 |
| MTP draft head, kept BF16 for speculative decoding |
lm_head, embed_tokens | 2 |