GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
maci0
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
maci0
Model Tree
DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP
Input Modalities
Output Modalities
Supported Functionality
The base ships a multi-token prediction (MTP) head (mtp_num_hidden_layers: 1) that lets the
model self-speculate: it drafts the next token(s) and verifies in one pass, cutting decode
latency without a separate draft model.
That head sits on the acceptance-rate critical path. Quantizing it to 4-bit would push its drafts
away from what the full-precision main model predicts, the verifier would reject more of them, and
the speedup would shrink. So the recipe ignores re:.*mtp.* and keeps the head at bf16 (it is
tiny next to the 27B backbone). The 4-bit backbone does the heavy lifting; the full-precision MTP
head keeps acceptance high. The vision tower (re:.*visual.*) is kept at bf16 for the same
fidelity reason.
Near-lossless versus the bf16 source, ~20 GB vs 55.6 GB bf16 (~36%). GPTQ error compensation and an MSE observer keep the drop from bf16 minimal; the header lists the full characteristics and Quantization covers the recipe.
NVFP4 is auto-detected from config.json (compressed-tensors); no quantization flag needed.
--reasoning-parser qwen3 splits the <think> block into reasoning_content.
bash
vllm serve maci0/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4 \--served-model-name fable-fusion-27b-nvfp4 \--max-model-len 131072 \--gpu-memory-utilization 0.90 \--kv-cache-dtype fp8 \--reasoning-parser qwen3
--language-model-only to skip the vision tower and free KV cache for text use.The bf16 MTP head enables vLLM's built-in self-speculative decoding:
bash
vllm serve maci0/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4 \--served-model-name fable-fusion-27b-nvfp4 \--max-model-len 131072 --kv-cache-dtype fp8 --reasoning-parser qwen3 \--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
num_speculative_tokens matches the head depth (mtp_num_hidden_layers: 1). Acceptance is not
penalized by quantization because the head is bf16.
| Scheme | NVFP4, W4A4 |
| Weight rounding | GPTQ (Hessian-based error compensation), MSE observer |
| Weights | FP4 (E2M1), group_size=16, tensor_group, FP8 (E4M3) group scales, shared across fused layers |
| Activations | FP4, dynamic per-group, FP8 (E4M3) scales |
| Quantized | all language-model Linear layers |
| Kept in bf16 | MTP head (mtp.*), vision tower (model.visual.*), |
The model is loaded with AutoModelForImageTextToText so the vision tower is quantized-and-kept
(bf16) rather than dropped; the MTP head, which lives outside the transformers model graph, is
re-attached at bf16 from the base. GPTQ is a quantization-time cost only; inference speed and
format are identical to plain round-to-nearest NVFP4, but it chooses better 4-bit values.
DavidAU's Fable-Fusion-711 is a 27B Qwen3.5-family (qwen3_5) vision-language model with thinking mode, a 256K context window, and an MTP head, already run through Heretic uncensoring.
mtp_num_hidden_layers: 1) for speculative decoding.max_position_embeddings 262144).Thinking mode is the default.
temperature=0.6, top_p=0.95, top_k=20temperature=1.0, top_p=0.95, top_k=20temperature=0.7, top_p=0.80, top_k=20--speculative-config.--reasoning-parser is not auto-detected; pass it explicitly.Apache-2.0, following the base model. Intended use and all responsibility for use follow the base model.
lm_head| Untouched | gated delta-net Conv1d and SSM params (A_log, dt_bias), never Linear |