GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
lued
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
lued
Model Tree
DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP
Input Modalities
Output Modalities
Supported Functionality
On sm_86 (RTX 3090) there is no FP8 tensor-core path. vLLM --quantization fp8 falls back to Marlin: FP8 weights unpacked to FP16 every GEMM. W8A16 uses the same Marlin weight-only path with per-channel/group INT8, which measured lower weight error than FP8-Marlin on this GPU (rel-L2 ≈ 0.006 vs ≈ 0.026 on random-Gaussian probes) at essentially identical decode throughput.
| Format | Role on 2×3090 @ 262K | Notes |
|---|---|---|
| W8A16 (this repo) | Best default | Near-lossless weights; decode ≈ FP8-Marlin; fits 262K with selective GDN |
Online --quantization fp8 | Convenient | Same decode class, worse weight fidelity on Ampere |
| W8A8 INT8 | Prefill play | Native INT8 GEMMs help cold TTFT; ~10% decode regression on skinny batches |
| W4A16 AutoRound | Memory/decode play | ~19.5 GB; real long-context risk on hybrid GDN — see EMNLP’25 RULER@128K |
We need 262144 context with MTP + vision on 48 GB. Keeping all linear_attn.* in BF16 (TheHouseOfTheDude / havenoammo “max safety” recipe) costs ~11 GB for GDN and only left ~2 GB KV (~97K). Selective GDN (quantize qkv/z/out, keep gates) lands at ~14.9 GiB/GPU weights → ~5.0 GiB KV → ~282K tokens.
QuantizationModifier · scheme W8A16 · data-free RTN (no calibration set)compressed-tensors / pack-quantized (group INT8, memoryless_minmax)Linearlm_headmodel.visual.* (vision tower)yaml
# recipe.yamldefault_stage:default_modifiers:QuantizationModifier:targets: [Linear]ignore:- lm_head- re:.*visual.*- re:.*mtp.*- re:.*linear_attn[.]in_proj_a$- re:.*linear_attn[.]in_proj_b$scheme: W8A16
Sibling W8A16 of stock Qwen/Qwen3.6-27B with full linear_attn ignore
(TheHouseOfTheDude/Qwen3.6-27B-INT8):
On the same Fable-Fusion lineage, bf16 vs Q8_0 in a matched harness showed identical WikiText-2 PPL to four decimals (6.1978) — 8-bit is effectively free on this model when done carefully.
This checkpoint uses selective GDN quantization (required for 262K on 2×24 GB), so expect KLD in a similar ballpark, possibly slightly above the full-ignore Dude number. Measure on this repo with the script in the model card discussion / tools/measure_kld.py pattern: teacher = DavidAU BF16 logits, student = this W8A16, fixed prompts, mean KL(p_teacher || p_student) over next-token distributions (teacher-forced).
| Check | Result |
|---|---|
max_model_len | 262144 |
| Weights / GPU (TP2) | 14.85 GiB |
| Available KV (fp8_e4m3) | 5.01 GiB (~282K tokens) |
| Smoke gen | Exact string match |
| MTP accept (coding sample) | 72 / 75 draft tokens (96%) → ~3.9 tok/forward |
| Kernel path | CompressedTensorsWNA16 → Marlin |
bash
vllm serve lued/Qwen3.6-27B-Fable-Fusion-711-INT8-W8A16-MTP \--tensor-parallel-size 2 \--max-model-len 262144 \--gpu-memory-utilization 0.93 \--kv-cache-dtype fp8_e4m3 \--calculate-kv-scales \--enable-prefix-caching \--enable-chunked-prefill \--reasoning-parser qwen3 \--tool-call-parser qwen3_xml \--enable-auto-tool-choice \--disable-custom-all-reduce \--trust-remote-code \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
On GeForce dual-GPU without NVLink, keep NCCL_P2P_DISABLE=1 — the driver reports P2P as unsupported (GNS); the flag is a no-op for throughput and avoids init stalls.
From the upstream card (mxfp8 / mxfp4 of this lineage):
| arc/c | arc/e | boolq | hswag | obkqa | piqa | wino | |
|---|---|---|---|---|---|---|---|
| Fable-Fusion-711 mxfp8 | 0.711 | 0.879 | 0.910 | 0.790 | 0.514 | 0.823 | 0.763 |
| Fable-Fusion-711 mxfp4 | 0.701 | 0.873 | 0.909 | 0.786 |
W8A16 is intended to sit with the mxfp8 column, not the mxfp4 tradeoff.
model-*-of-*.safetensors — packed W8A16 language / GDN GEMM weights (+ BF16 vision & gates)model_mtp.safetensors — BF16 MTP head (~849 MB)recipe.yaml — exact llm-compressor recipechat_template.jinja / chat_template-instruct.jinja — from upstreamBASE_MODEL_README.md — full DavidAU cardApache-2.0 for this packaging. Respect upstream model terms for the base weights.
mtp.*linear_attn.in_proj_a / in_proj_b (GDN recurrent gates, out-dim 48)conv1d / A_log / dt_bias (non-Linear; never packed)in_proj_qkv / in_proj_z / out_proj| 0.488 |
| 0.813 |
| 0.759 |
| Qwen3.6-27B base mxfp8 | 0.647 | 0.803 | 0.910 | 0.773 | 0.450 | 0.806 | 0.742 |