GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
DoktorMincs
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
DoktorMincs
Model Tree
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
Input Modalities
Output Modalities
Supported Functionality
Perplexity measured on 50 held-out texts from wikitext-103, 16,007 evaluated tokens. Deliberately not the calibration set. Same texts, same tokenization, same method (raw text, no chat template), both models served by vLLM.
This checkpoint is the third and highest-fidelity arm of a series run on the same protocol. The BF16 baseline reproduced identically (8.0880) across the W6A16 and W8A16 runs, so the deltas below are directly comparable:
| Model | Bits | iters × samples | BF16 | quantized | delta |
|---|---|---|---|---|---|
…abliterated (different checkpoint) | 4 | 200 × 128 | 8.1133 | 8.8426 | +8.99% |
| this checkpoint | 6 | 200 × 128 | 8.0880 | 8.6836 | +7.36% |
| this checkpoint (this repo) | 8 | 400 × 256 | 8.0880 | 8.5658 | +5.91% |
+5.91% is still above the ~5% degradation usually accepted for weight-only quantization, even though this is 8-bit with double the tuning budget. Two honest caveats:
Three variables changed at once versus the W4A16 baseline — bit width, iterations and calibration samples. So this is a best-effort fidelity run, not a controlled isolation of the bit-width effect. Do not read the delta as "what 8 bits buys you".
The returns are clearly diminishing: +8.99% → +7.36% → +5.91%. Doubling bits and the tuning budget bought roughly one percentage point. That pattern suggests the residual error is not dominated by weight precision — if it were, 8-bit (int8, group 128) would typically land well under 1–2%.
A plausible next lever, and the one we would try first, is excluding the GatedDeltaNet
(linear-attention) projections from quantization — keeping in_proj_qkv, in_proj_z and the
linear-attention out_proj in BF16 — since hybrid linear-attention layers are a known sensitivity
in this architecture family. That trades some compression for fidelity instead of pushing the bit
width further on the same modules. Note this is a hypothesis, not a measured result.
Perplexity measures language-modelling fidelity only; it is not a substitute for task-specific evaluation. Short side-by-side chat samples against the BF16 original read as equivalent, which is exactly why the perplexity number — not eyeballing — is what is published here.
8-bit compressed-tensors weights are served by Marlin, so unlike the 6-bit build of this model there is no dependency on the JIT-compiled Humming kernels:
text
Using MarlinLinearKernel for CompressedTensorsWNA16No throughput advantage was measured. On the final checkpoints, same script and TP2, the W8A16 model was slower on plain decode (24.6 vs 26.5 tok/s) and faster with MTP speculative decoding (42.6 vs 34.5) and on vision (19.4 vs 17.3). These are 2-prompt cold-JIT runs, not benchmarks; the sign flips between phases, so no reliable throughput claim can be made in either direction. The reason to prefer this build is fidelity, and it costs ~6 GB more than the 6-bit build.
bash
vllm serve DoktorMincs/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W8A16-AutoRound \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --reasoning-parser qwen3 \ --enable-prefix-caching \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'in_proj_a/b correctly left in BF16.max_tokens the whole budget can be consumed by the
reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat
endpoint over raw completion.Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet
linear-attention : full-attention, 27-block ViT, 1 MTP layer).Inherited from the upstream model, which declares the Apache License 2.0. See LICENSE and
NOTICE in this repository. Upstream chain: Qwen/Qwen3.8-27B → this checkpoint's base by DavidAU.
vllm serve DoktorMincs/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W8A16-AutoRound \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --reasoning-parser qwen3 \ --enable-prefix-caching \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'