GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
randongg
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
randongg
Model Tree
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
Input Modalities
Output Modalities
Supported Functionality
This is a PrismAura quantized build of the DavidAU Qwen3.8-27B-TURBO-Fable-Cold-Fusion Heretic-Uncensored variant — a Cold-Fusion fine-tune of Qwen3.8-27B that has been decensored with the Heretic workflow (ablation of refusal/safety directions from the weight tensors).
This build changes none of that behavior — it is a numerical re-encoding of the same weights into a mixed-precision compressed-tensors layout for efficient vLLM serving. It does not add, remove, or retrain anything at the weight level beyond quantization.
⚠️ This is a decensored, general-purpose model whose upstream tuning intentionally reduces refusal behavior, including for explicit, violent, and NSFW content. It carries no built-in content moderation. Deploy it behind your own usage policy and access controls if that matters for your use case.
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAUQwen/Qwen3.8-27B — 27B dense, causal LM with vision encoder, 64 hidden layers, 5,120 hidden size, 262,144-token native context{NVFP4, FP8_E4M3, BF16} format menu)compressed-tensors, mixed precisionAURA was run across a sweep of target bit budgets against the source model, tracking the achieved bits/parameter and the resulting KL–Fisher loss estimate at each point, along with how many "body" (non-visual, non-MTP) Linears landed in each format:
| target bpp | achieved bpp | Δloss | NVFP4 | FP8_E4M3 | BF16 |
|---|---|---|---|---|---|
| 4.500 | 4.500 | 1.0184e+02 | 303 | 1 | 0 |
| 4.600 | 4.600 | 8.3320e+01 | 276 | 22 | 6 |
| 4.700 | 4.700 | 7.7203e+01 |
This build uses the 5.500 → 5.501 operating point (bolded row above), identified by the solver as the raw-linear knee of the Rate-Distortion curve. The RD curve is log-linear across the sweep (log10(Δloss) ≈ −0.267·bpp + 3.162, R² = 0.9925), so this point is chosen by explicit bit-budget target rather than by a sharp intrinsic knee.
markdown
[alloc] target_bits=5.5: achieved_bits=5.501, Δloss=5.011e+01[alloc] --visual-format=BF16: assigned 110 visual Linears uniformly (source=/model)[alloc] --mtp-format=BF16: assigned 8 MTP Linears uniformly[alloc] target=5.5 exact_assignment_payload_bpp=5.501NVFP4: 272 layersFP8_E4M3: 176 layersBF16: 166 layers
The body row above (168 / 112 / 24) reflects only the AURA-priced transformer Linears at the 5.5 target; the final totals (272 / 176 / 166) represent the entire network after the vision tower's 110 Linears and the 8 MTP Linears are added in as uniform BF16 passthrough alongside the rest of the quantizable Linears. The format menu includes an FP8 middle rung specifically so the allocator can spend extra bits on Linears that need it instead of jumping straight from NVFP4 to BF16.
Vanilla vLLM, no plugin, no custom kernels. This is the full command, including the shipped multi-token-prediction (MTP) head for speculative decoding:
bash
vllm serve randongg/Qwen3.8-27B-ColdFusion-Heretic-PrismAura-5.5bit \--host 0.0.0.0 --port 8000 \--quantization compressed-tensors \--max-model-len 32768 \--gpu-memory-utilization 0.85 \--kv-cache-dtype fp8 \--enable-prefix-caching \--reasoning-parser qwen3 \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
--quantization compressed-tensors is required to load the mixed
NVFP4/FP8/BF16 format; NVFP4 execution requires an NVIDIA Blackwell GPU. On
non-Blackwell hardware, vLLM falls back to dequantized execution.--speculative-config turns on the MTP head (mtp.* tensors, BF16) with
two draft tokens per step. Drop the line to serve without speculation. Depending
on your vLLM release, the method name may be "qwen3_next_mtp".--limit-mm-per-prompt '{"image":4}' (or your own
limit); the visual tower ships in BF16.--language-model-only to skip loading the vision tower on pure-text
workloads and free memory for KV cache.Memory. At ~22 GiB of weights this wants a 32 GB card (5090-class) or larger. Verified on GB10 / DGX Spark (Blackwell, 128 GB unified) under vLLM 0.27 with the CUTLASS NVFP4 and FP8 paths. A working DGX Spark invocation:
bash
vllm serve randongg/Qwen3.8-27B-ColdFusion-Heretic-PrismAura-5.5bit \--quantization compressed-tensors \--max-model-len 2048 \--gpu-memory-utilization 0.70 \--kv-cache-dtype fp8
On the DGX Spark specifically, if you see resets or NVRM: NV_ERR_NO_MEMORY
errors under load, you may want to lower --gpu-memory-utilization (the
recipe ships 0.70) and/or cap GPU clocks (sudo nvidia-smi -lgc 0-2200,
host-side) — see the GB10 notes in the sparkrun section. Both are
situation-dependent, not requirements.
Measured with a single combined run of tool-eval-bench v2.5.1 — bench --perf (llama-benchy throughput sweep + 69 agentic tool-call scenarios) with --seed 42 for reproducibility. Settings: --quantization compressed-tensors, fp8 KV cache, MTP speculative head (3 tokens), GB10 (121 GiB unified, sm_121a), served via vLLM.
Decode throughput (tg t/s):
| context depth | conc 1 | conc 2 | conc 4 |
|---|---|---|---|
| 0 | 17.5 | 32.8 | 59.1 |
| 8192 | 19.4 | 26.7 | 22.2 |
| 32768 | 18.0 | 9.0 | 6.1 |
Agentic tool-call correctness from the same seeded run (69 scenarios):
Full report (throughput tables + per-scenario results): see seed42-combined.md in bench-notes/. This is a tool-calling benchmark, not a standard instruction suite; see Limitations for what is not measured (perplexity, KL, GSM8K, MMLU).
The recipe below is the exact one used for every number in this card. It pins the eugr vLLM nightly that ships MTP/spec-decoding support and carries a conservative gpu_memory_utilization for unified-memory DGX Spark-class boxes (see GB10 notes below). It uses the model's default chat template (no custom mod). To use it, save the block as e.g. aura-mtp.yaml in a recipes dir and run sparkrun run aura-mtp.yaml --rootful.
yaml
# PrismAura-5.5bit Qwen3.8-27B + MTP speculative decoding (sparkrun recipe v2)## MTP (Multi-Token Prediction) uses the model's built-in MTP head# (mtp_num_hidden_layers=1) — no external drafter needed.## Run: sparkrun run aura-mtp.yaml --rootfulrecipe_version: "2"model: randongg/Qwen3.8-27B-ColdFusion-Heretic-PrismAura-5.5bitruntime: vllmcontainer: ghcr.io/spark-arena/dgx-vllm-eugr-nightly-tf5:20260822metadata:description: PrismAura-5.5bit Qwen3.8-27B with MTP speculative decodingdefaults:port: 30000host: 0.0.0.0name: qwentensor_parallel: 1pipeline_parallel: 1# conservative for GB10 unified memory (0.80 saw NVRM OOM + firmware reset under load)gpu_memory_utilization: 0.70max_model_len: 262144max_num_seqs: 64max_num_batched_tokens: 8192load_format: fastsafetensorskv_cache_dtype: fp8attention_backend: flashinfertool_call_parser: qwen3_xmlreasoning_parser: qwen3# MTP: built-in multi-token prediction, 3 speculative tokensspeculative_config: '{"method": "mtp", "num_speculative_tokens": 3}'generation_config: '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'env:VLLM_MARLIN_USE_ATOMIC_ADD: "1"CUTE_DSL_ARCH: "sm_121a"OMP_NUM_THREADS: "4"MAX_JOBS: "4"NUMBA_DISABLE_JIT: "1"command: |vllm serve {model} \--host {host} \--port {port} \--served-model-name {name} \--max-model-len {max_model_len} \--quantization compressed-tensors \--trust-remote-code \--kv-cache-dtype {kv_cache_dtype} \--load-format {load_format} \--attention-backend {attention_backend} \--speculative-config '{speculative_config}' \--gpu-memory-utilization {gpu_memory_utilization} \--max-num-seqs {max_num_seqs} \--max-num-batched-tokens {max_num_batched_tokens} \--enable-auto-tool-choice \--tool-call-parser {tool_call_parser} \--reasoning-parser {reasoning_parser} \--enable-chunked-prefill \--enable-prefix-caching \-tp {tensor_parallel} \-pp {pipeline_parallel} \--override-generation-config '{generation_config}'
Recommended speculative method: MTP. This model ships a built-in multi-token-prediction head (mtp.* tensors, BF16); vLLM's {"method": "mtp"} uses it directly with no external drafter. It roughly doubled DFlash2's decode throughput in our measurements.
Why we don't ship a DFlash2 recipe. We also tried the DFlash2 block-diffusion drafter (z-lab/Qwen3.8-27B-DFlash2, 8 draft tokens). That drafter was trained against the standard Qwen3.8-27B weights, but this is a Heretic variant whose refusal/safety directions have been ablated — the model's output distribution drifts far enough from what the drafter expects that its acceptance rate collapses. It was consistently slow, so we're not shipping it.
GB10 notes (situational, not requirements). On our test machine — an Asus Ascent GB10 (DGX Spark class, 121 GiB unified memory, power-capped SoC) — heavy load at the default gpu_memory_utilization: 0.8 caused a hard reset: unified memory squeezed to ~10%, the NVIDIA driver began failing allocations (NVRM: NV_ERR_NO_MEMORY), then a firmware-level reset with no kernel panic. That's why the recipe above ships with gpu_memory_utilization: 0.70.
If you're on a similar box and see resets or NVRM errors under load, you may find these helpful — they are situation-dependent (workload, other software, thermals), not requirements to run this model:
gpu_memory_utilization (recipe uses 0.70) to leave unified-memory headroom, and/orsudo nvidia-smi -lgc 0-2200 (not persistent — re-run after each reboot) to keep power draw in the safe band.With those knobs in place this machine served and benchmarked stably for 1+ hour under full load.
A modern LLM has thousands of weight matrices, each of which could be stored at any of several hardware precision formats. AURA splits quantization into two questions and answers the hard one by measurement rather than heuristic:
(Linear, format) pair using a KL–Fisher quadratic — the second-order effect of that Linear's quantization error on the model's full output distribution, measured with stochastic probes through the real model — then solves a multiple-choice knapsack over the total bit budget. The result is a heterogeneous, per-Linear precision assignment that no single-format quantization method can produce structurally.The total export size is ~23.6 GB across 20 safetensors shards. The underlying architecture is the full 27B-parameter Qwen3.8-27B model, compressed to ~5.5 bits per quantizable parameter on the body.
shipcard.json shipped here records the build identity (source model,
layer config hash, assignment hash, bpp, format histogram) but the
native_export.eager, native_export.graph, ship_gate, gold.kl and
gold.ppl slots are UNFILLED — the post-export KL-vs-BF16 and
WikiText-2-PPL measurements were not run for this export. No perplexity,
KL, or task-accuracy claim is made here. The model was smoke-tested by
serving it with vLLM and confirming coherent completion and chat output.lm_head, embed_tokens, the visual tower and MTP heads are held in BF16 and
are excluded from the bit-per-parameter figure, following this project's
convention. Compare against other checkpoints on disk bytes, not the bpp
label.ultrachat_200k
(8 samples × 1024 tokens, text-only modality); the visual tower was assigned
uniformly in BF16 and not run through the KL–Fisher allocator.Qwen/Qwen3.8-27B)DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU)RobTand/prismaquant)Inherits the license of the base model chain (apache-2.0). You are responsible for complying with any additional usage terms from upstream contributors and for your own deployment's content policy, given the decensored nature of the source model.
| 248 |
| 36 |
| 20 |
| 4.750 | 4.750 | 7.4856e+01 | 242 | 42 | 20 |
| 4.850 | 4.850 | 7.0635e+01 | 236 | 49 | 19 |
| 5.000 | 5.001 | 6.4907e+01 | 212 | 68 | 24 |
| 5.250 | 5.251 | 5.6919e+01 | 184 | 94 | 26 |
| 5.500 | 5.501 | 5.0110e+01 | 168 | 112 | 24 |
| 6.000 | 6.001 | 3.8790e+01 | 127 | 146 | 31 |
| 7.000 | 7.001 | 2.1618e+01 | 65 | 207 | 32 |
| 8.250 | 8.250 | 8.4307e+00 | 3 | 255 | 46 |