GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
PixelML
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
PixelML
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
The source is the F16.gguf from
prism-ml/Ternary-Bonsai-2-27B-gguf
@ 6ed5e12bf84b7a63069882c91dd9e9218647d17b, whose weights are stored in a
Hadamard-rotated, v-grouped, delta-normed layout for llama.cpp's custom
kernels. Conversion inverts that layout deterministically:
W_hf = W_stored @ blockdiag(H_sylv·diag(s_b)/√1024)
per 1024-block of the input dim; signs from the file's own
prism.hadamard.* manifest.attn_qkv (v segment), attn_gate, ssm_alpha,
ssm_beta, ssm_a, ssm_dt, ssm_conv1d (v part) and ssm_out
(columns), mirroring the Bonsai-demo runtime.py reorder().attn_norm, post_attention_norm, output_norm,
attn_q_norm, attn_k_norm are stored as g − 1 in the GGUF; restored
to g (the runtime adds the 1 inside its fused RMSNorm+rotate kernel).model.visual.* and mtp.* are absent from the ternary
GGUF and are copied verbatim (bf16) from Qwen/Qwen3.8-27B. The MTP module
is what DFlash2 speculative decoding drafts with.Verified: NLL 1.169 on a held-out sentence (base Qwen3.8-27B: 1.518 through the same harness), coherent greedy generation, structural index match.
GPTQ W4A16 via llmcompressor 0.13 (pack-quantized compressed-tensors):
256 samples × 1024 tokens of open_platypus, symmetric group-128. Then the
syv-ai single-user prepare chain: int8 (group-128) lm_head, embed_tokens and
MTP module in place, plus the 40960-token draft head
(mtp.draft_lm_head, 213 MB) for DFlash2 speculative decoding. Per-layer
GPTQ round-trip relative errors 0.6–0.8%. The vision tower stays bf16.
vLLM ≥ 0.28.0 (DFlash2 is native there; no patch needed for the drafter):
bash
vllm serve /path/to/Bonsai-2-27B-W4A16 \--speculative-config '{"method":"dflash2","method_config":{"draft_tokens":7}}'
or the syv-ai single-user launcher (SPEC=dflash2 CTX=fast), which also
enables lookup-augmented drafting.
Measured on 1× NVIDIA CMP 170HX (SM80, 180 W): 155.7 tok/s single-stream decode (256-token cohort), 251.6 tok/s on a 900-token cohort (adaptive draft depth), prefill 1876 tok/s @ ~6.6k prompt, DFlash2 acceptance 4.17 tokens/draft (59.6% per position) with the base-calibrated drafter at zero recalibration. Full receipts, protocol and the ternary-llama.cpp comparison lane: PixelML/club-170hx PR #49 and #47.
mtp.* and model.visual.* are required at serve time for DFlash2 and
image input respectively — do not strip them.