GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
alpha2beta
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
alpha2beta
Model Tree
huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated
Input Modalities
Output Modalities
Supported Functionality
qweight int32 + scales/qzeros): all 64 dense
mlp.{gate,up,down}_proj, all 16 full-attention self_attn.{q,k,v,o}_proj, and all
48 linear_attn.{in_proj_qkv,in_proj_z,out_proj}. in_proj_a/in_proj_b are
skipped, as in the reference recipe.lm_head, every norm, A_log/conv1d/dt_bias, the full
27-block visual tower, and the MTP draft block (mtp.*, 15 tensors) — held out of
quantization with dynamic={"-:.*mtp.*": {}} and stored unquantized inside shard 5,
so speculative decoding needs no separate draft file.text_config.mtp_num_hidden_layers is 1 (as shipped upstream).2,399 tensors / 5 shards / 18.24 GiB · SHA256SUMS included.
Structurally identical to the reference checkpoint (same 2,399 tensors, same per-dtype
element counts: F16 3,219,819,760 · I32 3,067,527,168 · BF16 424,699,392), which is
what makes the two directly comparable below.
| field | value |
|---|---|
| method / format | gptq |
| bits / group_size / sym | 4 / 128 / True |
desc_act | False |
lm_head | False |
true_sequential / act_group_aware |
The calibration is the one deliberate deviation from the classic GPTQ budget (128 × 2048 = 262,144): on the 45 GB-RAM host used here the larger pile terminated the job every time (GPU engine reset, or a host OOM-kill), and 64 × 1024 completed cleanly. The reference checkpoint does not record how it was calibrated, so token-set parity was never possible anyway. If you re-quantize, note that a different calibration changes the Hessians and therefore the weights — it is a different artifact, not a re-run.
Measured on Intel Arc Pro B60 (24 GB, Battlemage), i5-12400F host, GPU power capped
at 180 W, vLLM 0.30.0+xpu, --dtype float16 --kv-cache-dtype fp8 --language-model-only,
MTP speculative decoding k=4, prompt 512 / generation 128 tokens:
| metric | value |
|---|---|
| decode | 40.183 tok/s |
| TTFT | 0.460 s |
| mean accepted tokens / step | 3.00 |
| vs. reference checkpoint, same session | −0.14 % (40.183 vs 40.239) |
MTP is worth having: with the draft disabled the same build measured 7.987 tok/s
against 16.620 tok/s at k=4 — 2.08× — on an MXFP4 leg of the same experiment.
Note the acceptance ratio is an upper bound on the gain, never the gain itself: the
speculative step costs 1.361× more per step here (125.2 → 170.5 ms).
Long context (single stream, 12,288-token ceiling): prefill stays linear at
~1,372 tok/s through 4,096 tokens (marginal 1,268 tok/s against the fitted line,
R² = 0.9986), while decode degrades gently — 40.200 tok/s @512 → 37.315 tok/s @12,000
(−7.2 %), TTFT 0.463 → 9.539 s. With --kv-cache-dtype fp8 the KV cache is 1.82 GiB,
i.e. ~17,630 tokens of capacity at these settings.
15/16 (93.8 %) on a 16-prompt task suite spanning math, code (executed), instruction
constraints and fact/trap questions. Treat that as ±1 item: one item
(format_no_letter_e) converges on some server instances and loops to the token cap on
others at temperature=0, deterministically within an instance but not across them, so a
single-item delta between builds is not signal.
Two harness traps this checkpoint sets, both worth knowing before you benchmark it:
--reasoning-parser qwen3. The engine emits </think> (added token
248069) but the detokenizer suppresses it from content and leaves reasoning
null, so without the parser the whole reasoning trace arrives concatenated in front of
the answer with no delimiter. vLLM's own 16-prompt quality harness scores 8/16
that way versus 15/16 with the flag — a plausible-looking 50 % scored on the
model's private musings.bash
vllm serve Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GPTQ-Int4-sym-G128-MTP-BF16 \ --served-model-name coder390-gptq \ --quantization gptq --dtype float16 --kv-cache-dtype fp8 \ --max-model-len 12288 --max-num-seqs 1 --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.93 --language-model-only \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_xml \ --speculative-config '{"method":"mtp","num_speculative_tokens":4}'Notes:
--language-model-only skips the visual tower, which is what the benchmarks above
used. The tower is intact and BF16 (333 tensors) if you need vision.mtp.* block in shard 5, activated
by B70_MTP_BF16_DRAFT=1 on this stack and by the compressed-tensors ignore list.text
ValueError: ... KV cache is needed, which is larger than the available KV cache memory (0.18 GiB)torch.compile's transient pool eats the KV budget; the
retry succeeds against a warm compile cache. Run the server under
--restart unless-stopped (or restart the same container once) and expect ~220 s to
healthy rather than treating the first failure as fatal.alpha2beta/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-MXFP4
is the same base model quantized to MXFP4 (compressed-tensors mxfp4-pack-quantized,
W4A4). Same tensors held in BF16, same folder layout, same 5-shard/16-shard family
naming. On the same host MXFP4 is markedly slower than GPTQ at batch 1 — W4A4 dynamic
activation quantization and a non-fused FP4 GEMM, multiplied by eager execution — which
is why this GPTQ build exists as the throughput option.
Built with gptqmodel 7.6.0 on
vLLM 0.30.0+xpu / torch 2.13.0+xpu / transformers 5.16.1, driving the Arc Pro B60 as the
quantization accelerator, with offload_to_disk=True (meta-device shell + module-by-
module streaming) because the 52 GB BF16 source does not fit in the 45 GB host RAM.
Base weights: huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated
(BF16, Apache-2.0). Reference schema: SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16.
Full build record, attempt log, benchmarks and the operating guide:
alpha2beta/vllm.xpu @ b60 —
see REPRODUCIBILITY.md (§6, §7 deviation 8, §8 known issues 1/11/12/13) and
GPTQ_quantization.md.
True / True |
pack_dtype / pack_impl | int32 / cpu |
damp_percent / damp_auto_increment | 0.05 / 0.01 |
fallback | Fallback(rtn, "0.5%") |
hessian.staging_dtype | float32 |
dynamic | {"-:.*mtp.*": {}} — keeps the MTP draft in BF16 |
| calibration | mit-han-lab/pile-val-backup, split=validation, streaming (10,000-row shuffle buffer), 64 chunks × 1024 tokens = 65,536 tokens |