GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sakamakismile
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sakamakismile
Model Tree
huihui-ai/Huihui-Qwen3.8-27B-abliterated
Input Modalities
Output Modalities
Supported Functionality
Throughput (TP=4 on RTX PRO 2000 Blackwell ×4, 32k ctx, KV fp8, MTP n=3, 512 tok fixed, reasoning_effort: low):
| concurrency | aggregate t/s | MTP acceptance |
|---|---|---|
| 1 | 66.6 | 44.6 % |
| 4 | 191.3 | 46.7 % |
| 8 | 367.9 | 47.3 % |
Single-stream prefill: 3,250 tok/s on a 14.5k prompt (prefix cache cold, mean of 3). GPU KV cache at this config: 556,063 tokens (16.97× concurrency at 32k/request).
Against the previous revision, same hardware, same day, same conditions (TP=2, 32k, MTP n=3): 41.0 t/s vs 41.1 t/s, MTP acceptance 47.1 % vs 49.8 %. The narrower ablation costs nothing in speed.
Capability: an 8-probe set (arithmetic, multi-step probability, code generation, strict-format instruction following, Japanese prose, translation, an uncensored-behaviour check, and an epistemic-humility check) — both revisions pass every mechanically-checkable face, and the prose faces read equally well. On this probe set the "retains more of the original model's performance" claim does not show up as a measurable gain. It does not show up as a loss either. Where it does show up is the xhigh gate below.
bash
vllm serve sakamakismile/Huihui-Qwen3.8-27B-abliterated-NVFP4 \--trust-remote-code --tensor-parallel-size 4 \--max-model-len 32768 --kv-cache-dtype fp8 --reasoning-parser qwen3 \--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
On boards without P2P add NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce.
The xhigh thinking-runaway is fixed in this revision. On the previous revision, reasoning_effort at xhigh would sometimes never terminate the <think> phase: it grew past 19,000 characters, degenerated into repeating a single line, and the token budget was gone before any answer was emitted — 1 failure in a 9-case gate (French/English long-form, temperature 0 and 0.7). The same gate on this revision passes 9 of 9, with thinking between 1.5k and 8k characters and every run finishing on stop. medium remains a sensible default for long-form work, but it is no longer a workaround for a defect.
Worth knowing what these modes actually are: they are not three levels of capability, they are three system prompts. Reading the chat template — xhigh injects "think carefully through the task, validate key assumptions, consider plausible alternatives…", low injects "keep your thinking brief and focused, moving directly to the conclusion", and medium injects nothing at all — it is simply the model with no deliberation instruction.
json
"chat_template_kwargs": {"reasoning_effort": "medium"}
llm-compressor NVFP4 (W4A4, group 16), targets: [Linear], ignore: [lm_head, re:.*visual.*, re:.*conv1d.*, re:.*mtp.*], 32 calibration samples × 8192 tok from neuralmagic/calibration. MTP tensors grafted back in bf16 after saving and appended to quantization_config.ignore.
🙏 @huihui-ai for the model, Qwen team for the base, vLLM & llm-compressor teams for the tooling.
The 15 mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance and slower than no MTP.
W4A16 (NVFP4A16) does not serve on this architecture in vLLM 0.22 (gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.
Long single-file code generation drops a closing paren roughly 1–2 times in 14 regardless of sampling temperature (measured with a JS parser on the base model). Put a syntax check in the loop rather than tuning temperature.
Text-only serving shown above; the vision tower ships in bf16 but multimodal serving was not benchmarked here.