GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
sakamakismile
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sakamakismile
Model Tree
huihui-ai/Huihui-Qwen3.8-27B-abliterated
Input Modalities
Output Modalities
Supported Functionality
| concurrency | aggregate t/s |
|---|---|
| 1 | 77.3 |
| 4 | 204.8 |
| 8 | 380.9 |
Single-stream prefill: 3,590 tok/s on an 8k prompt (prefix cache disabled, mean of 3). GPU KV cache at this config: 613,655 tokens.
Capability after abliteration + 4-bit: an 8-probe set (Japanese fluency, English code, arithmetic, instruction-following, logic puzzle, domain knowledge, defensive-security explanation, and a trading backtest that punishes look-ahead bias) scores 8/8 — identical to the unmodified Qwen3.8-27B quantized with the same recipe, at 87.2 t/s vs 85.7 t/s. No measurable degradation.
bash
vllm serve sakamakismile/Huihui-Qwen3.8-27B-abliterated-NVFP4 \--trust-remote-code --tensor-parallel-size 4 \--max-model-len 32768 --kv-cache-dtype fp8 --reasoning-parser qwen3 \--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
On boards without P2P add NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce.
Send reasoning_effort: "medium" for long-form work. This is the one that will bite you. reasoning_effort defaults to xhigh, and at that setting the <think> phase sometimes never terminates: it grows past 19,000 characters, degenerates into repeating a single line, and the token budget is gone before any answer is emitted. On a 9-case gate (French/English long-form, temperature 0 and 0.7) this model fails 1 of 9 at xhigh and passes 9 of 9 at medium, where thinking stays near 1k characters. The unmodified Qwen3.8-27B passes the same gate at both settings, so the abliterated fine-tune is more prone to it — but medium is the right default either way.
json
"chat_template_kwargs": {"reasoning_effort": "medium"}
Worth knowing what these modes actually are: they are not three levels of capability, they are three system prompts. Reading the chat template — xhigh injects "think carefully through the task, validate key assumptions, consider plausible alternatives…", low injects "keep your thinking brief and focused, moving directly to the conclusion", and medium injects nothing at all — it is simply the model with no deliberation instruction. That matches what I measured: runs a few hundred characters at , ~1k at , 5–19k at , while answer quality was indistinguishable across all three on short verifiable tasks (96 runs, no errors at any setting). The risk at is not worse reasoning — it is deliberation outliving the token budget.
llm-compressor NVFP4 (W4A4, group 16), targets: [Linear], ignore: [lm_head, re:.*visual.*, re:.*conv1d.*, re:.*mtp.*], 32 calibration samples × 8192 tok from neuralmagic/calibration. MTP tensors grafted back in bf16 after saving and appended to quantization_config.ignore.
🙏 @huihui-ai for the model, Qwen team for the base, vLLM & llm-compressor teams for the tooling.
<think>lowmediumxhighxhighThe 15 mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance and slower than no MTP.
W4A16 (NVFP4A16) does not serve on this architecture in vLLM 0.22 (gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.
Long single-file code generation drops a closing paren roughly 1–2 times in 14 regardless of sampling temperature (measured with a JS parser on the base model). Put a syntax check in the loop rather than tuning temperature.
Text-only serving shown above; the vision tower ships in bf16 but multimodal serving was not benchmarked here.