GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
groxaxo
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
groxaxo
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| Parameter | Value |
|---|---|
| Tool | GPTQModel 6.1.0-dev (GPTQ-Pro) |
| Bits / symmetry | 4-bit, symmetric |
| Group size | 64 (desc_act=False) |
| Calibration | text mode, 128 samples |
| Preset | quality (true-sequential, activation-weighted MSE, group-aware) |
| Coverage | 400/400 linear modules across all 64 layers (48 linear-attention GDN + 16 full-attention) |
| MTP head | preserved unquantized (mtp.* tensors kept; mtp_num_hidden_layers=1) |
| Offload | disk offload during calibration |
Per-module quantization loss (quant_log.csv, included): mean 8.2e-5, max 4.9e-4. Damping 1.49–8.94 (mean 3.33).
vLLM 0.24.0, max_model_len=4096, greedy, 256 generated tokens, prompt about MTP itself. Greedy decoding → measured draft acceptance at position 0: 100%.
| Config | Single-stream tok/s | Aggregate tok/s (8 concurrent) |
|---|---|---|
| No MTP | 71.5 | 390.6 / 430.9 |
MTP (num_speculative_tokens=1) | 105.5 (+47.6%) | 426.4 / 555.4 (+9% / +28.9%) |
Single-GPU note: on one RTX 3090 (23.56 GiB) the MTP drafter does not fit alongside this checkpoint — target ~21.5 GiB + drafter ~2.4 GiB exceeds the card. Single-GPU no-MTP baseline for reference: 42.9 tok/s single-stream, ~273 tok/s at 8-way concurrency.
bash
vllm serve groxaxo/Huihui-Qwen3.8-27B-abliterated-GPTQ-Pro-4bit-g64-calib128 \--tensor-parallel-size 2 \--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
Without MTP, serve like any GPTQ checkpoint (single 24 GB GPU works with --max-model-len 4096 --gpu-memory-utilization 0.92 --max-num-seqs 32).
Note: min_p and logit_bias do not work with speculative decoding (vLLM warning).