GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
groxaxo
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
groxaxo
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Qwen3.8-27B-GPTQ-Pro-4bit-g64-calib128 is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by groxaxo.
It is intended for open-source evaluation, reproducible experimentation, and compatible local or
hosted inference workflows. The wording below is deliberately limited to what can be verified
from this repository's metadata and artifacts.
| Field | Details |
|---|---|
| Format | GPTQ |
| Source / base | Qwen/Qwen3.8-27B |
| Intended task | image-text-to-text |
| License | apache-2.0 |
*.safetensors (5 files)config.jsongeneration_config.jsontokenizer.jsontokenizer_config.jsonprocessor_config.jsonchat_template.jinjaquantize_config.jsonbash
vllm serve groxaxo/Qwen3.8-27B-GPTQ-Pro-4bit-g64-calib128 \--tensor-parallel-size 2 \--disable-custom-all-reduce \--dtype bfloat16 --language-model-only \--max-model-len 260000 --gpu-memory-utilization 0.92 --max-num-seqs 2 \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.
Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.
Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.
INT4 GPTQ-Pro quantization of the official Qwen/Qwen3.8-27B, with the MTP (multi-token prediction) draft head preserved and verified working in vLLM speculative decoding.
Sister build (abliterated source): groxaxo/Huihui-Qwen3.8-27B-abliterated-GPTQ-Pro-4bit-g64-calib128 — same recipe, same loss profile.
| Parameter | Value |
|---|---|
| Tool | GPTQModel 6.1.0-dev (GPTQ-Pro) |
| Bits / symmetry | 4-bit, symmetric |
| Group size | 64 (desc_act=False) |
| Calibration | text mode, 128 samples |
| Preset | quality (true-sequential, activation-weighted MSE, group-aware) |
| Coverage | 400/400 linear modules across all 64 layers (48 linear-attention GDN + 16 full-attention) |
| MTP head | preserved unquantized ( tensors kept; ) |
vLLM 0.24.0, TP=2: Qwen3_5MTP drafter resolves and loads; drafts accepted (25/26 on smoke test); coherent generations. Single RTX 3090 cannot fit INT4 27B + MTP drafter — use TP≥2.
⚠ If you serve this on multi-GPU RTX 3090 (PCIe, no NVLink) and get garbage output (e.g. !!!!...), pass --disable-custom-all-reduce. vLLM's custom allreduce kernel silently corrupts tensor-parallel output on this hardware class. Verified empirically: identical checkpoint, identical prompts — broken without the flag, byte-identical correct output with it. This affects any TP-served model on such boxes, not just this quant.
Speed reference from the sister Huihui build (identical recipe/arch, 2× RTX 3090 TP=2, temp 0.7, 256 tok): 78–87 tok/s single-stream with MTP draft-3 vs ~71 no-MTP; draft acceptance ~59–77% depending on temperature.
bash
vllm serve groxaxo/Qwen3.8-27B-GPTQ-Pro-4bit-g64-calib128 \--tensor-parallel-size 2 \--disable-custom-all-reduce \--dtype bfloat16 --language-model-only \--max-model-len 260000 --gpu-memory-utilization 0.92 --max-num-seqs 2 \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
min_p and logit_bias do not work with speculative decoding (vLLM limitation).
mtp.*mtp_num_hidden_layers=1| Per-module loss | mean 8.2e-5, max 5.0e-4 (quant_log.csv included) |