GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
devan-carlin
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
devan-carlin
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Runs a 27B dense model on 4x Intel Arc Pro B70 (32 GB each) with vLLM tensor parallelism. INT4 fits comfortably and runs faster than BF16 on this hardware.
bash
vllm serve devan-carlin/Qwen3.8-27B-int4-AutoRound \--quantization auto-round \--tensor-parallel-size 4 \--max-model-len 262144 \--kv-cache-dtype fp8 \--enable-prefix-caching \--gpu-memory-utilization 0.85 \--trust-remote-code
Note: loading mixed symmetric/asymmetric INT4 checkpoints on the XPU/ARK path requires the qzeros guard fix (see the vLLM PR referenced below). Without it, symmetric layers crash with a copy_() shape mismatch during weight loading.
| Model | Gen speed | Notes |
|---|---|---|
| INT4 (this repo) | 47.8 tok/s | 18 GB |
| Qwen3.8-27B BF16 | 30.2 tok/s | ~52 GB |
max_tokens cap earlier — raise the cap if neededmtp.fc kept in BF16; MTP layers quantized to INT4 (model stays MTP-capable)Apache-2.0 (same as the base model).
Quantized and benchmarked by devan-carlin on 4x Intel Arc Pro B70 (XPU).