GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
devan-carlin
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
devan-carlin
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Runs a 27B dense model on 4x Intel Arc Pro B70 (32 GB each) with vLLM tensor parallelism. INT4 fits comfortably and runs faster than BF16 on this hardware.
bash
vllm serve devan-carlin/Qwen3.8-27B-int4-AutoRound \--quantization auto-round \--tensor-parallel-size 4 \--max-model-len 262144 \--kv-cache-dtype fp8 \--enable-prefix-caching \--gpu-memory-utilization 0.85 \--trust-remote-code
Note: loading symmetric INT4 checkpoints with empty-qzeros layers on the XPU/ARK path requires the qzeros guard fix (vllm PR #52428). Without it, symmetric layers crash with a copy_() shape mismatch during weight loading.
| Model | Gen speed | Notes |
|---|---|---|
| INT4 (this repo) | 47.8 tok/s | 18 GB |
| Qwen3.8-27B BF16 | 30.2 tok/s | ~52 GB |
max_tokens cap earlier — raise the cap if neededmodel.language_model.layers is quantized to INT4; embeddings, lm_head, the MTP head, and the vision tower stay in their original precision (model stays MTP-capable)linear_attn.in_proj_a/b are kept 16-bitApache-2.0 (same as the base model).
model.visual.*) plus processor configs, and the vLLM Qwen3_5ForConditionalGeneration port loads it. Send images as image_url content parts (base64 or URL); text-only prompts work unchanged.Quantized and benchmarked by devan-carlin on 4x Intel Arc Pro B70 (XPU).