GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
XReyRobert
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
XReyRobert
Model Tree
dbirks/Qwen3.8-27B-W4A16-AutoRound
Input Modalities
Output Modalities
Supported Functionality
Qwen/Qwen3.8-27Bdbirks/Qwen3.8-27B-W4A16-AutoRound1f05c441c4e64ae0549de44fa9ea5a6d43610314syv-ai/qwen38-27b-rtx3090e681a8f6b190cca1f86d58b8b333b64d014994d7pack-quantizedThe exact transformation scripts and vLLM patch used are included in this repository.
The INT8 token embedding table requires a small vLLM patch. Stock vLLM 0.27.1
contains the dequantization kernel but does not pass the quantization config to
Qwen3.5/Qwen3.8's VocabParallelEmbedding constructors.
Apply the included patch to the installed vLLM package:
bash
patch -p1 -d /path/to/site-packages/vllm \< vllm-qwen3_5-embed-quant.patch
The patch was written and verified against vLLM 0.27.1. Revalidate it after
upgrading vLLM. Without this patch, the checkpoint is not expected to load
correctly.
The following profile mirrors the batch-oriented syv-ai recipe:
bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \VLLM_USE_FLASHINFER_SAMPLER=0 \vllm serve XReyRobert/Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings \--served-model-name qwen3.8-27b-autoround-int8-head-embeddings \--language-model-only \--kv-cache-dtype fp8 \--max-model-len 150000 \--max-num-seqs 64 \--max-num-batched-tokens 2048 \--gpu-memory-utilization 0.972 \--async-scheduling \--compilation-config '{"max_cudagraph_capture_size":64,"custom_ops":["+rms_norm","+silu_and_mul"]}' \--trust-remote-code
--language-model-only avoids loading the vision tower. The published syv-ai
measurements use an RTX 3090 and describe 150k as the conservative deployment
context; the reported single-request ceiling after requantization is around
195k. Actual capacity depends on the runtime version, CUDA allocator state,
concurrency and prompt shape.
For single-user speculative decoding, use the MTP profile documented in the source recipe and ensure the second hunk of the included patch is applied.
The source model recommends:
0.7, top-p 0.81.0, top-p 0.95Starting from the pinned source revision:
bash
python quant_lm_head.py /path/to/checkpointpython quant_embed.py /path/to/checkpoint
The scripts rewrite the relevant safetensors shards, update
model.safetensors.index.json, and add dedicated compressed-tensors config
groups for lm_head and embed_tokens.
lm_head relative round-trip error: 0.0064embed_tokens relative round-trip error: 0.0056No accuracy benchmark has been rerun specifically for this published artifact. Refer to the source AutoRound model card for body-quantization evaluations and to the syv-ai repository for the INT8 head/embedding serving benchmarks.
dbirks for the W4A16 AutoRound checkpointsyv-ai for the INT8 head/embedding transformation and vLLM serving recipe