GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
avyukth
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
avyukth
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
bash
vllm serve avyukth/Qwen3.8-27B-AWQ-INT4 \--max-model-len 8192 \--gpu-memory-utilization 0.93 \--enable-auto-tool-choice --tool-call-parser qwen3_xml
Text-only serving reclaims the vision encoder cache (~5k extra KV tokens on a 24 GB card):
bash
--limit-mm-per-prompt '{"image": 0}'
python
from transformers import AutoModelForImageTextToText, AutoProcessormodel = AutoModelForImageTextToText.from_pretrained("avyukth/Qwen3.8-27B-AWQ-INT4", device_map="auto")processor = AutoProcessor.from_pretrained("avyukth/Qwen3.8-27B-AWQ-INT4")
Measured on an RTX 3090 (24 GB, sm_86), vLLM, max_model_len=8192.
| Benchmark | Result |
|---|---|
| GSM8K (exact match, n = 40, temperature 0) | 39/40 = 97.5% |
| Vision: read rendered text | exact |
| Vision: identify shape / colour / position | correct |
The GSM8K figure is on 40 problems, not the full 1319-item test set. It is enough to show reasoning survived quantisation; it is not a leaderboard number, and no bf16 side-by-side was run, so the exact delta from the base model is unmeasured.
ignore_eos)| Concurrency | Aggregate tok/s | Per request | TTFT p50 |
|---|---|---|---|
| 1 | 41.1 | 41.1 | 0.38 s |
| 4 | 136.4 | 34.1 | 1.28 s |
| 8 | 221.5 | 27.7 | 2.21 s |
| 16 | 216.9 | 13.6 | 3.14 s |
Throughput saturates at concurrency 8; beyond that, aggregate is flat and only latency grows.
linear_attn
(Qwen3_5GatedDeltaNet) layers. AWQ's scaling search must replay a parent
module to collect reference outputs, and Qwen3_5GatedDeltaNet.forward has
the runtime signature (self, *args, **kwargs) — a decorator drops the real
one — so llm-compressor's captured arguments collapse into a single nested
kwargs key and replay fails. Those layers are therefore quantised by
round-to-nearest rather than AWQ. Their MLPs and all full-attention layers
do get proper AWQ scaling. No measurable reasoning cost was found
(GSM8K above), but this is not the same as a full AWQ model.intermediate_size is 4304, which is not
divisible by group_size=128 (4304/128 = 33.625; the only divisor under 256
is 16), so those layers cannot be group-quantised at all. This costs 0.92 GB.
Per-channel quantisation would be the workaround if that matters.Produced with llm-compressor 0.13.0. Two details that are easy to get wrong
on this architecture:
python
from llmcompressor.modifiers.transform.awq import AWQModifier # NOT modifiers.awqfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.modifiers.transform.awq.dynamic_mappings import (build_hybrid_attention_mappings)# The dynamic builder emits layer-index-scoped rules per attention kind.# Drop only the linear-attention rule; the rest keep proper AWQ scaling.maps = [m for m in build_hybrid_attention_mappings(model)if not any("linear_attn" in b for b in m.balance_layers)]recipe = [AWQModifier(mappings=maps, duo_scaling="both"),QuantizationModifier(targets=["Linear"], scheme="W4A16",ignore=["lm_head", "re:.*visual.*", "re:.*mtp.*"]),]
Load the model with the class its config declares
(Qwen3_5ForConditionalGeneration), not AutoModelForCausalLM — the
latter resolves to the text-only class, which silently drops the vision tower
and writes a config that no longer matches the weights.
Save the tokenizer from a fresh instance. HF fast tokenizers persist
truncation state, so saving the tokenizer used for calibration bakes
"truncation": {"max_length": 512} into tokenizer.json, which silently
clips every prompt beyond the calibration length.
Apache 2.0, inherited from the base model. All credit for the model itself goes to the Qwen team; this repository contains only a quantisation of their weights.
| 32 | 211.8 | 6.6 | 13.72 s |
W4A16_ASYMHuggingFaceH4/ultrachat_200k.
Calibration data is general chat, not domain-specific.