GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
TheUnderscore
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
TheUnderscore
Model Tree
aifeifei798/Qwen3.8-Queen-27B
Input Modalities
Output Modalities
Supported Functionality
W4A16_ASYM — 4-bit asymmetric per-group quantization (group size 128) of all Linear weights, stored in the compressed-tensors pack-quantized format (weight_packed / weight_scale / weight_zero_point / weight_shape), which LMDeploy turbomind auto-detects and loads natively (including the MTP heads and vision tower, which stay BF16).compressed_tensors.offload.load_offloaded_model), so the full-precision source fits on a 2×16 GB VRAM setup.AWQModifier with the layer-scoped hybrid-attention mappings from build_hybrid_attention_mappings — full-attention input_layernorm → self_attn.q/k/v, post_attention_layernorm → mlp.gate/up, and mlp.up_proj → mlp.down_proj, with duo_scaling="both" and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.lm_head, norms, linear_attn.in_proj_a/b, the vision tower, and MTP heads.Tested with LMDeploy turbomind:
bash
from lmdeploy import pipeline, TurbomindEngineConfigpipe = pipeline("TheUnderscore/Qwen3.8-Queen-27B-W4A16-AWQ",backend_config=TurbomindEngineConfig(tp=2,model_format="compressed-tensors",language_model_only=True,cpu_realtime_conversion=True,),)print(pipe("Hello, who are you?").text)
quantize-awq-hybrid.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered -of-N shards). --offload_dir selects where per-rank CPU offload temp folders live (defaults to the current working directory).model-nonquant.safetensors — unquantized tensors (mtp.* and model.visual.*) preserved BF16 so the full model architecture is loadable.