GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
lokeshe09
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
lokeshe09
Model Tree
google/gemma-4-26B-A4B-it
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Method | W4A16 (int4 weight-only) |
| Strategy | Per-channel (group_size=-1) |
| Scheme | symmetric |
| Tool | llmcompressor |
| Target GPU | A100 (Ampere cc=80) via AllSpark kernel |
On Ampere GPUs vLLM's only working W4A16 kernel is AllSpark, which only
accepts group_size=-1. Grouped kernels (Marlin, Conch) require either
group_size=128 and all weight input dims divisible by 128 — the MoE
expert down_proj layers have input_size=2112 (not divisible by 128), so
grouped quantization fails at serve time regardless of group size.
lm_headmodel.embed_vision.embedding_projectionpython
from transformers import AutoModelForCausalLM, AutoProcessorimport torchmodel = AutoModelForCausalLM.from_pretrained("lokeshe09/gemma-4-26B-A4B-it-W4A16-A100",device_map="auto",torch_dtype=torch.bfloat16,trust_remote_code=True,)processor = AutoProcessor.from_pretrained("lokeshe09/gemma-4-26B-A4B-it-W4A16-A100", trust_remote_code=True)