GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
GotoAI-Inc
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
GotoAI-Inc
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
Weights were quantized from bfloat16 to int4, group size 128, symmetric, weight-only
(activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was
used and the model was never loaded — the quantizer operates directly on the safetensors.
Architecture, tokenizer, chat template and processor config are the vendor's, unmodified.
410 Linear modules were converted, covering 79.2% of the checkpoint's bytes:
| component | precision | size |
|---|---|---|
| language-model linears (60 layers) | int4 g128 | 15.10 GB (79.2%) |
embed_tokens (tied to the output head) | bfloat16 | 2.82 GB (14.8%) |
vision tower + embed_vision projection | bfloat16 | 1.15 GB (6.0%) |
| norms, layer scalars | bfloat16 | 0.003 GB |
| total | 19.07 GB |
Left at bfloat16:
vision_tower, embed_vision — the tower's intermediate_size is 4304, which is
not divisible by 64, so int4 Marlin-style kernels cannot serve it. vLLM's Gemma 4 loader
has an explicit guard for exactly this case and builds the towers unquantized; a
checkpoint carrying quantized vision weights is asking for a version-dependent load
failure.embed_tokens — precision-sensitive, and it is the output head here
(tie_word_embeddings: true, and no lm_head tensor exists in the checkpoint).Unlike the 12B, this model has no audio tower (audio_config: null, zero audio
tensors — Gemma 4 ships audio only on E2B, E4B and 12B), so the profile's re:.*audio.*
pattern matches nothing here. It also uses a conventional dedicated vision encoder rather
than the 12B's encoder-free "Unified" design, which is why the tower shows up as 1.15 GB of
separate weights.
| this repo | google/…-qat-w4a16-ct | |
|---|---|---|
| total size | 19.07 GB | 23.27 GB |
| method | data-free RTN (PTQ) | quantization-aware training |
| group size | 128 | 32 |
| quantized modules | 410 (79.2% of bytes) | 410 (70.8% of bytes) |
| int4 payload | 14.64 GB | 14.64 GB |
| scales | 0.46 GB |
Both builds quantize the same 410 modules to an identical 14.64 GB of packed int4. The
entire 4.20 GB difference is the other two rows: group-32 scales cost 1.37 GB more than
group-128, and Google's build materializes an untied lm_head (2.82 GB) even though its
config still says tie_word_embeddings: true.
bash
vllm serve GotoAI-Inc/gemma-4-31B-it-W4A16 \--max-model-len 65536 \--enable-auto-tool-choice --tool-call-parser gemma4 \--reasoning-parser gemma4
Do not pass --quantization; compressed-tensors is detected from config.json. The int4
W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.
Version note. Gemma 4 needs transformers >= 5.10.1, but vLLM ≤ 0.27.1 cannot serve
Gemma 4 with transformers >= 5.15: at 5.15 the config turns global_head_dim into a
per-layer head_dim override, and older vLLM reads head_dim globally, raising
AmbiguousGlobalPerLayerAttributeError at engine-config time. Use either
transformers < 5.15 with vLLM 0.25.1–0.27, or vLLM >= 0.28, which handles both
layouts. The architecture itself (Gemma4ForConditionalGeneration) and both gemma4
parsers are present from 0.25.1 on.
Gemma 4 interleaves 50 sliding-attention layers (window 1024, 16 KV heads, head_dim 256)
with 10 global layers (every 6th, 4 KV heads, global_head_dim 512). The sliding layers'
cache is bounded by the window at roughly 0.8 GB per sequence regardless of context
length; only the global layers grow, at about 80 KB/token:
| context | KV cache | + weights |
|---|---|---|
| 32k | ~3.4 GB | ~22.5 GB |
| 128k | ~11.3 GB | ~30.4 GB |
| 256k (max) | ~21.8 GB | ~40.9 GB |
So a 48 GB card runs this comfortably at 128k. A 24 GB card is marginal even at 32k once
activations and CUDA graphs are counted — add --language-model-only, which frees the
1.15 GB tower (vLLM skips tower weights entirely when every modality limit is zero) plus
the multimodal profiling headroom, and keep the context modest. This is arithmetic from
config.json, not a measured deployment.
Note that the global layers use unified keys and values (attention_k_eq_v: true, and the
checkpoint has no v_proj on those layers). That saves weight bytes, but vLLM loads the K
weights into both the K and V slots, so the cache still holds both copies — the table above
already assumes that.
The chat template defaults enable_thinking to false, so this model does not think
unless asked. Both knobs are template variables passed through chat_template_kwargs:
json
{"chat_template_kwargs": {"enable_thinking": true}} // injects <|think|> into the system turn{"chat_template_kwargs": {"preserve_thinking": true}} // keep thinking on tool-call turns
With thinking on, reasoning is emitted as <|channel>thought … <channel|>, which
--reasoning-parser gemma4 splits into reasoning_content. Per the base model card,
thinking from earlier turns should not be replayed into history — except on tool-call
turns, which is exactly what preserve_thinking keeps.
Sampling, per the base model card: temperature=1.0, top_p=0.95, top_k=64 for all use
cases. Place image content before the text in a prompt. The visual token budget is
configurable (70/140/280/560/1120, default 280) — lower it for video and captioning,
raise it for OCR and document parsing.
Built with llm-quantizer:
bash
./llmq.py run --profile gemma-4-31b-it
which re-shards the source — it ships as 2 shards, the larger 49.78 GB, which no consumer GPU can hold — into 17 pieces of ~4 GB, then:
python
# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,# transformers==5.15.1, torch==2.13.0from llmcompressor import model_free_ptqmodel_free_ptq(model_stub="gemma-4-31B-it-resharded",save_directory="gemma-4-31B-it-W4A16",scheme="W4A16",ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed_tokens.*"],device="cuda:0",)
A job holds one shard at a time, so after re-sharding the build peaks at a few GB of VRAM rather than the model size — no large GPU required.
No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than the vendor's QAT build; how much, for your task, is unmeasured here. Treat published Gemma 4 benchmark numbers as describing the bf16 model, not this one — and note that a directly comparable QAT checkpoint exists, so if quality matters more than the 4.2 GB, use Google's.
Apache 2.0, inherited from the base model — see LICENSE and Google's
Gemma 4 license page. The base
repository ships no LICENSE file, so the Apache-2.0 text is included here for
redistribution. "Gemma" is Google's mark; this repository is not endorsed by or
affiliated with Google.
| 1.83 GB |
| output head | tied to embed_tokens | separate lm_head (2.82 GB, bf16) |