GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
GotoAI-Inc
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
GotoAI-Inc
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
Weights were quantized from bfloat16 to int4, group size 128, symmetric, weight-only
(activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was
used and the model was never loaded — the quantizer operates directly on the safetensors.
Architecture, tokenizer, chat template and processor config are the vendor's, unmodified.
Left at bfloat16 — quantizing these produces a checkpoint vLLM cannot load, because it
builds Gemma 4's non-text towers with quant_config=None:
vision_tower, vision_embedder, embed_vision and the vision projectionsaudio_tower, embed_audio and the audio projectionembed_tokens (tied to the output head in this checkpoint) and lm_head| this repo | google/…-qat-w4a16-ct | |
|---|---|---|
| total size | 7.74 GB | 10.26 GB |
| method | data-free RTN (PTQ) | quantization-aware training |
| group size | 128 | 32 |
| quantized modules | 328 (72.6% of bytes) | 328 (59.7% of bytes) |
| output head | tied to embed_tokens | separate lm_head.weight (2.01 GB, bf16) |
The 2.5 GB difference is almost entirely those two rows: Google's build materializes an
untied lm_head (2.01 GB) and its group-32 scales cost ~0.5 GB more than group-128.
Built with llm-quantizer — one command:
bash
./llmq.py run --profile gemma-4-12b-it
which is equivalent to:
python
# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,# transformers==5.15.1, torch==2.13.0from llmcompressor import model_free_ptqmodel_free_ptq(model_stub="google/gemma-4-12B-it", # re-sharded to 4 GB shards firstsave_directory="gemma-4-12B-it-W4A16",scheme="W4A16",ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed_tokens.*"],max_workers=2,device="cuda:0",)
The whole build ran on a 16 GB laptop GPU: model_free_ptq loads one shard at a time, so
re-sharding the single 23.92 GB source file into 4 GB pieces puts the peak at ~5 GB.
bash
vllm serve <this-repo> --max-model-len 65536
Do not pass --quantization; compressed-tensors is detected from config.json. The
int4 W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.
Version note. Gemma 4 needs transformers >= 5.10, but vLLM ≤ 0.27.1 cannot serve
Gemma 4 with transformers >= 5.15: head_dim became a per-layer attribute and older
vLLM reads it globally, raising AmbiguousGlobalPerLayerAttributeError at engine-config
time. Use either transformers < 5.15 with vLLM 0.25–0.27, or vLLM >= 0.28, which
handles both layouts.
None. This checkpoint has not been benchmarked, and round-to-nearest quantization is expected to degrade quality more than the vendor's QAT build — how much, for your task, is unmeasured here. Treat published Gemma 4 benchmark numbers as describing the bf16 model, not this one.
Apache 2.0, inherited from the base model — see LICENSE and Google's
Gemma 4 license page. The base
repository ships no LICENSE file, so the Apache-2.0 text is included here for
redistribution. "Gemma" is Google's mark; this repository is not endorsed by or
affiliated with Google.