GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
zankich
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
zankich
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
Two serving modes on pre-SM100 GPUs, one image, chosen by flags.
bash
docker run -d --gpus all -p 8000:8000 \-v /path/to/gemma-4-12B-it-W4A16-FP8KV:/models/gemma-4-12B-it-W4A16-FP8KV:ro \zankich/vllm-openai:0.29.0z \/models/gemma-4-12B-it-W4A16-FP8KV \--dtype bfloat16 \--kv-cache-dtype fp8 \--attention-backend FLASHINFER \--attention-config '{"use_trtllm_attention": false}' \--language-model-only
--language-model-only is required on this path: FlashInfer does not support the multimodal attention of the underlying Gemma4Unified architecture, so the audio and vision paths cannot serve here and only waste memory. Text-only serving with the calibrated E4M3 scales is the configuration this checkpoint was built on.
bash
docker run -d --gpus all -p 8000:8000 \-v /path/to/gemma-4-12B-it-W4A16-FP8KV:/models/gemma-4-12B-it-W4A16-FP8KV:ro \zankich/vllm-openai:0.29.0z \/models/gemma-4-12B-it-W4A16-FP8KV \--dtype bfloat16 \--kv-cache-dtype fp8_e5m2 \--attention-backend TRITON_ATTN \--max-num-batched-tokens 3072
Both vision and audio inputs serve on this path. --max-num-batched-tokens must exceed 2496: one image is 2496 encoder tokens and smaller budgets crash the multimodal budget check at startup. The image ships the vllm[audio] extras, so audio input needs no extra installation.
The FlashInfer mode serves the calibrated E4M3 scales as shipped with --language-model-only. The Triton mode serves vision and audio with the KV cache as E5M2 — the same scales but roughly twice the per-element quantization noise, because Triton cannot compile the E4M3 dtype below SM89. Verified on the Triton path: image inputs answered exactly (shape and color correct on a synthetic test image), and a 20-second real-speech clip transcribed with correct semantic content at roughly 80-85% word accuracy, the errors consistent with W4A16 weights plus the E5M2 KV path on accented archive audio. No logprob-level comparison against E4M3 has been run.
One interaction worth knowing when serving audio: verbatim transcription requests should disable thinking (chat_template_kwargs: {"enable_thinking": false} at request level). With thinking enabled, transcription requests tend to transcribe inside the reasoning channel and exhaust the token budget before producing content. Analysis-style audio requests behave normally with thinking on.
| Property | Value |
|---|---|
| Base model | google/gemma-4-12B-it |
| Weights | INT4, group size 128, symmetric, weight-only GPTQ |
| Activations | bfloat16 |
| KV cache | FP8 E4M3, static per-tensor symmetric scales |
| Format | compressed-tensors, auto-detected by vLLM from config.json |
| Size | 7.3 GB total across 4 shards, down from ~24 GB bf16 |
| Multimodal projections | audio and vision embedding projections preserved at bf16 |
The KV cache scales are calibrated during the same forward passes as the GPTQ weights, via kv_cache_scheme on the modifier, so no separate calibration step runs. The unified 12B mixes two KV geometries: 40 sliding-window layers at 8 KV heads and head_dim 256, and 8 full-attention multi-query layers at 1 KV head and head_dim 512. The checkpoint carries 96 k_scale/v_scale tensors, one pair per layer, each sized for its own geometry. Serve with --kv-cache-dtype fp8 so vLLM loads them.
[!WARNING] Serve with the calibrated FP8 scales, not bfloat16 KV. Bfloat16 KV silently corrupts generation while the server reports healthy: greedy output degrades to garbage and every structured probe fails, with nothing in the logs. The calibrated scales are part of the checkpoint's contract, not a tuning option.
Every Linear except the ignore list below: all attention and FFN projections across both attention geometries.
lm_headThe ignore-list shape as executed:
python
KV_CACHE_SCHEME = {"num_bits": 8,"type": "float","strategy": "tensor","dynamic": False,"symmetric": True,}recipe = GPTQModifier(targets="Linear",scheme="W4A16",ignore=["lm_head","re:.*vision.*","re:.*audio.*","re:.*embed.*",],kv_cache_scheme=KV_CACHE_SCHEME,)
As executed the run also used dampening_frac: 0.2. The full recipe ships as recipe.yaml.
512 samples at max sequence length 2048, a chat-agent mix of interactive coding turns and function-level public code in Python, TypeScript, Go and shell. The calibration set is built from private session data and is not published. No accuracy benchmark delta against the bf16 base is published for this checkpoint.
Both fixes are on the v0.29.0z branch of the zankich/vllm fork, whose README documents them. zankich/vllm-openai:0.29.0z on Docker Hub is the prebuilt serve image.
What the fixes cover for this model: the unified 12B's full-attention layers are large-head (head_dim 512), and FlashInfer gates one-byte-KV large-head modules to SM100+, so FP8 KV needs the widened SM8 opt-in; the Triton fix serves the multimodal path below SM100 with the KV cache as e5m2, calibrated scales loading unchanged. The FlashInfer lines are required together on anything earlier than SM100: the default attention path selects a trtllm-gen variant that is SM100+ only.
The compressed-tensors-heterogeneous-kv.patch in this repo is needed only to produce the quant — it patches llm-compressor's compressed-tensors dependency, not vLLM. The quant was built with llm-compressor 0.13.0, which installs compressed-tensors 0.18.0, and the patch is written against exactly that version. The unified 12B mixes two attention geometries in one model, Transformers 5 exposes that as a per_layer_config list, and stock compressed-tensors 0.18.0 hands every attention module the global text config, so get_num_kv_heads raises AmbiguousGlobalPerLayerAttributeError before any quantization runs. The patch resolves each module's own per_layer_config slice by layer index. Homogeneous models take the unchanged global path, so it is inert for everything except heterogeneous models. Check upstream first: a release carrying per-layer config resolution in initialize_hooked_kv_cache makes it obsolete.
The repo ships the official chat_template.jinja, verbatim from the base model. vLLM and transformers auto-detect a chat_template.jinja file next to the weights, so this is what you get by default.