GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
zankich
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
zankich
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
Two serving modes on pre-SM100 GPUs, one image, chosen by flags.
bash
docker run -d --gpus all -p 8000:8000 \-v /path/to/gemma-4-31B-it-W8A16-FP8KV:/models/gemma-4-31B-it-W8A16-FP8KV:ro \zankich/vllm-openai:0.29.0z \/models/gemma-4-31B-it-W8A16-FP8KV \--dtype bfloat16 \--kv-cache-dtype fp8 \--attention-backend FLASHINFER \--attention-config '{"use_trtllm_attention": false}' \--language-model-only
--language-model-only is required: FlashInfer does not support this model's multimodal attention, so the vision tower cannot serve and only wastes memory.
bash
docker run -d --gpus all -p 8000:8000 \-v /path/to/gemma-4-31B-it-W8A16-FP8KV:/models/gemma-4-31B-it-W8A16-FP8KV:ro \zankich/vllm-openai:0.29.0z \/models/gemma-4-31B-it-W8A16-FP8KV \--dtype bfloat16 \--kv-cache-dtype fp8_e5m2 \--attention-backend TRITON_ATTN
The vision tower loads by default and the server accepts images.
The FlashInfer mode serves the calibrated E4M3 scales as shipped with --language-model-only. The Triton mode serves image support with the KV cache as E5M2, the same scales but roughly twice the per-element quantization noise, because Triton cannot compile the E4M3 dtype below SM89. Quality in E5M2 mode was verified with greedy and structured probes plus an exact image-content test; no logprob-level comparison against E4M3 has been run.
| Property | Value |
|---|---|
| Base model | google/gemma-4-31B-it |
| Weights | INT8, group size 128, symmetric, weight-only GPTQ |
| Activations | bfloat16 |
| KV cache | FP8 E4M3, static per-tensor symmetric scales |
| Format | compressed-tensors, auto-detected by vLLM from config.json |
| Size | 33.7 GB total across 17 shards, down from ~60 GB bf16 |
| Vision tower | preserved at bf16, the checkpoint remains a vision-language model |
[!WARNING] Never serve this checkpoint with bfloat16 KV. It does not fail at boot. It silently corrupts generation: greedy output degrades to CJK garbage, every structured probe fails, and the server reports healthy throughout. The calibrated FP8 scales are part of the checkpoint's contract, not a tuning option.
The KV cache scales are calibrated during the same forward passes as the GPTQ weights, via kv_cache_scheme on the modifier, so no separate calibration step runs. Gemma-4 mixes two KV geometries, 50 sliding-window layers at 16 KV heads and head_dim 256 and 10 full-attention layers at 4 KV heads and head_dim 512. The checkpoint carries 120 k_scale/v_scale tensors, one pair per layer, each sized for its own geometry. Serve with --kv-cache-dtype fp8 so vLLM loads them.
Every Linear except the ignore list below, 410 quantized GEMMs: all attention and FFN projections across both of gemma-4's attention geometries, the 50 sliding-window layers and the 10 full-attention layers.
lm_headThe ignore-list shape as executed:
python
KV_CACHE_SCHEME = {"num_bits": 8,"type": "float","strategy": "tensor","dynamic": False,"symmetric": True,}recipe = GPTQModifier(targets="Linear",scheme="W8A16",ignore=["lm_head","re:.*vision.*","re:.*audio.*","re:.*embed.*",],kv_cache_scheme=KV_CACHE_SCHEME,)
As executed the run also used dampening_frac: 0.2 and actorder: static. The full recipe ships as recipe.yaml.
512 samples at max sequence length 2048, a chat-agent mix of interactive coding turns and function-level public code in Python, TypeScript, Go and shell. The calibration set is built from private session data and is not published. No accuracy benchmark delta against the bf16 base is published for this checkpoint.
Both fixes are on the v0.29.0z branch of the zankich/vllm fork, whose README documents them and whose build-fork-image.sh builds the serve image. zankich/vllm-openai:0.29.0z on Docker Hub is that build. See Running the model for both serving modes.
What the two fixes do:
No supported CUDA architectures found for major versions [10, 11, 12]. The fix widens that existing opt-in to the FP8 one-byte dtypes. The FlashInfer lines are required together with it below SM100: the default attention path selects a trtllm-gen variant that is SM100+ only.Both become unnecessary upstream: a FlashInfer that recognizes FP8 as one-byte KV on the SM8 large-head path, and a Triton path that compiles e4m3 below SM89, each retire their half.
compressed-tensors-heterogeneous-kv.patch is needed only to produce the quant. The quant was built with llm-compressor 0.13.0, which installs compressed-tensors 0.18.0, and the patch is written against exactly that version. Gemma-4 mixes two attention geometries in one model, Transformers 5 exposes that as a per_layer_config list, and stock compressed-tensors 0.18.0 hands every attention module the global text config, so get_num_kv_heads raises AmbiguousGlobalPerLayerAttributeError before any quantization runs. The patch resolves each module's own per_layer_config slice by layer index. Homogeneous models take the unchanged global path, so it is inert for everything except heterogeneous models. Check upstream first: a release carrying per-layer config resolution in initialize_hooked_kv_cache makes it obsolete.
To build a compatible llm-compressor image from the patch:
dockerfile
FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime# the base image's torchvision is compiled against torch 2.5.1; llm-compressor# pulls a newer torch and the stale extensions break the transformers import chainRUN pip uninstall -y torchvision torchaudio 2>/dev/null || true# Triton JIT-compiles CUDA kernels at first use; the runtime image ships no compilerRUN apt-get update && apt-get install -y --no-install-recommends \build-essential \&& rm -rf /var/lib/apt/lists/*RUN pip install --no-cache-dir \"llmcompressor==0.13.0" \"accelerate" \"safetensors" \"datasets" && \pip install --no-cache-dir --force-reinstall "transformers"COPY compressed-tensors-heterogeneous-kv.patch /tmp/compressed-tensors-heterogeneous-kv.patchRUN cd /opt/conda/lib/python3.11/site-packages/compressed_tensors && \patch --batch --fuzz=3 -p0 -i /tmp/compressed-tensors-heterogeneous-kv.patch