GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
gratex
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
gratex
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Base Model | google/gemma-4-31B-it |
| Quantization Method | AutoRound SignRound (symmetric, group_size 64) |
| Weight Precision | INT4 (4-bit, symmetric, weight-only) |
| Activation Precision | BF16 |
| Group Size | 64 |
| Quantization Library | auto-round 0.16.0 → llm-compressor |
| Format | compressed-tensors, pack-quantized |
| Architecture | Gemma4ForConditionalGeneration (dense, hybrid local/global attention) |
| Text layers | 60 (5:1 local sliding-window : global attention; sliding window 1024) |
| Hidden size | 5,376 |
| Intermediate size | 21,504 |
| Attention heads | 32 (query), 16 (key/value, GQA); head_dim 256 (local) / 512 (global) |
| Vocabulary | 262,144 |
| Context window | 262,144 tokens |
| Quantized | Text-decoder Linear layers (attention q/k/v/o, MLP gate/up/down) |
| Preserved in BF16 | Vision tower (191 tensors), lm_head, embed_tokens, layer-0 attention block |
| Model Size | ~19.7 GB (single shard) |
| Calibration | 512 samples (SK + EN mix, 23.4% SK token share), seqlen 4,096 |
text
auto-round \ --model_name gemma-4-31B-it-BF16 \ --scheme W4A16 --bits 4 --group_size 64 \ --format auto_round:llm_compressor \ --dataset calibration_data.jsonl \ --nsamples 512 --seqlen 4096 \ --device_map 0quantization_config.json uses a per-module regex target (Linear) with group_size 64, 4-bit symmetric weights; 196 tensors (vision tower + lm_head + layer-0 attention) are kept in BF16.
Evaluated with lm-evaluation-harness v0.4.12 against a vLLM deployment of this checkpoint (TP=1, RTX 5090). Deterministic decoding (temperature 0). Few-shot per the harness convention (5-shot for LL/GSM8K, 3-shot CoT for BBH).
Loglikelihood tasks were run with --apply_chat_template: without the template the raw completion scores ~random (ARC acc_norm 0.326 without vs 0.654 with).
| Benchmark | Metric | Score |
|---|---|---|
| MMLU | acc (5-shot, 57 subjects) | 0.876 |
| MMLU-Pro | exact_match (5-shot, 14 categories) | 0.838 |
| Global MMLU (en) | acc (5-shot) | 0.890 |
| ARC-Challenge | acc_norm (5-shot) | 0.654 |
| BBH | exact_match (3-shot CoT, 27 subtasks) | 0.951 ¹ |
| GSM8K | exact_match, strict (5-shot) |
¹ Re-parsed from sample JSONL — the harness get-answer filter under-reports for two subtask families: BBH 0.9181 → 0.9509 (formal_fallacies 0.348→0.996, geometric_shapes 0.632→0.804), BABILong 0.7986 → 0.8316 (qa7 0.352→0.976 — model answered a numeral where the target was a word). Same re-parse convention as previously published Gratex rows. Other tasks audited with no false negatives.
LongBench by category: few-shot 0.654, synthetic 0.681, multi-doc QA 0.526, code 0.519, single-doc QA 0.509, summarization 0.222.
A graded SK+EN battery (arithmetic, factoid, multi-step) plus long-form generation probes (degeneration / repetition detection), driven through the vLLM /v1/chat/completions API. This checkpoint:
| Metric | Value |
|---|---|
| Graded probes (12, 6 SK + 6 EN) | 12/12 (SK 1.000 / EN 1.000) |
| Long-form degeneracy (max repeat coverage) | 0.029 |
| Long-form average length | ~4,714 chars |
Perplexity and KL-divergence-vs-base were not reliably measurable for this checkpoint on the current stack, so they are not reported.
Gemma-4 uses a hybrid local/global attention architecture (head_dim 256 on sliding layers, global_head_dim 512 on full layers, sliding window 1024). At the time of measurement, the text-only forward path for Gemma-4 in the current inference stacks returned corrupted token-level logits. The BF16 base model is affected identically, so the distortion comes from the inference stack and not from this quantization. Two upstream reports describe the same behaviour:
gemma-4-31B-it and gemma-4-26B-A4B-it.Because raw-text continuation also degenerates for this instruction-tuned model without its chat template, a perplexity number on a raw corpus would not be meaningful. The downstream benchmark scores and the generative probes above are measured on the chat-template path and are unaffected by this issue.
The BF16 base model was measured as a control and reproduces the same behaviour, so this checkpoint adds no degradation of its own.
Benchmarked with a stdlib OpenAI client harness on 1× NVIDIA RTX 5090 32 GB, TP=1, vLLM 0.25.1 (compressed-tensors W4A16), fp8 KV cache, --max-model-len 32768, --max-num-seqs 8, gpu-memory-utilization 0.92, CUDA graphs enabled. Unique random prompts per request (no prefix-cache hits); temperature 0; 4 runs, average of last 3.
Prefill throughput was measured at four input lengths (−512, −2,048, −8,192, −16,384 tokens) because both prefill and decode speed depend on prompt/context length. Prefill = TTFT of a max_tokens=1 request over a unique random prompt; decode = per-request rate over a 256-token continuation from a short (~20-token) prompt.
| Metric | Value |
|---|---|
| Prefill throughput (512–16,384 in) | ~2,928–3,261 tok/s (median ~3,150) |
| Decode throughput (per request, 256 out) | ~77.7 tok/s |
KV cache (at gpu-memory-utilization 0.92, context 32,768):
| KV dtype | KV tokens | KV GiB |
|---|---|---|
| fp8 | 113,402 | 8.99 |
| bfloat16 | 56,408 | 8.95 |
fp8 KV cache provides ~2.0× the token capacity of bf16 at the same gpu_memory_utilization, with no measurable decode impact and only ~3–5 % lower prefill (within run-to-run noise).
The compressed-tensors W4A16 scheme dispatches to fused INT4 GEMM kernels (Marlin on NVIDIA sm_75+; Triton/ROCm on AMD gfx).
| GPU layout | Total VRAM | Weights | Status |
|---|---|---|---|
| 1× 32 GB (RTX 5090, TP=1) | 32 GB | 19.7 GB | full context, config below |
Tensor-parallel size 1 (single shard). The 262K model context does not fit on a 32 GB card with fp8 KV cache. You need 48 GB VRAM or offload mechanisms.
text
docker run -d --name vllm-gemma4-g3 \ --gpus all \ --ipc=host \ -v /path/to/model:/models:ro \ -p 8000:8000 \ vllm/vllm-openai:vllm-x86_64-cu13-0.25.1-7a33ba9 \ --model /models \ --served-model-name gemma-4-31b-w4a16 \ --tensor-parallel-size 1 \ --max-model-len 65536 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.90 \ --dtype bfloat16 \ --kv-cache-dtype fp8 \ --kv-offloading-size 32 \ --trust-remote-code \ --limit-mm-per-prompt '{"image": 4}'text
curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"gemma-4-31b-w4a16","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":16}'python
from transformers import AutoProcessor, AutoModelForImageTextToTextimport torch model_id = "gratex/Gemma-4-31B-it-W4A16-g64-sym"processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForImageTextToText.from_pretrained( model_id, device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True,) messages = [{"role": "user", "content": "Explain INT4 weight quantization in two sentences."}]inputs = processor.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt",).to(model.device) with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=256)print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))| File | Description |
|---|---|
model.safetensors | Quantized model weights (INT4 + BF16), single shard, ~19.7 GB |
config.json | Model configuration (Gemma4 hybrid attention, 262K context) |
quantization_config.json | compressed-tensors pack-quantized parameters (g64, 4-bit symmetric) |
tokenizer.json / tokenizer_config.json | Tokenizer (vocab 262,144) |
chat_template.jinja | Native chat template (inherited from base) |
This quantization is released under the Apache License 2.0, following the base model's license.
The base model google/gemma-4-31B-it is licensed under Apache 2.0.
bibtex
@misc{gemma-4-31b-it-w4a16-g64-sym, title = {Gemma-4-31B-it AutoRound W4A16 Quantization (g64, Symmetric)}, author = {Gratex International}, year = {2026}, howpublished = {\url{https://huggingface.co/gratex/Gemma-4-31B-it-W4A16-g64-sym}}, note = {Quantized with Intel AutoRound 0.16.0 (SignRound), compressed-tensors pack-quantized format}}This quantization was produced using hardware and infrastructure provided by Gratex International, a.s.
Original Model: google/gemma-4-31B-it Quantization Tool: AutoRound Quantization Format: compressed-tensors (pack-quantized) Deployment Engine: vLLM
| 0.961 |
| GSM8K Platinum | exact_match, strict (5-shot) | 0.989 |
| HellaSwag | acc_norm (5-shot) | 0.697 |
| Winogrande | acc (5-shot) | 0.725 |
| PIQA | acc_norm (5-shot) | 0.789 |
| TruthfulQA | mc2 (0-shot) | 0.644 |
| HumanEval | pass@1 (0-shot, markdown extraction) | 0.982 |
| MBPP | pass@1 (0-shot) | 0.876 |
| IFEval | prompt-level strict (0-shot) | 0.909 |
| BABILong | mean acc (0-shot, 20 tasks) | 0.832 ¹ |
| LongBench | mean score (0-shot, 21 subtasks) | 0.511 |
generation_config.json | Default generation parameters |
processor_config.json | Multimodal processor (vision + audio, inherited from base) |
LICENSE | Apache License 2.0 (base model license) |