GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nameistoken
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
nameistoken
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
| Base Model | google/gemma-4-31B-it |
| Architecture | Gemma4ForConditionalGeneration (multimodal: text + vision) |
| Parameters | 31 B text decoder (quantized) + vision tower & embeddings kept in BF16 |
| Quantization | W8A8 INT8 (per-channel weight + per-token dynamic activation) |
| Quantizer | AMD Quark 0.11.1 (ptpc_int8 scheme, pack_method='order') |
| Model Size | ~32 GB (single model.safetensors) |
| Original Size | ~62.5 GB (BF16) |
| Compression | ~2× size reduction |
| Component | dtype | Granularity | Mode |
|---|---|---|---|
| Weight | INT8 | per-channel (ch_axis=0) | symmetric, static |
| Activation | INT8 | per-token (ch_axis=1) | symmetric, dynamic |
lm_head | BF16 | — | unquantized |
embed_tokens | BF16 | — | unquantized |
vision_tower / embed_vision | BF16 | — | unquantized (multimodal preserved) |
GSM8K 8-shot evaluation on the full 1319-question test split (vLLM, temperature=0, concurrency=16, max_tokens=512, standard chat template with #### answer format):
| Model | Scheme | Accuracy | Correct |
|---|---|---|---|
google/gemma-4-31B-it (BF16 baseline) | — | 96.74% | 1276 / 1319 |
| This model (Quark W8A8 INT8) | per-channel weight + per-token act. | 96.66% | 1275 / 1319 |
Δ vs BF16: −0.08pp (essentially lossless).
bash
# Start the server (single MI300X / MI350X / MI355X is enough; A100-80G also works)vllm serve nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 \ --tensor-parallel-size 1 \ --max-model-len 8192 \ --gpu-memory-utilization 0.9 \ --trust-remote-code # Chat completioncurl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8", "messages": [{"role": "user", "content": "Hello! What is the capital of France?"}], "max_tokens": 256, "temperature": 0.7}'This model was quantized using AMD Quark's per-token per-channel INT8 scheme:
lm_head, *embed_tokens*, *vision_tower*, *embed_vision* (output head + token embedding + the entire vision tower remain in BF16).pack_method='order', weight_format='real_quantized', custom_mode='quark' → real INT8 weights with BF16 scales (no fake-quant, no zero-point).bash
# 1. Environmentpip install amd-quark==0.11.1 datasets accelerategit clone https://github.com/huggingface/transformers.gitcd transformers && pip install -e . --no-deps # transformers main (>= 5.6.0.dev0)python
# quark_gemma4_int8.pyimport os, torchfrom transformers import AutoTokenizer, Gemma4ForConditionalGenerationfrom quark.torch import ModelQuantizerfrom quark.torch.quantization.config.config import ( QTensorConfig, QuantizationConfig, Config, Dtype,)from quark.torch.quantization.config.type import ( RoundType, ScaleType, QSchemeType,)from quark.torch.quantization.observer import PerChannelMinMaxObserver MODEL_IN = "google/gemma-4-31B-it"MODEL_OUT = "./Gemma-4-31B-it-Quark-W8A8-INT8" tokenizer = AutoTokenizer.from_pretrained(MODEL_IN, trust_remote_code=True)model = Gemma4ForConditionalGeneration.from_pretrained( MODEL_IN, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,) weight_spec = QTensorConfig( dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver, symmetric=True, is_dynamic=False, qscheme=QSchemeType.per_channel, ch_axis=0, round_method=RoundType.round, scale_type=ScaleType.float,)input_spec = QTensorConfig( dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver, symmetric=True, is_dynamic=True, qscheme=QSchemeType.per_channel, ch_axis=1, round_method=RoundType.round, scale_type=ScaleType.float,) q_cfg = Config( global_quant_config=QuantizationConfig( input_tensors=input_spec, weight=weight_spec, ), exclude=[ "lm_head", "*embed_tokens*", "*vision_tower*", "*embed_vision*", ],) quantizer = ModelQuantizer(q_cfg, multi_device=True)model = quantizer.quantize_model(model, dataloader=None) # PTQ, no calibration data needed for dynamic actquantizer.freeze(model) quantizer.export_model( model, MODEL_OUT, pack_method="order", weight_format="real_quantized", custom_mode="quark",)tokenizer.save_pretrained(MODEL_OUT)If you use this model, please cite the original Gemma 4 release:
bibtex
@misc{google2026gemma4, title = {Gemma 4}, author = {Google DeepMind}, year = {2026}, url = {https://huggingface.co/google/gemma-4-31B-it}}This model is released under the Apache License 2.0, following the Gemma 4 license under which the upstream google/gemma-4-31B-it weights are distributed by Google DeepMind.
This is a quantized derivative of google/gemma-4-31B-it. Per Apache 2.0 §4:
model.safetensors and the appended quantization_config block in config.json) carry this notice as part of the model card.NOTICE).LICENSE.Original weights © Google DeepMind. Quantization performed by the model author; no warranty of any kind is provided (see LICENSE §7–8).
# Start the server (single MI300X / MI350X / MI355X is enough; A100-80G also works)vllm serve nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 \ --tensor-parallel-size 1 \ --max-model-len 8192 \ --gpu-memory-utilization 0.9 \ --trust-remote-code # Chat completioncurl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8", "messages": [{"role": "user", "content": "Hello! What is the capital of France?"}], "max_tokens": 256, "temperature": 0.7}'# 1. Environmentpip install amd-quark==0.11.1 datasets accelerategit clone https://github.com/huggingface/transformers.gitcd transformers && pip install -e . --no-deps # transformers main (>= 5.6.0.dev0)# quark_gemma4_int8.pyimport os, torchfrom transformers import AutoTokenizer, Gemma4ForConditionalGenerationfrom quark.torch import ModelQuantizerfrom quark.torch.quantization.config.config import ( QTensorConfig, QuantizationConfig, Config, Dtype,)from quark.torch.quantization.config.type import ( RoundType, ScaleType, QSchemeType,)from quark.torch.quantization.observer import PerChannelMinMaxObserver MODEL_IN = "google/gemma-4-31B-it"MODEL_OUT = "./Gemma-4-31B-it-Quark-W8A8-INT8" tokenizer = AutoTokenizer.from_pretrained(MODEL_IN, trust_remote_code=True)model = Gemma4ForConditionalGeneration.from_pretrained( MODEL_IN, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,) weight_spec = QTensorConfig( dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver, symmetric=True, is_dynamic=False, qscheme=QSchemeType.per_channel, ch_axis=0, round_method=RoundType.round, scale_type=ScaleType.float,)input_spec = QTensorConfig( dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver, symmetric=True, is_dynamic=True, qscheme=QSchemeType.per_channel, ch_axis=1, round_method=RoundType.round, scale_type=ScaleType.float,) q_cfg = Config( global_quant_config=QuantizationConfig( input_tensors=input_spec, weight=weight_spec, ), exclude=[ "lm_head", "*embed_tokens*", "*vision_tower*", "*embed_vision*", ],) quantizer = ModelQuantizer(q_cfg, multi_device=True)model = quantizer.quantize_model(model, dataloader=None) # PTQ, no calibration data needed for dynamic actquantizer.freeze(model) quantizer.export_model( model, MODEL_OUT, pack_method="order", weight_format="real_quantized", custom_mode="quark",)tokenizer.save_pretrained(MODEL_OUT)@misc{google2026gemma4, title = {Gemma 4}, author = {Google DeepMind}, year = {2026}, url = {https://huggingface.co/google/gemma-4-31B-it}}