nameistoken

Gemma-4-31B-it-Quark-W8A8-INT8

README

License: apache-2.0

Model Details

Table

Base Model	`google/gemma-4-31B-it`
Architecture	`Gemma4ForConditionalGeneration` (multimodal: text + vision)
Parameters	31 B text decoder (quantized) + vision tower & embeddings kept in BF16
Quantization	W8A8 INT8 (per-channel weight + per-token dynamic activation)
Quantizer	AMD Quark `0.11.1` (`ptpc_int8` scheme, `pack_method='order'`)
Model Size	~32 GB (single `model.safetensors`)
Original Size	~62.5 GB (BF16)
Compression	~2× size reduction

Quantization Scheme

Table with columns: Component, dtype, Granularity, Mode
Component	dtype	Granularity	Mode
Weight	INT8	per-channel (`ch_axis=0`)	symmetric, static
Activation	INT8	per-token (`ch_axis=1`)	symmetric, dynamic
`lm_head`	BF16	—	unquantized

Accuracy

GSM8K 8-shot evaluation on the full 1319-question test split (vLLM, temperature=0, concurrency=16, max_tokens=512, standard chat template with #### answer format):

Table with columns: Model, Scheme, Accuracy, Correct
Model	Scheme	Accuracy	Correct
`google/gemma-4-31B-it` (BF16 baseline)	—	96.74%	1276 / 1319
This model (Quark W8A8 INT8)	per-channel weight + per-token act.	96.66%	1275 / 1319

Δ vs BF16: −0.08pp (essentially lossless).

How to Use

With vLLM (Recommended)

bash
# Start the server (single MI300X / MI350X / MI355X is enough; A100-80G also works)
vllm serve nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code

# Chat completion
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8",
  "messages": [{"role": "user", "content": "Hello! What is the capital of France?"}],
  "max_tokens": 256,
  "temperature": 0.7
}'

Hardware Requirements

Minimum: 1× GPU with ≥48 GB VRAM (e.g., AMD MI300X / MI350X / MI355X, NVIDIA A100-80G / H100).
For longer context or larger batches use TP=2 across two of the same GPUs.

Quantization Details

This model was quantized using AMD Quark's per-token per-channel INT8 scheme:

Weight quantization: INT8 per-channel (one scale per output channel), symmetric, static.
Activation quantization: INT8 per-token (one scale per token), symmetric, dynamic (computed at inference time).
Excluded layers: lm_head, *embed_tokens*, *vision_tower*, *embed_vision* (output head + token embedding + the entire vision tower remain in BF16).
Export: pack_method='order', weight_format='real_quantized', custom_mode='quark' → real INT8 weights with BF16 scales (no fake-quant, no zero-point).

Reproduce Quantization

bash
# 1. Environment
pip install amd-quark==0.11.1 datasets accelerate
git clone https://github.com/huggingface/transformers.git
cd transformers && pip install -e . --no-deps   # transformers main (>= 5.6.0.dev0)

python
# quark_gemma4_int8.py
import os, torch
from transformers import AutoTokenizer, Gemma4ForConditionalGeneration
from quark.torch import ModelQuantizer
from quark.torch.quantization.config.config import (
    QTensorConfig, QuantizationConfig, Config, Dtype,
)
from quark.torch.quantization.config.type import (
    RoundType, ScaleType, QSchemeType,
)
from quark.torch.quantization.observer import PerChannelMinMaxObserver

MODEL_IN  = "google/gemma-4-31B-it"
MODEL_OUT = "./Gemma-4-31B-it-Quark-W8A8-INT8"

tokenizer = AutoTokenizer.from_pretrained(MODEL_IN, trust_remote_code=True)
model = Gemma4ForConditionalGeneration.from_pretrained(
    MODEL_IN, torch_dtype=torch.bfloat16,
    device_map="auto", trust_remote_code=True,
)

weight_spec = QTensorConfig(
    dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver,
    symmetric=True, is_dynamic=False,
    qscheme=QSchemeType.per_channel, ch_axis=0,
    round_method=RoundType.round, scale_type=ScaleType.float,
)
input_spec = QTensorConfig(
    dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver,
    symmetric=True, is_dynamic=True,
    qscheme=QSchemeType.per_channel, ch_axis=1,
    round_method=RoundType.round, scale_type=ScaleType.float,
)

q_cfg = Config(
    global_quant_config=QuantizationConfig(
        input_tensors=input_spec, weight=weight_spec,
    ),
    exclude=[
        "lm_head", "*embed_tokens*",
        "*vision_tower*", "*embed_vision*",
    ],
)

quantizer = ModelQuantizer(q_cfg, multi_device=True)
model = quantizer.quantize_model(model, dataloader=None)  # PTQ, no calibration data needed for dynamic act
quantizer.freeze(model)

quantizer.export_model(
    model, MODEL_OUT,
    pack_method="order",
    weight_format="real_quantized",
    custom_mode="quark",
)
tokenizer.save_pretrained(MODEL_OUT)

Citation

If you use this model, please cite the original Gemma 4 release:

bibtex
@misc{google2026gemma4,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2026},
  url    = {https://huggingface.co/google/gemma-4-31B-it}
}

License

This model is released under the Apache License 2.0, following the Gemma 4 license under which the upstream google/gemma-4-31B-it weights are distributed by Google DeepMind.

This is a quantized derivative of google/gemma-4-31B-it. Per Apache 2.0 §4:

Modified files (the INT8-quantized model.safetensors and the appended quantization_config block in config.json) carry this notice as part of the model card.
Original copyright and attribution notices from the base model are preserved (see NOTICE).
A copy of the Apache 2.0 license text is included as LICENSE.

Original weights © Google DeepMind. Quantization performed by the model author; no warranty of any kind is provided (see LICENSE §7–8).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

nameistoken

Model Tree

Base

google/gemma-4-31B-it

Quantized

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table

Base Model	`google/gemma-4-31B-it`
Architecture	`Gemma4ForConditionalGeneration` (multimodal: text + vision)
Parameters	31 B text decoder (quantized) + vision tower & embeddings kept in BF16
Quantization	W8A8 INT8 (per-channel weight + per-token dynamic activation)
Quantizer	AMD Quark `0.11.1` (`ptpc_int8` scheme, `pack_method='order'`)
Model Size	~32 GB (single `model.safetensors`)
Original Size	~62.5 GB (BF16)
Compression	~2× size reduction

Quantization Scheme

Table with columns: Component, dtype, Granularity, Mode
Component	dtype	Granularity	Mode
Weight	INT8	per-channel (`ch_axis=0`)	symmetric, static
Activation	INT8	per-token (`ch_axis=1`)	symmetric, dynamic
`lm_head`	BF16	—	unquantized

Accuracy

GSM8K 8-shot evaluation on the full 1319-question test split (vLLM, temperature=0, concurrency=16, max_tokens=512, standard chat template with #### answer format):

Table with columns: Model, Scheme, Accuracy, Correct
Model	Scheme	Accuracy	Correct
`google/gemma-4-31B-it` (BF16 baseline)	—	96.74%	1276 / 1319
This model (Quark W8A8 INT8)	per-channel weight + per-token act.	96.66%	1275 / 1319

Δ vs BF16: −0.08pp (essentially lossless).

How to Use

With vLLM (Recommended)

bash
# Start the server (single MI300X / MI350X / MI355X is enough; A100-80G also works)
vllm serve nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code

# Chat completion
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8",
  "messages": [{"role": "user", "content": "Hello! What is the capital of France?"}],
  "max_tokens": 256,
  "temperature": 0.7
}'

Hardware Requirements

Minimum: 1× GPU with ≥48 GB VRAM (e.g., AMD MI300X / MI350X / MI355X, NVIDIA A100-80G / H100).
For longer context or larger batches use TP=2 across two of the same GPUs.

Quantization Details

This model was quantized using AMD Quark's per-token per-channel INT8 scheme:

Weight quantization: INT8 per-channel (one scale per output channel), symmetric, static.
Activation quantization: INT8 per-token (one scale per token), symmetric, dynamic (computed at inference time).
Excluded layers: lm_head, *embed_tokens*, *vision_tower*, *embed_vision* (output head + token embedding + the entire vision tower remain in BF16).
Export: pack_method='order', weight_format='real_quantized', custom_mode='quark' → real INT8 weights with BF16 scales (no fake-quant, no zero-point).

Reproduce Quantization

bash
# 1. Environment
pip install amd-quark==0.11.1 datasets accelerate
git clone https://github.com/huggingface/transformers.git
cd transformers && pip install -e . --no-deps   # transformers main (>= 5.6.0.dev0)

python
# quark_gemma4_int8.py
import os, torch
from transformers import AutoTokenizer, Gemma4ForConditionalGeneration
from quark.torch import ModelQuantizer
from quark.torch.quantization.config.config import (
    QTensorConfig, QuantizationConfig, Config, Dtype,
)
from quark.torch.quantization.config.type import (
    RoundType, ScaleType, QSchemeType,
)
from quark.torch.quantization.observer import PerChannelMinMaxObserver

MODEL_IN  = "google/gemma-4-31B-it"
MODEL_OUT = "./Gemma-4-31B-it-Quark-W8A8-INT8"

tokenizer = AutoTokenizer.from_pretrained(MODEL_IN, trust_remote_code=True)
model = Gemma4ForConditionalGeneration.from_pretrained(
    MODEL_IN, torch_dtype=torch.bfloat16,
    device_map="auto", trust_remote_code=True,
)

weight_spec = QTensorConfig(
    dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver,
    symmetric=True, is_dynamic=False,
    qscheme=QSchemeType.per_channel, ch_axis=0,
    round_method=RoundType.round, scale_type=ScaleType.float,
)
input_spec = QTensorConfig(
    dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver,
    symmetric=True, is_dynamic=True,
    qscheme=QSchemeType.per_channel, ch_axis=1,
    round_method=RoundType.round, scale_type=ScaleType.float,
)

q_cfg = Config(
    global_quant_config=QuantizationConfig(
        input_tensors=input_spec, weight=weight_spec,
    ),
    exclude=[
        "lm_head", "*embed_tokens*",
        "*vision_tower*", "*embed_vision*",
    ],
)

quantizer = ModelQuantizer(q_cfg, multi_device=True)
model = quantizer.quantize_model(model, dataloader=None)  # PTQ, no calibration data needed for dynamic act
quantizer.freeze(model)

quantizer.export_model(
    model, MODEL_OUT,
    pack_method="order",
    weight_format="real_quantized",
    custom_mode="quark",
)
tokenizer.save_pretrained(MODEL_OUT)

Citation

If you use this model, please cite the original Gemma 4 release:

bibtex
@misc{google2026gemma4,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2026},
  url    = {https://huggingface.co/google/gemma-4-31B-it}
}

License

This model is released under the Apache License 2.0, following the Gemma 4 license under which the upstream google/gemma-4-31B-it weights are distributed by Google DeepMind.

This is a quantized derivative of google/gemma-4-31B-it. Per Apache 2.0 §4:

Modified files (the INT8-quantized model.safetensors and the appended quantization_config block in config.json) carry this notice as part of the model card.
Original copyright and attribution notices from the base model are preserved (see NOTICE).
A copy of the Apache 2.0 license text is included as LICENSE.

Original weights © Google DeepMind. Quantization performed by the model author; no warranty of any kind is provided (see LICENSE §7–8).