What is quantized
Table with columns: Part, Precision, Why| Part | Precision | Why |
|---|
Decoder MLPs gate_proj / up_proj / down_proj (60 layers) | int8 weights per channel, int8 dynamic per token | most of the bytes; fast Triton int8 scaled-mm on gfx90a |
Attention q_proj / k_proj / o_proj (60 layers), v_proj (50 layers) | as above | as above |
Norms, embeddings (embed_tokens, 2.8 GB), vision tower | bf16 | standard; the vision tower is small and left alone |
- 410 quantized modules; 33.3 GB in 2 shards (the bf16 source is 62.5 GB).
- No calibration data: the weight scales come from a
memoryless-mse observer, and the activation scales are computed at
runtime per token.
How it was made
- Tool: llm-compressor 0.14.0
model_free_ptq with compressed-tensors 0.19.0, data-free, on CPU only.
- Scheme: weights
num_bits=8, type=int, symmetric=true, strategy=channel, observer memoryless-mse; input
activations num_bits=8, type=int, symmetric=true, strategy=token, dynamic=true.
- Targets (both the checkpoint's and vLLM's module names):
re:^(?:model\.language_model|language_model\.model)\.layers\.\d+\.(?:mlp\.(?:gate_proj|up_proj|down_proj)|self_attn\.[qkvo]_proj)$
- Source revision:
842da3794eaa0b77d5f08bae87a17459d91ff475.
- Built in 22 min on one worker process. A dequantization check on one shard gave a relative error of 0.010 to 0.013 per tensor.
- Verified after the build: every scheme field, the targets in both name forms, every shard present.
Serving (vLLM, ROCm)
Tested with vLLM v0.33.0, one copy per MI210. Quantization is detected from the config; do not pass --quantization.
vllm serve <this repo> \ --max-model-len=65536 --gpu-memory-utilization=0.95 --kv-cache-dtype=auto \ --max-num-batched-tokens=8192 \ --limit-mm-per-prompt='{"image":0,"audio":0}' \ --reasoning-parser=gemma4 --tool-call-parser=gemma4 \ --chat-template=/app/vllm/examples/tool_chat_template_gemma4.jinja
The startup log should name TritonInt8ScaledMMLinearKernel for CompressedTensorsW8A8Int8. The weights take 31.7 GiB of the
card. One 65,536-token request needs 18.29 GiB of KV cache, which does not fit at --gpu-memory-utilization=0.90 (17.21 GiB
available), so use 0.95 or a shorter context.
Results
Measured on one MI210 per copy on vLLM v0.33.0, bf16 activations:
Table with columns: Check, Result| Check | Result |
|---|
KV cache capacity (--gpu-memory-utilization=0.95) | 73,136 tokens; 1.12x concurrency at 65,536 tokens |
| Output tokens per second, 400 output tokens per request | 17.9 at concurrency 1, 33.7 at 2, 67.6 at 4 (total across requests) |
| Correctness checks (temperature 0, thinking on and off) | 17 x 60 = 1,020; train trip of 155 min both ways; a get_weather tool call |
--dtype=float16 gave 16.9 / 35.0 / 69.1 output tokens per second, within run-to-run noise, so bf16 is the default.
The tool call chose the right function but returned the city argument with literal quote characters around the value
("Amsterdam"); it did so on both dtypes. These are sanity checks of the quantization, not a quality benchmark.
License and attribution
This is a derivative of Gemma 4 31B it by Google, distributed under the license of the original (Apache License 2.0 as stated
in the original model card; see the Gemma 4 license page). The tokenizer,
chat template and configuration files are copied unchanged from the original repository; config.json adds only the
quantization config.
The quantization changes the numerical precision of the weights listed above and nothing else. It is not an official Google
release, and Google has not reviewed or endorsed it.