Model Overview
This is a mixed-precision quantization of
google/gemma-4-26B-A4B-it optimized for
inference. The 128 MoE expert weights per layer are quantized to INT4 with GPTQ (group size 64);
non-expert transformer linear layers use FP8 dynamic quantization.
- Model Architecture: Gemma4ForConditionalGeneration
- Input: Text / Image
- Output: Text
- Model Optimizations:
- Weight quantization: INT4 (GPTQ, group size 64) for MoE expert weights; FP8 for all other linear layers
- Activation quantization: FP8 (dynamic, per-token) for non-expert linear layers; none for experts (W4A16)
- Unquantized: router, embeddings,
lm_head, vision tower
- Intended Use Cases: same as the base model — multimodal chat, instruction following, reasoning, tool use
- Release Date: 2026-09-03
- Version: 1.0
- Base model developers: Google DeepMind
- Quantization and release: Systalyze
The base model is a 30-layer / 128-expert MoE. Expert weights dominate the parameter count, but
each token touches only a few of them, so experts are stored in INT4 while the always-resident
path (attention, dense MLP, router, embeddings, vision tower) stays at FP8 or higher. The
checkpoint is stored in the compressed-tensors format and loads directly in vLLM.
Table with columns: Precision, On-disk | Precision | On-disk |
|---|
| Base | BF16 | 49 GB |
| FP8 dynamic | FP8 | 27 GB |
| ExpertsINT4 | experts INT4 / rest FP8 | 16 GB |
3.1× smaller than BF16, and small enough to serve from a single 24 GB card's worth
of weights (KV cache and activations are extra).
Models compared
Every number below was measured here, on the same vLLM stack, against these checkpoints of the
same base model. The BF16 base appears in the accuracy tables only, the AWQ checkpoint in the
throughput tables only:
Table with columns: Label in tables, Checkpoint, Scheme| Label in tables | Checkpoint | Scheme |
|---|
| Base BF16 | google/gemma-4-26B-A4B-it | Unquantized reference |
| ExpertsINT4 | systalyze/gemma-4-26B-A4B-it-ExpertsINT4-FP8 | MoE expert weights INT4 via GPTQ (W4A16, group size 64); non-expert transformer linear layers FP8 dynamic; router, embeddings, lm_head, and vision tower unquantized |
| FP8-dynamic | |
Throughput
Measured with aiperf against vLLM on 8×H200, ISL ≈ 6,883 / OSL ≈ 322,
completion_frac = 1.0 on every cell. The three checkpoints marked above use the same harness, same box,
same request stream.
out_tok_per_gpu_s — higher is better:
Table with columns: Concurrency, FP8-dynamic, ExpertsINT4, W4A16 (AWQ), ExpertsINT4 vs FP8| Concurrency | FP8-dynamic | ExpertsINT4 | W4A16 (AWQ) | ExpertsINT4 vs FP8 |
|---|
| 1 | 186.3 | 209.4 | 193.9 | +12.4% |
| 4 | 513.2 | 721.9 | 681.3 | +40.7% |
| 8 | 839.3 | 1295.5 | 1232.9 |
Latency at the same points (p50 TTFT / p50 ITL, ms):
Table with columns: Concurrency, FP8-dynamic, ExpertsINT4, W4A16 (AWQ)| Concurrency | FP8-dynamic | ExpertsINT4 | W4A16 (AWQ) |
|---|
| 1 | 158.2 / 4.92 | 207.0 / 4.16 | 242.6 / 4.45 |
| 4 | 166.8 / 6.15 | 59.5 / 5.16 | 61.4 / 5.43 |
| 8 | 183.8 / 6.54 | 61.6 / 5.43 | 62.6 / 5.72 |
| 16 | 209.8 / 8.35 | 284.8 / |
The win is concurrency-dependent. Mixed precision leads in the memory-bound regime
(concurrency 1–8, peaking at +54% over FP8 at concurrency 8) and trails FP8 once
the workload becomes compute-bound (concurrency 16–32, down to −15%). If you serve
at high concurrency, plain FP8 is the better choice. ExpertsINT4 beats the all-INT4 W4A16 checkpoint at every
concurrency point.
Accuracy
All three checkpoints were evaluated in the same session, on the same stack and the same
hardware, over full test sets (no subsampling). Higher is better.
Setup: lm_eval 0.4.13 against vLLM 0.27.1 (OpenAI-compatible API), one H200 per checkpoint,
identical chat template for all three arms, --apply_chat_template, greedy decoding
(temperature=0), --seed 1234, enable_thinking=False. Thinking-mode results are
reported separately below.
Table with columns: Category, Benchmark, Base BF16, FP8-dynamic, ExpertsINT4| Category | Benchmark | Base BF16 | FP8-dynamic | ExpertsINT4 |
|---|
| Instruction following | IFEval (0-shot, prompt-level strict) | 89.28 | 88.91 | 89.83 |
| IFEval (0-shot, inst-level strict) | 92.81 | 92.57 | 93.17 |
| Reasoning and knowledge | MMLU-Pro (5-shot CoT) | 82.06 | 82.09 | 81.30 |
Sanity check on the harness: our measured base scores land within about 1.4 points of the
figures Red Hat AI publishes for this same base model (IFEval 89.28 here, MMLU-Pro 82.06,
GPQA Diamond 72.22), so the numbers above are comparable to the wider literature even though
every value in the table was measured here.
MMLU-ProX lite, per language (5-shot)
Table with columns: Language, Base BF16, FP8-dynamic, ExpertsINT4| Language | Base BF16 | FP8-dynamic | ExpertsINT4 |
|---|
| German | 71.43 | 71.60 | 71.94 |
| Spanish | 79.59 | 79.59 | 77.55 |
| French | 79.59 | 79.25 | 79.42 |
| Japanese | 75.51 | 75.68 | 75.51 |
| Chinese |
Context retrieval robustness (RULER, 0-shot, 4K / 8K)
Measured separately, ExpertsINT4 against FP8-dynamic only, on 1xH100 with vLLM 0.28.0 and
lm_eval 0.4.12, limit=200 per cell.
Table with columns: Context, Task, ExpertsINT4, FP8-dynamic, Delta| Context | Task | ExpertsINT4 | FP8-dynamic | Delta |
|---|
| 4096 | ruler_cwe | 0.5085 | 0.4395 | +0.0690 |
| 4096 | ruler_fwe | 0.8867 | 0.9017 | -0.0150 |
| 4096 | ruler_qa_squad | 0.7600 | 0.7750 | -0.0150 |
| 8192 |
Results move in both directions relative to FP8, with no consistent degradation across the
evaluated 4K and 8K cells. Per-cell standard error is roughly +-0.035. These results evaluate
only 4K and 8K contexts and do not evaluate the model's full supported context window.
Thinking mode
Gemma 4 can also be served with thinking enabled. Measured that way the three checkpoints agree to
within about a point of each other, and accuracy is clearly higher than with thinking off. One
behaviour drives everything else: on some questions the model never closes its thought channel,
exhausts the token budget and returns no answer, which scores zero. Raising the cap from 16k to 32k
tokens does not remove it and the unquantized BF16 base is affected as much as the quantized
checkpoints, so each cell gives accuracy over the questions that produced an answer, with the share
that produced none beside it. Budgets were 8k-30k output tokens per task.
Table with columns: Benchmark, Base BF16, FP8-dynamic, ExpertsINT4| Benchmark | Base BF16 | FP8-dynamic | ExpertsINT4 |
|---|
| IFEval (0-shot, inst-level strict) | 96.93 (23.7% none) | 96.49 (28.1% none) | 96.29 (26.1% none) |
| MMLU-Pro (5-shot CoT) | 92.61 (41.9% none) | 93.41 (44.5% none) | 92.11 (39.1% none) |
| GPQA Diamond (0-shot CoT) | 88.66 (51.0% none) | 87.00 (49.5% none) | 86.24 (44.9% none) |
| BIG-Bench Hard (3-shot CoT) | 96.38 (31.3% none) | 96.70 (34.3% none) |
Scope
Except where the thinking-mode section says otherwise, every number on this card was produced with
thinking disabled, which is the mode we would recommend for throughput-sensitive serving. The
throughput and RULER measurements are thinking-disabled only.
Usage
from vllm import LLM, SamplingParams
llm = LLM(model="systalyze/gemma-4-26B-A4B-it-ExpertsINT4-FP8", max_model_len=12288)
out = llm.generate(
["Explain mixture-of-experts routing in two sentences."],
SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=256),
)
print(out[0].outputs[0].text)
Sampling defaults follow the base model's generation_config.json
(temperature=1.0, top_p=0.95, top_k=64).
License
Apache 2.0, inherited from
google/gemma-4-26B-A4B-it.