GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
xbill9
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
xbill9
Model Tree
google/gemma-4-E2B-it-qat-q4_0-unquantized
Input Modalities
Output Modalities
Supported Functionality
google/gemma-4-E2B-it-qat-w4a16-ctGoogle publishes its own W4A16 QAT checkpoint for E2B. Two differences:
-q4_0-unquantized export exactly. Google's W4A16 export records llm-compressor's memoryless_minmax observer in its quantization_config: the QAT weights were re-quantized by round-to-nearest with each group's scale at max|w| / 7.5, which moves every weight to a shifted grid (measured at 6.67% relative error on the 12B and 31B exports; not measured tensor by tensor on E2B).lm_head.weight in bf16 (0.75 GiB), a byte-for-byte copy of the token embedding although the config ties them (tie_word_embeddings: true). This repack does not, so it is 0.75 GiB smaller on disk and on the chip.On a 3,880-record public suite read by label probability, paired record for record:
| E2B build | Where | Suite | This repack, difference (95% range) |
|---|---|---|---|
google/gemma-4-E2B-it bf16 | the same v5e chip | 68.5% | −0.6 points (−1.6 to +0.3) |
google/gemma-4-E2B-it-qat-w4a16-ct | the same v5e chip, same flags | 65.5% | +2.4 points (+1.4 to +3.4) |
| This repack | one v5e chip | 67.8% |
On one TPU v5e chip (v5litepod-1, 15.75 GiB HBM), vLLM TPU vllm/vllm-tpu@sha256:19a1a052… with the patches below:
| This repack | google/gemma-4-E2B-it-qat-w4a16-ct | |
|---|---|---|
| Weights resident | 6.42 GiB | 7.17 GiB |
| First-token latency, one request, median | 16.2 ms | 16.0 ms |
| Output tokens/s at 1 / 4 / 16 concurrent requests (256 tokens each) | 136.3 / 531.6 / 1,906.2 | not measured |
Logs, per-record outputs and scripts: https://github.com/xbill9/gemma4-dev/tree/main/jev-tpu-v5e1
pack-quantized, symmetric int4, group size 32, bf16 scales, activations unquantized (W4A16).per_layer_input_gate, per_layer_projection, per_layer_model_projection), which QAT also puts on the grid. This is the same set Google's export quantizes.The -qat-q4_0-unquantized export stores bf16 values that already lie on a 4-bit grid: within every group of 32 weights along the input dimension, each weight is step × level with level from −8 to 7. The repack recovers each group's step and writes the levels as packed int4. It does no new quantization. repack_q4_0.py finds, for each group, the step max|w| / m (m from 1 to 8) that reproduces all 32 values, refines it by least squares and stores it at bf16.
repack_q4_0.py verify rereads both checkpoints and checks every group: 58,650,624 groups of 32, none off the source grid, 90.0% of values bit-identical; the rest differ through the bf16 scale, by at most 1.09e-2 relative. All 1,675 unquantized tensors are byte-identical to the source, and every linear left bf16 is on the ignore list. repack_report.json and verify_report.json in this repository are that run's outputs.
Google Cloud TPU. vLLM's TPU backend needs tpu-inference #3653 (W4A16 linear layers) and #3299 (the KV-shared layers of the E models own no K/V parameters). On one v5e chip, leave HBM for the compiled model:
bash
vllm serve <this-repo> --max-model-len 2048 --gpu-memory-utilization 0.80 \ --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}'NVIDIA GPUs. Untested with this checkpoint.
Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed container format, under the same license. The weights, training and the original model card are Google DeepMind's; the repack and this card are not affiliated with or endorsed by Google.
vllm serve <this-repo> --max-model-len 2048 --gpu-memory-utilization 0.80 \ --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}'