Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, o (35 layers) | bfloat8_b, 8.5 bits a weight |
| attention k, v (15 layers) | bfloat8_b, 8.5 bits a weight |
| MLP gate, up, down (35 layers) | bfloat4_b, 4.5 bits a weight |
| LM head (tied to the embedding), embedding, norms and every other tensor | bf16, as in the base model |
audio_tower, embed_audio, embed_vision, vision_tower | bf16, unchanged |
11.68 bits a weight on average, 6.76 GB packed. quant4.json lists each projection's format. The bfloat4_b
tensors were quantized with GPTQ and a per-block exponent search, calibrated on 65,536 tokens of WikiText-2
and Korean Wikipedia; the bfloat8_b tensors with round to nearest.
The token and per-layer embeddings stay bf16 (16 bits a weight); the MLP projections use 4.5 bits a weight and the attention projections use 8.5, so the average over all language-model weights is higher.
Only the language model was quantized and evaluated.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/gemma-4-E2B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP4-MLP"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run on Tensix
Checked with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator. Pack one matrix and
check that no value changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "gemma-4-E2B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP4-MLP" # hf download Xylolabs/gemma-4-E2B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP4-MLP --local-dir gemma-4-E2B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP4-MLPname = "model.language_model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py gemma-4-E2B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP4-MLP checks every matrix this way
(--device also round-trips it through device DRAM).
python verify_grid.py gemma-4-E2B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP4-MLP checks the grid with torch alone.
The same values also pack exactly into ttnn.bfloat8_b.
tt-metal Gemma 4 demo (v0.81.1). tt-transformers does not support Gemma 4; tt-metal's Gemma 4
demo runs E2B, E4B and 12B on one
Blackhole P150 and has no bfloat4_b option. With no entry for this repository in its
precision_overrides.json, every module loads in bf16, which is exact. This entry packs attention and MLP in
bfloat8_b, which was exact on the first 2 layers of the base E2B export and was not run on this one:
"gemma-4-E2B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP4-MLP": {"default": {"attention": "bfp8", "shared_mlp": "bfp8"}}
tt-forge (tt-xla). Not run with Gemma 4 yet. By default it keeps the weights in bf16, which is exact; its
bfp_bf4 option re-quantizes them in PyTorch's [out, in] layout.
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (google/gemma-4-E2B-it-qat-q4_0-unquantized) | 16 | 9.26 | 0 | 100 | 31 of 33 | - |
| this model | 11.68 | 6.758 | 0.06007 | 88.34 | 31 of 33 | 0 |
| Q4_0 (llama.cpp) | 5.76 | 3.350 | 0.04435 | 89.47 | 32 of 33 | 0 |
KL and top-1 cover 7,968 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), in the chat
template. Answers are 33 short prompts with automatic checks. "Lost" means right in BF16 but wrong or broken
here; near ties are not counted. The 7 thinking-on prompts were not run for this export. Q4_0 was made from
the same checkpoint with llama.cpp v0.9.0 (cd959bae) and no importance matrix; its sha256 is
7976063c67c363ed292081c862051cd7b1f7caf529d972e42aa2e2fe098fd63e. Q4_0 is the reference because Google ships
this QAT checkpoint as Q4_0. Our bar: mean KL at most 1.5 times Q4_0's and at most one more lost answer.
Passed: 1.35 times Q4_0's KL, 0 lost against 0.
These numbers measure closeness to BF16, not task quality. Speed is not reported here.
License
Apache 2.0, as the base model's card says; its repository has no LICENSE file, so LICENSE holds the standard
text. The changes are listed in NOTICE. pack_tensix.py and verify_grid.py are by Xylolabs under the same
license. Provided as is, without warranty. Not affiliated with or endorsed by Tenstorrent.
Citation
@misc{gemmateam2026gemma4, title={Gemma 4 Technical Report}, author={Gemma Team}, year={2026}, eprint={2607.02770}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2607.02770},}