Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, o (30 layers) | bfloat8_b, 8.5 bits a weight |
| attention v (25 layers) | bfloat8_b, 8.5 bits a weight |
| MLP gate, up, down and routed experts down, gate, up (30 layers) | bfloat4_b, 4.5 bits a weight |
| router per-expert scale and router (30 layers) | bf16 |
| LM head (tied to the embedding), embedding, norms and every other tensor | bf16, as in the base model |
embed_vision, vision_tower | bf16, unchanged |
5.02 bits a weight on average, 15.83 GB packed. quant4.json lists each projection's format. The bfloat4_b
tensors were quantized with GPTQ and a per-block exponent search, calibrated on 65,536 tokens of WikiText-2
and Korean Wikipedia; the bfloat8_b tensors with round to nearest. The routed experts' gate and up share one
Hessian per layer; each expert's down takes its own, or uses round to nearest when it saw too few calibration
tokens.
Only the language model was quantized and evaluated.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/gemma-4-26B-A4B-it-qat-q4_0-unquantized-Tensix-BFP4-MLP"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run on Tensix
Tested with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator; not on hardware. Pack one matrix and
check that no value changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "gemma-4-26B-A4B-it-qat-q4_0-unquantized-Tensix-BFP4-MLP" # hf download Xylolabs/gemma-4-26B-A4B-it-qat-q4_0-unquantized-Tensix-BFP4-MLP --local-dir gemma-4-26B-A4B-it-qat-q4_0-unquantized-Tensix-BFP4-MLPname = "model.language_model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py gemma-4-26B-A4B-it-qat-q4_0-unquantized-Tensix-BFP4-MLP checks every matrix this way
(--device also round-trips it through device DRAM).
python verify_grid.py gemma-4-26B-A4B-it-qat-q4_0-unquantized-Tensix-BFP4-MLP checks the grid with torch
alone. The same values also pack exactly into ttnn.bfloat8_b.
tt-transformers and tt-forge were not run with this checkpoint.
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (google/gemma-4-26B-A4B-it-qat-q4_0-unquantized) | 16 | 50.47 | 0 | 100 | 33 of 33 | - |
| this model | 5.02 | 15.826 | 0.06233 | 90.87 | 33 of 33 | 0 |
| Q4_0 (llama.cpp) | 4.57 | 14.439 | 0.05293 | 91.55 | 33 of 33 | 0 |
KL and top-1 cover 8,048 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), in the chat
template. Answers are 33 short prompts with automatic checks. "Lost" means right in BF16 but wrong or broken
here; near ties are not counted. Q4_0 was made from the same checkpoint with llama.cpp v0.9.0 (cd959bae) and
no importance matrix; its sha256 is 0f70faabfedac04c31f7ef661601c3fca4fc80069cef7e9eaf509b4b0e1e3c85. Q4_0
is the reference because Google ships this QAT checkpoint as Q4_0. Our bar: mean KL at most 1.5 times Q4_0's
and at most one more lost answer. Passed: 1.18 times Q4_0's KL, 0 lost against 0.
These numbers measure closeness to BF16, not task quality; no device speed was measured. For the model itself,
see the base card.
License
Apache 2.0, as the base model's card says; its repository has no LICENSE file, so LICENSE holds the standard
text. The changes are listed in NOTICE. pack_tensix.py and verify_grid.py are by Xylolabs under the same
license. Provided as is, without warranty. Not affiliated with or endorsed by Tenstorrent.
Citation
@misc{gemmateam2026gemma4, title={Gemma 4 Technical Report}, author={Gemma Team}, year={2026}, eprint={2607.02770}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2607.02770},}