Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, o (42 layers) | bfloat8_b, 8.5 bits a weight |
| attention k, v (layers 0 to 23; the last 18 layers reuse earlier layers' KV) | bfloat8_b, 8.5 bits a weight |
| MLP gate, up, down (42 layers) | bfloat8_b, 8.5 bits a weight |
| LM head (tied to the embedding), token and per-layer embeddings, norms and every other tensor | bf16, as in the base model |
audio_tower, embed_audio, embed_vision, vision_tower | bf16, as in the base model |
The quantized matrices are stored as bf16 values on the BFP8 grid, so TT-NN packs them without changing their
values. quant4.json lists each projection's format. Quantized with GPTQ on the bfloat8_b grid, keeping each block's
shared exponent as the packer sets it, calibrated on 65,536 tokens of WikiText-2 and Korean Wikipedia. The token and
per-layer embeddings stay bf16, so the average over all language-model weights is above 8.5 bits. Only the language
model was quantized and evaluated; the vision and audio towers are the base model's.
Quickstart
import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizer repo = "Xylolabs/gemma-4-E4B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True).to(model.device)with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=32, do_sample=False)print(tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Run on Tensix
python pack_tensix.py gemma-4-E4B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8 packs every listed matrix with TT-NN and checks that no value changes (--device also
round-trips it through device DRAM); python verify_grid.py gemma-4-E4B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8 checks the grid with torch alone. Both were run on
ttsim with ttnn 0.81.1 and pass for every matrix.
Evaluation
google/gemma-4-E4B-it-qat-q4_0-unquantized
Table with columns: Weights, Formats per tensor class, Bits a weight, Size (GB), Mean KL to bf16, Top-1 (%), Answers right, Lost, Accuracy check| Weights | Formats per tensor class | Bits a weight | Size (GB) | Mean KL to bf16 | Top-1 (%) | Answers right | Lost | Accuracy check |
|---|
| bf16 (original) | every tensor bf16 | 16.00 | 15.04 | 0 | 100 | 38/40 | - | - |
| BFP8 | every linear layer (attention and MLP) bfloat8_b by GPTQ; head, embedding and norms bf16 | 12.09 | 11.279 | 0.0014 | 98.41 | 32/33 | 0 | passed (0.05× KL; lost 0 against 0) |
| Q4_0 (llama.cpp) | tensor types BF16 1, F32 321, Q4_0 342, Q6_K 2 (plain, no imatrix) | 5.51 | 5.155 | 0.0279 | 91.58 | 32/33 | 0 | reference |
All numbers are host evaluations of the weights, never device measurements; speed is not reported here. Mean KL is the KL divergence of each next-token distribution from the bf16 model's, averaged over the positions of fixed English and Korean texts; top-1 is how often both pick the same next token. The answer check is 33 fixed prompts with known answers plus 7 asked with thinking on (40 in all); an item is lost when the quantized model answers it wrong or broken where bf16 answers it right. The quantization accuracy check: a configuration passes when it loses at most one more item than llama.cpp's Q4_K_M of the same model (or the reference format named below) and its mean KL is at most 1.5 times that reference's on the same texts. Bits a weight and size count every weight read, embedding and head included. llama.cpp rows were made with llama.cpp's quantize tool from the same bf16 checkpoint. The accuracy check's reference for this model is Q4_0, the format its maker ships.
These numbers measure closeness to bf16, not task quality. Speed is not reported here.
License
Apache 2.0, as the base model's card says; its repository has no license file, so LICENSE holds the standard
Apache 2.0 text. The changes are listed in NOTICE. pack_tensix.py and verify_grid.py are
tensix-bfp-gptq v0.1.0's tools, unchanged, under Apache 2.0 (CODE_LICENSE). Provided as is, without
warranty. Not affiliated with or endorsed by Tenstorrent.
Citation
@misc{gemmateam2026gemma4, title={Gemma 4 Technical Report}, author={Gemma Team}, year={2026}, eprint={2607.02770}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2607.02770},}