Table with columns: Tensors, Format| Tensors | Format |
|---|
| 328 declared decoder projections | bfloat8_b, 8.5 bits per weight; GPTQ |
| Embedding, explicit tied-head copy and all other stored tensors | BF16 (350 entries) |
All stored entries pack to about 15.71 GB (9.70 bits per stored weight); sharing the tied head once gives
about 13.70 GB (9.165 bits per unique weight). Both explicit BF16 embedding/head copies are byte-identical.
The embedding stays BF16, while quantized projections average 8.5 bits per weight. The memory figure above
conservatively includes both stored copies.
quant4.json records each projection format. GPTQ uses damping 0.1 and 65,536 WikiText-2 and Korean Wikipedia
tokens. BF16 grid storage is larger than the packed representation.
Quickstart
Use Transformers 5.23.0 with PyTorch. The original tokenizer and chat template are retained.
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/gemma-4-12B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")messages = [{"role": "user", "content": "What is the capital of France? Reply with the city name only."}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The example uses text only; image and audio execution are not qualified here.
Run on Tensix
All 328 declared matrices pass the grid and native TT-NN packing checks (the same format as packing as BFP8).
The exact example also passes on the Blackhole simulator, checked with TT-NN 0.81.1. After downloading this
repository locally, pack one matrix in TT-NN's [in, out] layout:
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "gemma-4-12B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8" # hf download Xylolabs/gemma-4-12B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8 --local-dir gemma-4-12B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8name = "model.language_model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous()device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py gemma-4-12B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8 checks every declared quantized
matrix; --device adds a device DRAM round trip.
python verify_grid.py gemma-4-12B-it-qat-q4_0-unquantized-Tenstorrent-Tensix-BFP8 checks the grid using torch alone.
These are representation checks; tt-transformers and tt-forge were not checked with this checkpoint.
Evaluation
Table with columns: Profile, Mean KL, Normal right / 33, Thinking right / 7, Counted normal losses| Profile | Mean KL | Normal right / 33 | Thinking right / 7 | Counted normal losses |
|---|
| Original BF16 | 0 | 33 | 7 | — |
| This export | 0.001488625 | 33 | 7 | 0 |
| Matched Q4_0 | 0.032543125 | 33 | 7 | 0 |
The quantization accuracy check passes: 0.045743 times the reference KL, within the 1.5-times limit, with no more
than one additional counted loss.
KL covers 8,048 scored positions on the same saved English/Korean chat texts only, four chunks of each. The
selected-text llama.cpp BF16 floor is 0.000184375. Every loaded text-model tensor is checked against the
indexed export values. Answer counts and the original thinking runs come from the retained exact-weight
checks, rescored with the canonical CLI. These are host checks of the saved weights. Speed is not reported
here.
The reference is Q4_0 because the original model was trained for that format. The Gemma instruct comparison
selects chat texts; the full raw-plus-chat file mean is not substituted for this bar.
Source: google/gemma-4-12B-it-qat-q4_0-unquantized at b6ed86275a6a5735884e208bfed95b445a684ca2.
License
Apache 2.0, as declared by the original model (LICENSE). Changes are documented in NOTICE; original
Xylolabs generic packing helpers use Apache 2.0 (CODE_LICENSE). Provided as is, without warranty. Not
affiliated with or endorsed by Tenstorrent. pack_tensix.py and verify_grid.py are tensix-bfp-gptq v0.1.0's tools, unchanged.
Citation
Cite the original google/gemma-4-12B-it-qat-q4_0-unquantized
release.