Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, v, o and MLP gate, up, down (36 layers) | bfloat4_b, 4.5 bits a weight |
| LM head (tied to the embedding), embedding, norms and every other tensor | bf16, as in the base model |
5.61 bits a weight on average, 2.82 GB packed. quant4.json lists each projection's format. Quantized with
GPTQ and a per-block exponent search, calibrated on 65,536 tokens of WikiText-2 and Korean Wikipedia.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/Qwen3-4B-Thinking-2507-Tenstorrent-Tensix-BFP4"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=512)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run on Tensix
Checked with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator. Pack one matrix and check that no value
changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3-4B-Thinking-2507-Tenstorrent-Tensix-BFP4" # hf download Xylolabs/Qwen3-4B-Thinking-2507-Tenstorrent-Tensix-BFP4 --local-dir Qwen3-4B-Thinking-2507-Tenstorrent-Tensix-BFP4name = "model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py Qwen3-4B-Thinking-2507-Tenstorrent-Tensix-BFP4 checks every matrix this way (--device also
round-trips it through device DRAM). python verify_grid.py Qwen3-4B-Thinking-2507-Tenstorrent-Tensix-BFP4 checks the
grid with torch alone. The same values also pack exactly into ttnn.bfloat8_b.
tt-transformers and tt-forge were not run with this checkpoint.
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (Qwen/Qwen3-4B-Thinking-2507) | 16 | 8.04 | 0 | 100 | 31 of 33 | - |
| this model | 5.61 | 2.822 | 0.04599 | 90.95 | 32 of 33 | 1 |
| Q4_K_M (llama.cpp) | 4.955 | 2.497 | 0.04862 | 90.64 | 29 of 33 | 1 |
KL and top-1 cover 16,256 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), plain and
in the chat template. Answers are 33 short prompts with automatic checks. "Lost" means right in BF16 but wrong
or broken here; near ties are not counted. Q4_K_M was made from the same checkpoint with llama.cpp v0.9.0
(cd959bae) and no importance matrix; its sha256 is
e424bbb9045b10ca1eca4ffc0ad547805c12467f7276b53579d89ebbb022d2c1. Our bar: mean KL at most 1.5 times
Q4_K_M's and at most one more lost answer. Passed: 0.946 times Q4_K_M's KL, 1 lost against 1.
These numbers measure closeness to BF16, not task quality. Speed is not reported here.
License
Apache 2.0, as the base model (LICENSE). The changes are listed in NOTICE. pack_tensix.py and
verify_grid.py are by Xylolabs under the same license. Provided as is, without warranty. Not affiliated with
or endorsed by Tenstorrent.
Citation
@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}, eprint={2505.09388}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.09388},}