Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, v, o and MLP gate, up, down (36 layers) | bfloat8_b, 8.5 bits a weight |
| LM head, embedding, norms and every other tensor | bf16, as in the base model |
9.64 bits a weight on average, 9.87 GB packed. quant4.json lists each projection's format. Quantized with
GPTQ on the bfloat8_b grid, keeping each block's shared exponent as the packer sets it, calibrated on 65,536
tokens of WikiText-2 and Korean Wikipedia.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/Qwen3-8B-Tenstorrent-Tensix-BFP8"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run on Tensix
Checked with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator. Pack one matrix and
check that no value changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3-8B-Tenstorrent-Tensix-BFP8" # hf download Xylolabs/Qwen3-8B-Tenstorrent-Tensix-BFP8 --local-dir Qwen3-8B-Tenstorrent-Tensix-BFP8name = "model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py Qwen3-8B-Tenstorrent-Tensix-BFP8 checks every matrix this way (--device also round-trips it
through device DRAM). python verify_grid.py Qwen3-8B-Tenstorrent-Tensix-BFP8 checks the grid with torch alone.
tt-transformers (tt-metal v0.81.1). Pass --use_hf_rope; without it the demo permutes the q and k rows
and re-quantizes the fused QKV. With these settings every projection of the first 2 layers packed exactly, and
a 2-layer prefill matched transformers (PCC 0.9990). The demo packs the head in bfloat8_b, so this
checkpoint's bf16 head is not exact there.
git clone --branch v0.81.1 https://github.com/tenstorrent/tt-metal.git && cd tt-metal# install tt-metal v0.81.1 as its INSTALLING.md describes, thenpip install -r tt_metal/python_env/requirements-dev.txtexport HF_MODEL=Xylolabs/Qwen3-8B-Tenstorrent-Tensix-BFP8export TT_CACHE_PATH=$PWD/model_cache/Qwen3-8B-Tenstorrent-Tensix-BFP8pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1" --use_hf_rope \ --optimizations 'precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}'
tt-forge (tt-xla). Not run with this checkpoint yet. By default it keeps the weights in bf16, which is
exact. Its bfp_bf8 and bfp_bf4 options re-quantize them in PyTorch's [out, in] layout.
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (Qwen/Qwen3-8B) | 16 | 16.38 | 0 | 100 | 38 of 40 | - |
| this model | 9.64 | 9.870 | 0.00100 | 98.68 | 38 of 40 | 0 |
| Q4_K_M (llama.cpp) | 4.90 | 5.028 | 0.04236 | 91.24 | 38 of 40 | 0 |
KL and top-1 cover 16,240 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), plain and
in the chat template. Answers are 33 short prompts with automatic checks, plus the 7 reasoning ones again with
thinking on (40 in all). "Lost" means right in BF16 but wrong or broken here; near ties are not counted.
Q4_K_M was made from the same checkpoint with llama.cpp v0.9.0 (cd959bae) and no importance matrix; its
sha256 is 6fbd1680a62b31c652eae1b151ffbfe00c4e64d303fcf7e089566f5116a10cb7. Our bar: mean KL at most 1.5
times Q4_K_M's and at most one more lost answer. Passed: 0.02 times Q4_K_M's KL, 0 lost against 0.
These numbers measure closeness to BF16, not task quality. Speed is not reported here.
License
Apache 2.0, as the base model (LICENSE). The changes are listed in NOTICE. pack_tensix.py and
verify_grid.py are by Xylolabs under the same license. Provided as is, without warranty. Not affiliated with
or endorsed by Tenstorrent.
Citation
@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}, eprint={2505.09388}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.09388},}