Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, v, o and MLP gate, up, down (36 layers) | bfloat4_b, 4.5 bits a weight |
| LM head, embedding, norms and every other tensor | bf16, as in the base model |
6.25 bits a weight on average, 6.40 GB packed. quant4.json lists each projection's format. Quantized with
GPTQ and a per-block exponent search, calibrated on 65,536 tokens of WikiText-2 and Korean Wikipedia.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/Qwen3-8B-Tensix-BFP4"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run on Tensix
Tested with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator; not on hardware. Pack one matrix and
check that no value changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3-8B-Tensix-BFP4" # hf download Xylolabs/Qwen3-8B-Tensix-BFP4 --local-dir Qwen3-8B-Tensix-BFP4name = "model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py Qwen3-8B-Tensix-BFP4 checks every matrix this way (--device also round-trips it
through device DRAM). python verify_grid.py Qwen3-8B-Tensix-BFP4 checks the grid with torch alone. The same
values also pack exactly into ttnn.bfloat8_b.
tt-transformers (tt-metal v0.81.1). Pass --use_hf_rope; without it the fused QKV is re-quantized. The
demo packs the head in bfloat8_b, so the head is not exact there.
git clone --branch v0.81.1 https://github.com/tenstorrent/tt-metal.git && cd tt-metal# install tt-metal v0.81.1 as its INSTALLING.md describes, thenpip install -r tt_metal/python_env/requirements-dev.txtexport HF_MODEL=Xylolabs/Qwen3-8B-Tensix-BFP4export TT_CACHE_PATH=$PWD/model_cache/Qwen3-8B-Tensix-BFP4pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1" --use_hf_rope \ --optimizations 'precision_cfg = {ff1_3: bfp4, ff2: bfp4, wqkv: bfp4, wo: bfp4, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}'
tt-forge (tt-xla). It runs the weights in bf16 by default, which is exact. Its bfp_bf4 option
re-quantizes them in PyTorch's [out, in] layout, so use tt-transformers for 4.5-bit weights. Not yet run end
to end.
pip install pjrt-plugin-tt==1.4.0 --extra-index-url https://pypi.eng.aws.tenstorrent.com/tt-forge-installpython - <<'PY'import torch, torch_xla, torch_xla.runtime as xrfrom transformers import AutoModelForCausalLM, AutoTokenizerxr.set_device_type("TT")device = torch_xla.device()tok = AutoTokenizer.from_pretrained("Xylolabs/Qwen3-8B-Tensix-BFP4")model = AutoModelForCausalLM.from_pretrained("Xylolabs/Qwen3-8B-Tensix-BFP4", torch_dtype=torch.bfloat16).eval().to(device)compiled = torch.compile(model, backend="tt")ids = tok("The capital of France is", return_tensors="pt").input_ids.to(device)with torch.no_grad(): logits = compiled(input_ids=ids).logits[0, -1].cpu()print(tok.decode(logits.argmax()))PY
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (Qwen/Qwen3-8B) | 16 | 16.38 | 0 | 100 | 38 of 40 | - |
| this model | 6.25 | 6.397 | 0.0449 | 91.26 | 38 of 40 | 0 |
| Q4_K_M (llama.cpp) | 4.90 | 5.028 | 0.04236 | 91.24 | 38 of 40 | 0 |
KL and top-1 cover 16,240 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), plain and
in the chat template. Answers are 33 short prompts with automatic checks, plus the 7 reasoning ones again with
thinking on (40 in all). "Lost" means right in BF16 but wrong or broken here; near ties are not counted.
Q4_K_M was made from the same checkpoint with llama.cpp v0.9.0 (cd959bae) and no importance matrix; its
sha256 is 6fbd1680a62b31c652eae1b151ffbfe00c4e64d303fcf7e089566f5116a10cb7. Our bar: mean KL at most 1.5
times Q4_K_M's and at most one more lost answer. Passed: 1.06 times Q4_K_M's KL, 0 lost against 0.
These numbers measure closeness to BF16, not task quality; no device speed was measured. For the model itself,
see the base card.
License
Apache 2.0, as the base model (LICENSE). The changes are listed in NOTICE. pack_tensix.py and
verify_grid.py are by Xylolabs under the same license. Provided as is, without warranty. Not affiliated with
or endorsed by Tenstorrent.
Citation
@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}, eprint={2505.09388}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.09388},}