Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, v, o (16 layers) | bfloat4_b, 4.5 bits a weight |
| Gated DeltaNet q, k, v, z, a, b, output (48 layers) | bfloat4_b, 4.5 bits a weight |
| MLP gate, up, down (64 layers) | bfloat4_b, 4.5 bits a weight |
| LM head | bfloat8_b, 8.5 bits a weight |
| embedding, norms and every other tensor | bf16, as in the base model |
mtp, visual | bf16, unchanged |
5.23 bits a weight on average, 17.60 GB packed. quant4.json lists each projection's format. The bfloat4_b
tensors were quantized with GPTQ and a per-block exponent search, calibrated on 65,536 tokens of WikiText-2
and Korean Wikipedia; the bfloat8_b tensors with round to nearest.
Only the language model was quantized and evaluated.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/Qwen3.8-27B-Tenstorrent-Tensix-BFP4"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run on Tensix
Checked with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator. Pack one matrix and
check that no value changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3.8-27B-Tenstorrent-Tensix-BFP4" # hf download Xylolabs/Qwen3.8-27B-Tenstorrent-Tensix-BFP4 --local-dir Qwen3.8-27B-Tenstorrent-Tensix-BFP4name = "model.language_model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py Qwen3.8-27B-Tenstorrent-Tensix-BFP4 checks every matrix this way (--device also round-trips it
through device DRAM). python verify_grid.py Qwen3.8-27B-Tenstorrent-Tensix-BFP4 checks the grid with torch alone. The
same values also pack exactly into ttnn.bfloat8_b.
tt-metal qwen38_27b_qb2 demo (v0.81.1). tt-transformers does not support this architecture; tt-metal's
qwen38_27b_qb2 demo runs
it tensor-parallel on four Blackhole chips (QuietBox 2) and refuses other hardware. Set
QWEN_PRECISION_CONFIG=baseline, which matches this checkpoint; the shipped config/precision.json would
re-quantize the head to bfloat4_b. Its decoder packed every projection of layers 0 and 3 exactly; the head and
a full run need the four chips and were not tested.
# build tt-metal v0.81.1 as the demo's README describes, thenexport MODEL_WEIGHTS_DIR=/path/to/Qwen3.8-27B-Tenstorrent-Tensix-BFP4export QWEN_PRECISION_CONFIG=baselinepython models/demos/qwen38_27b_qb2/demo/text_demo.py --full --length 128 --generate 128 --output qwen38.json
tt-forge (tt-xla). Not run with this checkpoint yet.
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (Qwen/Qwen3.8-27B) | 16 | 53.79 | 0 | 100 | 40 of 40 | - |
| this model | 5.23 | 17.596 | 0.02783 | 93.19 | 38 of 40 | 0 |
| Q4_K_M (llama.cpp) | 4.92 | 16.811 | 0.02084 | 93.50 | 40 of 40 | 0 |
KL and top-1 cover 4,094 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), as plain
text. Answers are 33 short prompts with automatic checks, plus the 7 reasoning ones again with thinking on (40
in all). "Lost" means right in BF16 but wrong or broken here; near ties are not counted. The two reasoning
prompts it misses with thinking off come out right with thinking on, so neither counts as lost. Q4_K_M was
made from the same checkpoint with llama.cpp v0.9.0 (cd959bae) and no importance matrix; its sha256 is
66927ba20072bb323235927c4d1b54b860dce33ace2b801982a9febddb46c07a. Our bar: mean KL at most 1.5 times
Q4_K_M's and at most one more lost answer. Passed: 1.34 times Q4_K_M's KL, 0 lost against 0.
These numbers measure closeness to BF16, not task quality. Speed is not reported here.
License
Apache 2.0, as the base model (LICENSE). The changes are listed in NOTICE. pack_tensix.py and
verify_grid.py are by Xylolabs under the same license. Provided as is, without warranty. Not affiliated with
or endorsed by Tenstorrent.
Citation
For Qwen family background, cite the Qwen3 technical report:
@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}, eprint={2505.09388}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.09388}}