Overview
Qwen/Qwen3-4B is a 4-billion-parameter dense language model of the Qwen3 series by the Qwen team at Alibaba Cloud, with a thinking and a non-thinking mode. This repository holds that model with
its linear projections rounded to bfloat4_b (BFP4) by GPTQ with an exponent search, made by Xylolabs.
The files store bf16 values that lie exactly on each matrix's block floating-point grid. In
transformers the checkpoint loads like the base model and computes what the rounded model computes;
a TT-NN packer stores the same values without rounding them again. quant4.json lists every
projection's format per layer.
Intended hardware
These formats run on Tenstorrent's Tensix Processors, such as Wormhole™ and Blackhole®. A weight in
bfloat4_b (BFP4) takes 4.5 bits against 8.5 bits in bfloat8_b (BFP8), so a matrix in BFP4 streams
from memory at about half the bytes of the same matrix in BFP8. We have not benchmarked these exports
on Tenstorrent's own cards.
Not affiliated with or endorsed by Tenstorrent.
Table with columns: Tensors, Format, Bits a weight| Tensors | Format | Bits a weight |
|---|
| attention q, k, v, o (36 layers) | bfloat4_b (BFP4) | 4.5 |
| MLP gate and up (36 layers) | bfloat4_b (BFP4) | 4.5 |
| MLP down (36 layers) | bfloat4_b (BFP4) | 4.5 |
| LM head (tied to the embedding) | bf16 | 16 |
| embedding, norms, biases and every other tensor of the language model | bf16, as in the base model | 16 |
bfloat4_b (BFP4) and bfloat8_b (BFP8) are TT-NN's block floating-point formats: in tile layout, 16
consecutive values share one 8-bit exponent, and each value keeps a sign and a 3-bit (BFP4) or 7-bit
(BFP8) mantissa. Over every weight read, embedding and head included, the model averages 5.61
bits a weight: 2.822 GB packed in these formats (8.8 GB as the bf16 files here).
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "hletrd/Qwen3-4B-Tensix-BFP4"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Running on Tenstorrent Tensix
These steps use only Tenstorrent's public software, tt-metal and TT-NN
(docs).
- Install TT-NN by tt-metal's installation guide
(the Python package is
ttnn).
- Download this repository:
hf download hletrd/Qwen3-4B-Tensix-BFP4 --local-dir Qwen3-4B-Tensix-BFP4.
- Pack a matrix onto the device and check that no value changed. TT-NN's matmul reads a weight as
[in, out], the transpose of the PyTorch layout, and the block grid of this export is laid out that way:
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3-4B-Tensix-BFP4" # the downloaded repositoryname = "model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"] # "bfp4"dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() # [in, out]: the layout TT-NN's matmul reads device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float()) # packing changed no valuettnn.close_device(device)
- Every rounded matrix at once:
python pack_tensix.py Qwen3-4B-Tensix-BFP4 packs each one with the format
quant4.json names and checks the unpacked values against the stored ones; --device also sends
each tensor to device DRAM and back, and --cache-dir DIR writes TT-NN weight cache files that
ttnn.as_tensor(..., cache_file_name=...) loads later without packing again.
- Without any Tenstorrent software:
python verify_grid.py Qwen3-4B-Tensix-BFP4 re-rounds every listed matrix with
its block's exponent and reports any value that would change (needs torch and safetensors only).
Tested with ttnn 0.81.1 (PyPI) with torch 2.15.0, on ttsim, Tenstorrent's functional simulator of a Blackhole chip: TT-NN packed all 252 rounded matrices to ttnn.bfloat4_b and unpacked every value bit for bit; layers 0 and 35 also went through device DRAM and back unchanged; the snippet above ran as written; on random matrices in both formats, verify_grid.py's rounding gives exactly the values TT-NN's packer stores. Not run on Tenstorrent hardware.
The same values packed into ttnn.bfloat8_b instead also come back unchanged: a block keeps the same shared exponent, and a 3-bit mantissa step is 16 of the 7-bit ones (16 × 7 = 112 ≤ 127). A bfloat8_b setting therefore holds exactly this model's weights too, at 8.5 bits a weight (checked on ttsim: all 252 matrices packed as ttnn.bfloat8_b came back unchanged).
tt-transformers (models/tt_transformers in tt-metal) loads these bf16 safetensors and packs each matrix at load time into the precision its setting names (--optimizations precision_cfg, or its performance / accuracy defaults). Every value of this checkpoint already lies on the bfloat4_b grid, so the packing is exact when each matrix keeps its stored layout. Pass --use_hf_rope: it is required for exact weights.
Table with columns: Setting, Decoder matrices| Setting | Decoder matrices |
|---|
--use_hf_rope with precision_cfg = {ff1_3: bfp4, ff2: bfp4, wqkv: bfp4, wo: bfp4, ...} | every matrix packs to bfloat4_b bit for bit |
--use_hf_rope with bfloat8_b attention and FF2 (as in the performance default) | every matrix packs bit for bit; attention and FF2 take 8.5 bits a weight |
without --use_hf_rope (the default) | the fused QKV is re-rounded: tt-transformers reorders the q and k rows for its Meta-style RoPE, so 5.5-6.3 % of the QKV values change, by up to 0.031 |
The head is tied to the embedding and stored in bf16. tt-transformers' demo packs the head in bfloat8_b (the model-level dtype; there is no command-line option for it) and keeps the embedding in bf16, so the head is not bit-exact there: it computes with a bfloat8_b rounding of this model's head.
git clone --branch v0.81.1 https://github.com/tenstorrent/tt-metal.git && cd tt-metal# install tt-metal v0.81.1 as its INSTALLING.md describes, thenpip install -r tt_metal/python_env/requirements-dev.txtexport HF_MODEL=hletrd/Qwen3-4B-Tensix-BFP4 # or a local directory with these filesexport TT_CACHE_PATH=$PWD/model_cache/Qwen3-4B-Tensix-BFP4pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1" --use_hf_rope \ --optimizations 'precision_cfg = {ff1_3: bfp4, ff2: bfp4, wqkv: bfp4, wo: bfp4, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}'
In tt-metal v0.81.1, --optimizations needs fidelity_cfg as well as precision_cfg: with precision_cfg alone, parse_optimizations fails. The fidelity_cfg above is the README's example.
What was checked: tt-metal v0.81.1 (commit d80d9482), ttnn 0.81.1, transformers 5.12.1, Python 3.12, on Tenstorrent's Blackhole simulator ttsim v1.15.1 (one P150-class chip). The model was cut to its first 2 of 36 layers on the simulator, and nothing here was run on Tenstorrent hardware. The runs covered the load, the packing and a 32-token prefill through tt-transformers' own model code; the demo command above was tried on the same 2-layer cut, where it loaded and packed the model but did not finish generating.
- Packing: every matrix tt-transformers handed to ttnn as bfloat4_b or bfloat8_b was packed and unpacked again, and compared bit for bit.
- Prefill: against Hugging Face transformers on the CPU (same 2 layers, float32), with the head kept in bf16. The last token's logits had PCC 0.99922 with
--use_hf_rope and 0.99899 without; the same top-1 token and 5 of 5 top-5 tokens either way.
With tt-forge
Not covered yet: tt-forge (tt-xla) is being tested on this model, and this card will say how to run it once that is verified.
How it was made
- GPTQ (Frantar et al., 2022) per projection: input channels in decreasing order of the Hessian diagonal, blocks of 128 columns, damping 0.1 of the mean diagonal. For bfloat4_b and bfloat8_b a column's blocks are its groups of 16 output channels, and each block's exponent is chosen from the updated column exactly as TT-NN's packer will choose it.
- Exponent search: for each block, the exponent of its largest value and one lower are both tried (one lower clips the largest values and halves the step for the rest); the one with the smaller error is kept. Every value it produces is exactly representable in the format.
- Calibration: 65,536 tokens of WikiText-2 (train) and Korean Wikipedia articles, run through the bf16 model; none of it is in the evaluation texts below.
- Grid check: every rounded matrix re-rounded with its block's shared exponent gives back its
stored values (252 of 252 matrices, by
verify_grid.py and by our pipeline's own check).
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 agreement (%), Answers right, Answers lost against BF16 | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 agreement (%) | Answers right | Answers lost against BF16 |
|---|
BF16 original (Qwen/Qwen3-4B at 1cfa9a720891) | 16 | 8.04 | 0 | 100 | 36 of 40 | - |
| this model | 5.61 | 2.822 | 0.0711 | 89.30 | 36 of 40 | 1 |
| Q4_K_M (llama.cpp) | 4.95 | 2.497 | 0.06581 | 89.61 | 38 of 40 | 0 |
Size: every weight read, embedding and head included, as each format stores it (this model packed in
its block floating-point formats; Q4_K_M as its GGUF tensors).
The evaluation set.
- KL and top-1: four texts: English and Korean, each as plain text and as the reply of a chat turn in the model's template, with thinking off; 16240 scored positions in all, chunks of 1,024 tokens. The English
text is WikiText-2's test split and the Korean one Korean Wikipedia articles, neither in the
calibration data. Mean KL is KL(BF16 ‖ model) over the full vocabulary at every scored position;
top-1 agreement is how often the model's most likely next token is BF16's. BF16 is the original
checkpoint run in
transformers.
- Answers: 33 short prompts in English and Korean (arithmetic, facts, short reasoning,
instructions, translation, two small functions), each with an automatic checker, answered greedily
with thinking off; and the 7 reasoning items again with thinking on, as users run the model.
- A lost answer is one BF16 gets right and the model gets wrong or broken (empty, garbled, a loop,
the wrong language), counted only where BF16 was clear at the first token where the two parted (top-2
logit gap at least 1.0; closer calls are near ties, listed and not counted). A reasoning item lost
with thinking off is cleared when the model answers it right with thinking on.
Q4_K_M, where it comes from. Made by us from the same BF16 checkpoint (Qwen/Qwen3-4B at 1cfa9a720891): llama.cpp's convert_hf_to_gguf.py --outtype bf16, then llama-quantize Q4_K_M with no importance matrix, llama.cpp v0.9.0 (commit cd959bae99ff4bed22bef2604f45241c09bbb8c8); the GGUF's sha256 is f40a41777de3c1aaf31470ca19e004cc6579dfb57043b4f0e58a3701393d35b8. Its answers come from greedy runs on the same token ids, its KL from llama.cpp's log-softmax at every scored position. Scored with the same texts, positions and scorer.
The floor: llama.cpp's own BF16 GGUF of the same checkpoint sits at a mean KL of 0.00032 from our BF16 pass, under 0.01, so both sides are measured against the same BF16 pass.
Our bar: mean KL at most 1.5 times Q4_K_M's and at most one more lost answer than Q4_K_M. This model: passed (1.08 times Q4_K_M's mean KL; 1 lost against 0).
Limitations
- The numbers above measure closeness to the bf16 model on our English and Korean texts and short
answers; they are not task benchmarks, and no device speed was measured.
- TT-NN packing was tested on Tenstorrent's simulator, not on Tenstorrent hardware.
- Everything else about the model, its abilities, risks and intended use, is the base model's: read
its card.
License
A derivative of Qwen/Qwen3-4B, distributed under the base model's license:
the Apache License 2.0 (LICENSE).
Changes from the base model are listed in NOTICE. The scripts pack_tensix.py and
verify_grid.py are by Xylolabs, under the same license as this repository.
The model is provided as is, without warranty of any kind.
Citation
@misc{qwen3technicalreport, title={Qwen3 Technical Report}, author={Qwen Team}, year={2025}, eprint={2505.09388}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.09388}, }