Table with columns: Tensors, Format| Tensors | Format |
|---|
| 252 declared decoder projections | bfloat4_b, 4.5 bits per weight; GPTQ with per-block exponent search |
| Embedding, explicit tied-head copy and all other stored tensors | BF16 (147 entries) |
All stored entries pack to about 3.60 GB (6.53 bits per stored weight); sharing the tied head once gives about
2.82 GB (5.613 bits per unique weight). Both explicit BF16 embedding/head copies are byte-identical. The
embedding stays BF16, while quantized projections average 4.5 bits per weight. The memory figure above
conservatively includes both stored copies.
quant4.json records each projection format. GPTQ uses damping 0.1 and 65,536 WikiText-2 and Korean Wikipedia
tokens. BF16 grid storage is larger than the packed representation.
Quickstart
Use Transformers 5.23.0 with PyTorch. The original tokenizer and chat template are retained.
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/Qwen3-4B-SafeRL-Tenstorrent-Tensix-BFP4"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")messages = [{"role": "user", "content": "What is the capital of France? Reply with the city name only."}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, enable_thinking=False).to(model.device)out = model.generate(**inputs, max_new_tokens=64, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The example explicitly disables thinking through the original template.
Run on Tensix
All 252 declared matrices pass the grid and native TT-NN packing checks, including packing as BFP8. The exact
example also passes on the Blackhole simulator, checked with TT-NN 0.81.1. After downloading this repository
locally, pack one matrix in TT-NN's [in, out] layout:
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3-4B-SafeRL-Tenstorrent-Tensix-BFP4" # hf download Xylolabs/Qwen3-4B-SafeRL-Tenstorrent-Tensix-BFP4 --local-dir Qwen3-4B-SafeRL-Tenstorrent-Tensix-BFP4name = "model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous()device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py Qwen3-4B-SafeRL-Tenstorrent-Tensix-BFP4 checks every declared quantized matrix; --device adds a
device DRAM round trip. python verify_grid.py Qwen3-4B-SafeRL-Tenstorrent-Tensix-BFP4 checks the grid using torch alone.
These are representation checks; tt-transformers and tt-forge were not checked with this checkpoint.
Evaluation
Table with columns: Profile, Mean KL, Normal right / 33, Thinking right / 7, Counted normal losses| Profile | Mean KL | Normal right / 33 | Thinking right / 7 | Counted normal losses |
|---|
| Original BF16 | 0 | 28 | 7 | — |
| This export | 0.067215654 | 28 | 7 | 1 |
| Matched Q4_K_M | 0.061407450 | 29 | 7 | 0 |
The relative rule-7 gate passes: 1.094585 times the reference KL, within the 1.5-times limit, with no more
than one additional counted loss. The direct canonical benchmark CLI retains the configured margin and
thinking policy.
The strict normal-only check retains its one lost item; the relative gate allows that one additional loss
against the reference.
KL covers 16,240 scored positions on the same saved English/Korean raw and chat texts, four chunks of each.
The selected-text llama.cpp BF16 floor is 0.000316937. Every loaded text-model tensor is checked against the
indexed export values. Answer counts and the original thinking runs come from the retained exact-weight
checks, rescored with the canonical CLI. These are host checks of the saved weights. Speed is not reported
here.
Source: Qwen/Qwen3-4B-SafeRL at 1b95ccb88cabaf284ffb6968719c40ae6cd4085b.
License
Apache 2.0, as declared by the original model (LICENSE). Changes are documented in NOTICE; original
Xylolabs generic packing helpers use Apache 2.0 (CODE_LICENSE). Provided as is, without warranty. Not
affiliated with or endorsed by Tenstorrent.
Citation
Cite the original Qwen/Qwen3-4B-SafeRL release.