Scope
The text decoder is qualified here. Original vision and MTP parameters remain in the archive for source
coverage; their execution is outside these checks. The memory figure conservatively includes every stored
entry.
Table with columns: Tensors, Format| Tensors | Format |
|---|
| 248 declared decoder projections | bfloat4_b, 4.5 bits per weight; GPTQ |
| Embedding, explicit tied-head copy and all other stored tensors | BF16 (443 entries), F32 (48 entries) |
All stored entries pack to about 5.46 GB (8.25 bits per stored weight); sharing the tied head once gives about
4.19 GB (7.192 bits per unique weight). Both explicit BF16 embedding/head copies are byte-identical. The
embedding stays BF16, while quantized projections average 4.5 bits per weight. The memory figure above
conservatively includes both stored copies.
quant4.json records each projection format. Plain GPTQ uses damping 0.1, without block exponent search. BF16
grid storage is larger than the packed representation.
Quickstart
Use Transformers 5.23.0 with PyTorch. The original tokenizer and chat template are retained.
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/Qwen3.5-4B-Tenstorrent-Tensix-BFP4-GPTQ"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")messages = [{"role": "user", "content": "What is the capital of France? Reply with the city name only."}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, enable_thinking=False).to(model.device)out = model.generate(**inputs, max_new_tokens=64, do_sample=False)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The example explicitly disables thinking through the original template.
Run on Tensix
All 248 declared matrices pass the grid and native TT-NN packing checks, including packing as BFP8. The exact
example also passes on the Blackhole simulator, checked with TT-NN 0.81.1. After downloading this repository
locally, pack one matrix in TT-NN's [in, out] layout:
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3.5-4B-Tenstorrent-Tensix-BFP4-GPTQ" # hf download Xylolabs/Qwen3.5-4B-Tenstorrent-Tensix-BFP4-GPTQ --local-dir Qwen3.5-4B-Tenstorrent-Tensix-BFP4-GPTQname = "model.language_model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous()device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py Qwen3.5-4B-Tenstorrent-Tensix-BFP4-GPTQ checks every declared quantized matrix; --device adds a
device DRAM round trip. python verify_grid.py Qwen3.5-4B-Tenstorrent-Tensix-BFP4-GPTQ checks the grid using torch alone.
These are representation checks; tt-transformers and tt-forge were not checked with this checkpoint.
Evaluation
Table with columns: Profile, Mean KL, Normal right / 33, Thinking right / 7, Counted normal losses| Profile | Mean KL | Normal right / 33 | Thinking right / 7 | Counted normal losses |
|---|
| Original BF16 | 0 | 29 | 7 | — |
| This export | 0.047503000 | 29 | 7 | 0 |
| Matched Q4_K_M | 0.039805000 | 30 | 7 | 0 |
The relative rule-7 gate passes: 1.193393 times the reference KL, within the 1.5-times limit, with no more
than one additional counted loss. The direct canonical benchmark CLI retains the configured margin and
thinking policy.
KL covers 16,312 scored positions on the same saved English/Korean raw and chat texts, two chunks of 2,048
tokens each. The selected-text llama.cpp BF16 floor is 0.000315000. Every loaded text-model tensor is checked
against indexed values at its loaded dtype. Original BF16 and export answer runs are fresh; Q4_K_M uses the
retained same-base records. All are scored with the direct canonical CLI. These are host checks of the saved
weights. Speed is not reported here.
Source: Qwen/Qwen3.5-4B at 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
License
Apache 2.0, as declared by the original model (LICENSE). Changes are documented in NOTICE; original
Xylolabs generic packing helpers use Apache 2.0 (CODE_LICENSE). Provided as is, without warranty. Not
affiliated with or endorsed by Tenstorrent.
Citation
Cite the original Qwen/Qwen3.5-4B release.