Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, v, o (8 layers) | bfloat4_b, 4.5 bits a weight |
| Gated DeltaNet q, k, v, z (24 layers) | bfloat4_b, 4.5 bits a weight |
| Gated DeltaNet a, b (24 layers) | bfloat8_b, 8.5 bits a weight |
| Gated DeltaNet output (24 layers) | bfloat4_b, 4.5 bits a weight |
| MLP gate, up, down (32 layers) | bfloat4_b, 4.5 bits a weight |
| LM head (tied to the embedding), embedding, norms and every other tensor | bf16, except the original dtypes below |
mtp, visual | bf16, unchanged |
Original tensors: model.language_model.layers.N.linear_attn.A_log, model.language_model.layers.N.linear_attn.norm.weight | float32, unchanged |
6.25 bits a weight on average, 3.28 GB packed. quant4.json lists each projection's format. The bfloat4_b
tensors were quantized with GPTQ and a per-block exponent search, calibrated on 65,536 tokens of WikiText-2
and Korean Wikipedia; the bfloat8_b tensors with round to nearest.
Only the language model was quantized and evaluated.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/Qwen3.5-4B-Base-Tenstorrent-Tensix-BFP4"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") inputs = tok("The capital of France is", return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=16)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
This is the pretrained model: it continues text. Its tokenizer keeps the base model's chat template, but the
model is not instruction-tuned.
Run on Tensix
Checked with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator. Pack one matrix and
check that no value changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "Qwen3.5-4B-Base-Tenstorrent-Tensix-BFP4" # hf download Xylolabs/Qwen3.5-4B-Base-Tenstorrent-Tensix-BFP4 --local-dir Qwen3.5-4B-Base-Tenstorrent-Tensix-BFP4name = "model.language_model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py Qwen3.5-4B-Base-Tenstorrent-Tensix-BFP4 checks every matrix this way (--device also round-trips
it through device DRAM). python verify_grid.py Qwen3.5-4B-Base-Tenstorrent-Tensix-BFP4 checks the grid with torch alone.
The same values also pack exactly into ttnn.bfloat8_b.
tt-metal Qwen3.5/3.6 demo (v0.81.1). tt-transformers does not support this architecture. tt-metal's
Qwen3.5/3.6 demo does
not list Qwen3.5-4B, but it reads every shape from config.json and loads the text model only. With
MESH_DEVICE=P150 it built layers 0 to 3 of this checkpoint and packed every projection exactly: DeltaNet,
attention and MLP down in bfloat8_b, MLP gate and up in bfloat4_b. Two parts are re-quantized: the LM head,
which the demo always packs in bfloat8_b, and the MTP block, which this checkpoint keeps in bf16 and the demo
packs in bfloat8_b (gate and up in bfloat4_b). QWEN36_SPEC=0 skips the MTP block. The demo's default prompts
are plain text. A full run was not tested.
# build tt-metal v0.81.1 as its INSTALLING.md describes, thenexport HF_MODEL=Xylolabs/Qwen3.5-4B-Base-Tenstorrent-Tensix-BFP4export MESH_DEVICE=P150pytest models/demos/blackhole/qwen36/demo/text_demo.py -v -s -k "traced_128 and not traced_128k"
tt-forge (tt-xla). Not run with this checkpoint yet.
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (Qwen/Qwen3.5-4B-Base) | 16 | 8.41 | 0 | 100 | 27 of 28 | - |
| this model | 6.25 | 3.283 | 0.03398 | 90.53 | 27 of 28 | 0 |
| Q4_K_M (llama.cpp) | 5.13 | 2.783 | 0.02929 | 91.12 | 27 of 28 | 0 |
KL and top-1 cover 8,188 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), as plain
text. Answers are 28 short prompts with automatic checks, in completion form, 3-shot. "Lost" means right in
BF16 but wrong or broken here; near ties are not counted. Q4_K_M was made from the same checkpoint with
llama.cpp v0.9.0 (cd959bae) and no importance matrix; its sha256 is
fe93290186a5619ebc5e369248c872f69e80c21d65aeedd4569664882118cdd3. Our bar: mean KL at most 1.5 times
Q4_K_M's and at most one more lost answer. Passed: 1.16 times Q4_K_M's KL, 0 lost against 0.
These numbers measure closeness to BF16, not task quality. Speed is not reported here.
License
Apache 2.0, as the base model (LICENSE). The changes are listed in NOTICE. pack_tensix.py and
verify_grid.py are by Xylolabs under the same license. Provided as is, without warranty. Not affiliated with
or endorsed by Tenstorrent.
Citation
@misc{qwen3.5, title = {{Qwen3.5}: Towards Native Multimodal Agents}, author = {{Qwen Team}}, month = {February}, year = {2026}, url = {https://qwen.ai/blog?id=qwen3.5}}