Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, v, o (16 layers) | bfloat8_b, 8.5 bits a weight |
| Gated DeltaNet q, k, v, z, a, b, output (48 layers) | bfloat8_b, 8.5 bits a weight |
| MLP gate, up, down (64 layers) | bfloat8_b, 8.5 bits a weight |
| LM head, embedding, norms and every other tensor | bf16, as in the base model |
visual | bf16, unchanged |
9.21 bits a weight on average, 30.96 GB packed. quant4.json lists each projection's format. Quantized with
GPTQ on the bfloat8_b grid, keeping each block's shared exponent as the packer sets it, calibrated on 65,536
tokens of WikiText-2 and Korean Wikipedia.
Only the language model was quantized and evaluated.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "Xylolabs/AREX-2-Tensix-BFP8"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": "What is the capital of France?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=64)print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Run on Tensix
Tested with ttnn 0.81.1 on ttsim, Tenstorrent's Blackhole simulator; not on hardware. Pack one matrix and
check that no value changed (TT-NN reads weights as [in, out]):
import jsonimport torchimport ttnnfrom safetensors import safe_open d = "AREX-2-Tensix-BFP8" # hf download Xylolabs/AREX-2-Tensix-BFP8 --local-dir AREX-2-Tensix-BFP8name = "model.language_model.layers.0.mlp.down_proj.weight"fmt = json.load(open(f"{d}/quant4.json"))["layers"]["0"]["mlp.down_proj"]["format"]dtype = {"bfp4": ttnn.bfloat4_b, "bfp8": ttnn.bfloat8_b}[fmt]shard = json.load(open(f"{d}/model.safetensors.index.json"))["weight_map"][name]with safe_open(f"{d}/{shard}", "pt") as f: w = f.get_tensor(name).t().contiguous() device = ttnn.open_device(device_id=0)w_tt = ttnn.as_tensor(w, dtype=dtype, layout=ttnn.TILE_LAYOUT, device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)assert torch.equal(ttnn.to_torch(w_tt).float(), w.float())ttnn.close_device(device)
python pack_tensix.py AREX-2-Tensix-BFP8 checks every matrix this way (--device also round-trips it
through device DRAM). python verify_grid.py AREX-2-Tensix-BFP8 checks the grid with torch alone.
tt-metal qwen38_27b_qb2 demo (v0.81.1). tt-transformers does not support this architecture. tt-metal's
qwen38_27b_qb2 demo runs
it tensor-parallel on four Blackhole chips (QuietBox 2) and refuses other hardware. Its shipped
config/precision.json and its baseline both pack the projections in bfloat4_b, which re-quantizes these
weights. Write a policy that keeps them in bfloat8_b and point QWEN_PRECISION_CONFIG at it. With that
policy, the decoder packed every projection of layers 0 and 3 exactly. The head goes to bfloat8_b too, so this
checkpoint's bf16 head is not exact there. The head and a full run need the four chips and were not tested.
# build tt-metal v0.81.1 as the demo's README describes, then, from the tt-metal checkout:python - <<'PY'import jsonfrom models.demos.qwen38_27b_qb2.tt.precision import BASELINEp = dict(BASELINE, config_id="bfp8_hifi2", weight_groups={k: "bfloat8_b" for k in BASELINE["weight_groups"]}, compute_fidelities={k: "HiFi2" for k in BASELINE["compute_fidelities"]})json.dump(p, open("precision_bfp8.json", "w"), indent=2)PYexport QWEN_PRECISION_CONFIG=$PWD/precision_bfp8.jsonexport MODEL_WEIGHTS_DIR=/path/to/AREX-2-Tensix-BFP8 # hf download Xylolabs/AREX-2-Tensix-BFP8 --local-dir ...python models/demos/qwen38_27b_qb2/demo/text_demo.py --full --length 128 --generate 128 --output qwen38.json
tt-metal's blackhole/qwen36 demo also runs Qwen3.8-27B (P150x4 or x8). It always packs the MLP gate and up in
bfloat4_b, so it does not run this checkpoint at its precision.
tt-forge (tt-xla). Not run with this checkpoint yet.
Evaluation
Table with columns: Bits a weight, Size (GB), Mean KL to BF16, Top-1 (%), Answers right, Lost | Bits a weight | Size (GB) | Mean KL to BF16 | Top-1 (%) | Answers right | Lost |
|---|
| BF16 (BAAI/AREX-2) | 16 | 53.79 | 0 | 100 | 38 of 40 | - |
| this model | 9.21 | 30.963 | 0.00082 | 98.74 | 38 of 40 | 0 |
| Q4_K_M (llama.cpp) | 4.92 | 16.547 | 0.02480 | 92.68 | 38 of 40 | 0 |
KL and top-1 cover 16,240 positions of English and Korean text (WikiText-2 test, Korean Wikipedia), plain and
in the chat template. Answers are 33 short prompts with automatic checks, plus the 7 reasoning ones again with
thinking on (40 in all). "Lost" means right in BF16 but wrong or broken here; near ties are not counted.
Q4_K_M was made from the same checkpoint with llama.cpp v0.9.0 (cd959bae) and no importance matrix; its
sha256 is 94c482505ea5add0c59ac7d389ab7e9c6fac0f11fe8cb5edf9fad18942fd5815. Our bar: mean KL at most 1.5
times Q4_K_M's and at most one more lost answer. Passed: 0.03 times Q4_K_M's KL, 0 lost against 0.
These numbers measure closeness to BF16, not task quality; no device speed was measured. For the model itself,
see the base card.
License
Apache 2.0, as the base model (LICENSE). The changes are listed in NOTICE. pack_tensix.py and
verify_grid.py are by Xylolabs under the same license. Provided as is, without warranty. Not affiliated with
or endorsed by Tenstorrent.
Citation
@article{2026arex2, title = {AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks}, author = {Qian, Hongjin and Li, Chaofan and Luo, Kun and Wei, Wenqing and Chen, Jianlyu and Lu, Shuqi and Hu, Yuyang and Xiao, Hongwang and Wang, Hui and Li, Chaozhuo and Ye, Qiwei and Dou, Zhicheng and Lian, Defu and Liu, Zheng}, journal = {arXiv preprint arXiv:2609.38288}, year = {2026}, url = {https://arxiv.org/abs/2609.38288}}