Ternary-Bonsai-2-27B-vllm API & Inference Endpoint

What's in here

The ternary weights of the MLX pack, bit for bit: the same 2-bit codes and FP16 group scales (g128), in the same Hadamard-rotated basis.
Two tensors the MLX pack stores per layer are left out, as they carry no information: biases, which is always exactly -scales (MLX dequantizes as code * scale + bias; ternary is (code - 1) * scale), and signs, the ±1 vector of the Hadamard rotation, which is the same for every layer of a given input width and so is kept once per width in config.json. The converter checks both.
The embedding table is dequantized to BF16.
Norms and the linear-attention state path are Prism ML's own tensors from the MLX pack, converted to the Hugging Face layout.
The MTP head, which the MLX pack doesn't carry, is the BF16 head of Qwen/Qwen3.8-27B. It only drafts tokens for speculative decoding; every token is verified by the Bonsai model.
Text only: the vision tower is not included.

Produced by prism-ternary-convert from the repo above. For benchmarks, sampling parameters and everything else about the model, see the original model card.

License

Apache 2.0, as the original. Created using Bonsai by Prism ML; built from Qwen3.8-27B by Alibaba Cloud. See NOTICE.txt.

What's in here

The ternary weights of the MLX pack, bit for bit: the same 2-bit codes and FP16 group scales (g128), in the same Hadamard-rotated basis.
Two tensors the MLX pack stores per layer are left out, as they carry no information: biases, which is always exactly -scales (MLX dequantizes as code * scale + bias; ternary is (code - 1) * scale), and signs, the ±1 vector of the Hadamard rotation, which is the same for every layer of a given input width and so is kept once per width in config.json. The converter checks both.
The embedding table is dequantized to BF16.
Norms and the linear-attention state path are Prism ML's own tensors from the MLX pack, converted to the Hugging Face layout.
The MTP head, which the MLX pack doesn't carry, is the BF16 head of Qwen/Qwen3.8-27B. It only drafts tokens for speculative decoding; every token is verified by the Bonsai model.
Text only: the vision tower is not included.

Produced by prism-ternary-convert from the repo above. For benchmarks, sampling parameters and everything else about the model, see the original model card.

License

Apache 2.0, as the original. Created using Bonsai by Prism ML; built from Qwen3.8-27B by Alibaba Cloud. See NOTICE.txt.

Ternary-Bonsai-2-27B-vllm

README

What's in here

License

Explore FriendliAI today

README

What's in here

License