Table with columns: Tensors, Format| Tensors | Format |
|---|
| attention q, k, v, o (36 layers) | bfloat4_b |
| MLP gate and up (36 layers) | bfloat4_b |
| MLP down (36 layers) | bfloat4_b |
| LM head | bf16 |
| embedding, norms, biases and every other tensor of the language model | bf16, as in the base model |
bfloat4_b: blocks of 16 values (16 consecutive output channels of one input channel) share one
8-bit exponent; each value keeps a sign and a 3-bit mantissa: 4.5 bits a weight. bfloat8_b: the
same with a 7-bit mantissa: 8.5 bits a weight.
Average bits a weight over every weight read (embedding and head included): 6.25. Packed in
these formats the weights take 6.397 GB; the files here store them as bf16 (16.4 GB).
How it was made
- GPTQ (Frantar et al., 2022) per projection: input channels in decreasing order of the Hessian diagonal, blocks of 128 columns, damping 0.1 of the mean diagonal. For bfloat4_b and bfloat8_b a column's blocks are its groups of 16 output channels, and each block's exponent is chosen from the updated column exactly as the packer will choose it.
- Exponent search: for each block, the exponent of its largest value and one lower are both tried (one lower clips the largest values and halves the step for the rest); the one with the smaller error is kept. Every value it produces is exactly representable in the format.
- Calibration: 65,536 tokens of WikiText-2 (train) and Korean Wikipedia articles, run through the bf16 model; none of it is in the evaluation texts below.
Accuracy against bf16
Table with columns: Bits a weight, Mean KL to bf16, Top-1 agreement (%), Answers lost (of 33 + 7) | Bits a weight | Mean KL to bf16 | Top-1 agreement (%) | Answers lost (of 33 + 7) |
|---|
| this model | 6.25 | 0.0449 | 91.26 | 0 |
| Q4_K_M (llama.cpp) | 4.90 | 0.04236 | 91.24 | 0 |
- Mean KL: KL(bf16 ‖ model) over the full vocabulary at every scored position (16240
positions) of four texts: English and Korean, each as plain text and as the reply of a chat turn in the model's template, with thinking off.
- Top-1 agreement: how often the model's most likely next token is bf16's.
- Answers lost: 33 short-answer prompts with thinking off and 7 with thinking on, answered greedily;
an item is lost when bf16 answers it right and the model does not, or the answer is broken.
- Q4_K_M: llama.cpp's Q4_K_M of the same model (
llama-quantize Q4_K_M from a bf16 GGUF, no importance matrix), scored the same way.
- Our bar: mean KL at most 1.5 times Q4_K_M's and at most one more lost answer than Q4_K_M. This model: passed (1.06 times Q4_K_M's mean KL).
These numbers measure closeness to the bf16 model on our texts; they are not task benchmarks.
Use
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "hletrd/Qwen3-8B-BFP4"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")
License
A derivative of Qwen/Qwen3-8B, distributed under the base model's license:
the Apache License 2.0 (LICENSE).
Changes from the base model are listed in NOTICE.
The model is provided as is, without warranty of any kind.