Recipe
The recipe is the one of
dbirks/Qwen3.8-27B-W4A16-AutoRound
(revision 1f05c441):
- Scheme
W4A16: 4-bit symmetric integer weights, group size 128.
- Calibration:
NeelNanda/pile-10k, 128 samples of 2,048 tokens, batch 4,
200 iterations, seed 42.
- Quantized: 400 of 607 linear layers (the GatedDeltaNet
in_proj_qkv,
in_proj_z and out_proj, the full-attention q/k/v/o projections and
every MLP projection).
- Kept in BF16:
linear_attn.in_proj_a and in_proj_b (96 layers), the
vision tower, lm_head and the multi-token prediction (MTP) head.
- Time: 43 minutes on one RTX PRO 6000 Blackwell (96 GB). Peak 23.9 GB VRAM
and 24.9 GB host RAM.
The tensor names, data types and shapes are the same as in the dbirks
checkpoint (1,999 tensors), and config.json is the same apart from
transformers_version. Thus it loads in the same way.
MTP head
Arete has no MTP head: Nous Research disabled it during training
(mtp_num_hidden_layers: 0). This checkpoint contains the BF16 MTP head of
the base model Qwen/Qwen3.8-27B (revision 37561816, 15 tensors) in
model-mtp.safetensors, and config.json sets mtp_num_hidden_layers: 1.
That head was trained on the hidden states of the base model, not of Arete.
In our tests it drafts as well on Arete as on the base model: 2.5 to 2.6
tokens per verify step with three draft tokens, the same as the dbirks
checkpoint.
To serve without speculative decoding, do not set --speculative-config.
vLLM then does not use the MTP head.
Use with vLLM
vLLM reads the int4 scheme from config.json. No quantization flag is
necessary.
vllm serve supz/Arete-27B-W4A16-AutoRound \ --served-model-name arete-27b \ --max-model-len 32768 \ --reasoning-parser qwen3 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
Use the sampling settings of the base model: temperature 1.0, top-p 0.95,
top-k 20 in thinking mode. Greedy decoding is not recommended.
Evaluation
All tests used HyperQwen, not stock
vLLM. HyperQwen is a patched vLLM 0.31 that fits Qwen3.8-27B with a 150K-token
context on one 24 GB GPU. We ran it on one RTX PRO 4000 Blackwell (24 GB), with
MTP drafting and an FP8 KV cache. HyperQwen's prepare.sh converts
lm_head, embed_tokens and the MTP head to int8 before serving. Thus the
numbers below are for that int8 variant of these files, not for the BF16
layers as published. The baseline is the dbirks checkpoint on the same GPU
and runner.
Table with columns: Check, This checkpoint, dbirks W4A16| Check | This checkpoint | dbirks W4A16 |
|---|
| Decode, one stream / eight streams | 80 / 436 tok/s | 85 to 94 / 465 tok/s |
| Tokens per verify step (MTP) | 2.5 to 2.6 | 2.5 to 2.6 |
| 148K-token context check | pass | pass |
Decode is 6 to 15 percent slower than the dbirks "fast" variant. That
variant uses an int4 lm_head taken from the base model, which does not
apply to Arete.
We also asked both checkpoints eight questions on topics where Qwen models
use the framing of the Chinese state (Tiananmen 1989, the Tank Man,
Xinjiang, Taiwan, the Hong Kong National Security Law, criticism of
Xi Jinping, Tibet, Falun Gong), plus two control questions about other
governments. The dbirks checkpoint refused the Tiananmen and Tank Man
questions and used the framing of the Chinese state for Hong Kong, Tibet
and Xi Jinping. This checkpoint answered all eight with the documented facts
and gave the position of the government as context. Both answered the
control questions fully. This checkpoint made some errors in details, for
example names and dates. Check the details against a source.
No standard benchmark results (perplexity, lm-evaluation-harness) are
available.
License
Apache 2.0, the same as NousResearch/Arete-27B and Qwen/Qwen3.8-27B.