What changed
Only these weights differ from the base model, 5,114 numbers in all:
Table with columns: Layer, Neuron, Change| Layer | Neuron | Change |
|---|
| 22 | 387 | its gate row, scaled to the up row's length, subtracted from its up row (mlp.up_proj.weight[387, :]) |
| 22 | 573 | its gate row, scaled to the up row's length, subtracted from its up row (mlp.up_proj.weight[573, :]) |
| 22 | 2388 | its gate row, scaled to the up row's length, added to its up row (mlp.up_proj.weight[2388, :]) |
| 23 | 613 | its gate row, scaled to the up row's length, subtracted from its up row (mlp.up_proj.weight[613, :]) |
| 23 | 1681 | its gate row, scaled to the up row's length, subtracted from its up row (mlp.up_proj.weight[1681, :]) |
The exact new values are in edits/.
Results
MATH-500, held out from Beaker: 8.4% → 38.6% (all 500 problems, 4 answers each).
Evaluation settings: thinking on, temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5, up to 32,768 new tokens (Takehi et al.'s settings), seed 0, 4 answers per problem, graded with math-verify 0.9.0, served with vLLM 0.32.0.
Usage
Load it like the base model:
from transformers import AutoModelForImageTextToText, AutoTokenizer model = AutoModelForImageTextToText.from_pretrained("rilixai/Qwen3.5-0.8B-Beaker-math-c001f524", torch_dtype="bfloat16")tokenizer = AutoTokenizer.from_pretrained("rilixai/Qwen3.5-0.8B-Beaker-math-c001f524")messages = [{"role": "user", "content": "What is 12 × 7?"}]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
or serve it with vLLM: vllm serve rilixai/Qwen3.5-0.8B-Beaker-math-c001f524.
License
Apache-2.0, as the base model. This is a modified version of
Qwen/Qwen3.5-0.8B: only the weights listed above
were changed. See LICENSE.