What changed
Only these weights differ from the base model, 9,213 numbers in all:
Table with columns: Layer, Neuron, Change, From run| Layer | Neuron | Change | From run |
|---|
| 15 | 349 | output column mlp.down_proj.weight[:, 349] ×4 | 3c80f51b |
| 18 | 1261 | output column mlp.down_proj.weight[:, 1261] ×0.5 | 0adc73a9 |
| 22 | 387 | output column mlp.down_proj.weight[:, 387] ×2 | 0adc73a9 |
| 22 | 573 | output column mlp.down_proj.weight[:, 573] ×2 | 0adc73a9 |
| 22 | 2388 | its gate row, scaled to the up row's length, added to its up row (mlp.up_proj.weight[2388, :]) | c001f524 |
| 23 | 613 | its gate row, scaled to the up row's length, subtracted from its up row (mlp.up_proj.weight[613, :]) | c001f524 |
| 23 | 970 | output column mlp.down_proj.weight[:, 970] rewritten (the change is 2.0× the original column's length) | 6760131e |
| 23 | 1004 | output column mlp.down_proj.weight[:, 1004] rewritten (the change is 9.9× the original column's length) | 6760131e |
| 23 | 1681 | its gate row, scaled to the up row's length, subtracted from its up row (mlp.up_proj.weight[1681, :]) | c001f524 |
The exact new values are in edits/.
Results
MATH-500 (all 500 problems, 4 answers each), measured on this exact checkpoint:
The gain comes from fewer answers cut off at the token limit (49.9%, against 61.6% with neuron 573 ×2
alone); the answers that do finish are right a little less often (85.4% against 93.1%).
Evaluation settings: thinking on, temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5, up to 32,768 new tokens (Takehi et al.'s settings), seed 0, 4 answers per problem, graded with math-verify 0.9.0, served with vLLM 0.32.0.
Usage
Load it like the base model:
from transformers import AutoModelForImageTextToText, AutoTokenizer model = AutoModelForImageTextToText.from_pretrained("rilixai/Qwen3.5-0.8B-Beaker-math-all-fixes", torch_dtype="bfloat16")tokenizer = AutoTokenizer.from_pretrained("rilixai/Qwen3.5-0.8B-Beaker-math-all-fixes")messages = [{"role": "user", "content": "What is 12 × 7?"}]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
or serve it with vLLM: vllm serve rilixai/Qwen3.5-0.8B-Beaker-math-all-fixes.
License
Apache-2.0, as the base model. This is a modified version of
Qwen/Qwen3.5-0.8B: only the weights listed above
were changed. See LICENSE.