Results
Qwen3.5-4B in thinking mode averages ~13.4k reasoning tokens on MATH-500. With Diogenes-4B it averages ~1.1k, while keeping most of the accuracy and all of the code ability on our test.
Table with columns: Eval (seed 42), Base Qwen3.5-4B, Diogenes-4B| Eval (seed 42) | Base Qwen3.5-4B | Diogenes-4B |
|---|
| MATH-500, n=100 | 84 (32k cap, still truncated on 20 problems) | 75 (regraded) / 64 (strict) |
| Reasoning tokens, MATH-500 | 13.4k | 1.1k (~12x fewer) |
| GSM8K, n=100 | 88 | 80 |
| Reasoning tokens, GSM8K | 3.1k | 114 (~27x fewer) |
| Code, 20 HumanEval-style tasks | 14/20 | 14/20 |
| Valid notation | n/a | 83% MATH-500, 99% GSM8K |
- About 89% of the base's MATH-500 accuracy and 91% of its GSM8K accuracy at 8% and 4% of the tokens. Code ties.
- "Regraded" accepts equivalent answer forms (e.g.
x=3 vs 3) and never changes the value; "strict" is the raw string match.
- Base numbers on MATH-500 are a floor, since 20 base generations hit the 32k cap.
- Single seed, our own harness, small samples (n=100; 20 code tasks).
Before and after
A GSM8K test problem (both answers correct):
Martha's cat is 5 times faster than her turtle. If the cat can run 15 feet/second, how many feet can her turtle crawl in 40 seconds?
The raw reasoning text each model produced, untouched.
-
Analyze the Request:
- Problem: A grade-school math problem about speeds.
- Given information:
- Cat's speed = 5 times the turtle's speed.
- Cat's speed = 15 feet/second.
- Time = 40 seconds.
- Goal: Find the distance the turtle can crawl in 40 seconds.
- Format: Show reasoning step by step, then final numeric answer on its own line as 'Answer: <number>'.
-
Break Down the Problem:
- Step 1: Determine the turtle's speed.
- Relationship: Cat's speed = 5 * Turtle's speed.
- Equation: 15 = 5 \times \text{Turtle's Speed}.
- Solve for Turtle's Speed: \text{Turtle's Speed} = 15 / 5.
- Step 2: Determine the distance the turtle travels.
- Formula: Distance=Speed×Time.
- Values: \text{Speed} = \text{Turtle's Speed} (calculated in Step 1), Time=40 seconds.
- Equation: Distance=Speed×40.
-
Execute the Calculation:
- Step 1:
- 15/5=3 feet/second.
- So, the turtle crawls at 3 feet/second.
- Step 2:
- Distance=3 feet/second×40 seconds.
- 3×40=120 feet.
-
Draft the Reasoning:
- First, find the speed of the turtle. We know the cat is 5 times faster than the turtle and the cat runs at 15 feet/second. Therefore, the turtle's speed is 15÷5=3 feet/second.
- Next, calculate the distance the turtle crawls in 40 seconds. Using the formula Distance=Speed×Time, we multiply the turtle's speed (3 feet/second) by the time (40 seconds).
- 3×40=120.
-
Final Check:
- Cat speed = 15.
- Turtle speed = 15/5=3.
- Time = 40.
- Distance = 3×40=120.
- Units: Feet.
- Format: 'Answer: 120'.
-
Construct Output:
- Write down the steps clearly.
- End with the specific requested format.
Self-Correction/Refinement: The prompt asks for "Answer: <number>". I need to make sure there is no extra text after the answer line. The reasoning should be before it.
Drafting the response:
Step 1: Find the turtle's speed.
The cat's speed is 5 times the turtle's speed. Since the cat runs at 15 feet/second, we divide 15 by 5 to find the turtle's speed.
15 feet/second/5=3 feet/second.
Step 2: Calculate the turtle's distance.
Distance equals speed multiplied by time. The turtle's speed is 3 feet/second and the time is 40 seconds.
3 feet/second×40 seconds=120 feet.
Answer: 120
Final Polish: Ensure the format matches exactly. "Answer: <number>".
Let's write it out.
- Identify cat's speed: 15 feet/second.
- Identify relationship: Cat = 5 * Turtle.
- Solve for Turtle speed: 15 / 5 = 3 feet/second.
- Identify time: 40 seconds.
- Calculate distance: 3 * 40 = 120 feet.
- Format output.
Ready.cw
Same answer, about 18x fewer tokens. (Line breaks in the Diogenes column were added for readability; the model writes it as one line.)
Usage
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = "Qwen/Qwen3.5-4B"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "ademRguez/Diogenes-4B") msgs = [{"role": "user", "content": "What is the sum of the first 10 positive odd integers?"}]prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=True)out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=2048, do_sample=True, temperature=1.0, top_p=0.95, top_k=20)print(tok.decode(out[0], skip_special_tokens=False))
Use thinking mode (enable_thinking=True); the compressed notation appears in the thinking. The notation is described in NOTATION_SPEC.md.
Training
Fresh LoRA (r=16, alpha=32, attention + MLP projections) on Qwen/Qwen3.5-4B, 6,666 rows: 4,062 OpenR1-Math traces, 1,000 GSM8K and 1,604 OpenThoughts code traces, all rewritten into the notation and validated locally. 3 epochs (627 steps), bf16, max sequence length 2048.
Support
If you find this useful, consider buying me a coffee to support continued development: