Example Output — Untouched Base vs. Weight-Space Math Tutor
Zero-Shot Plain Prompt:
Question: Solve for x: 7x + 9 = 3x + 41.Answer:
- Untouched
Qwen/Qwen3.5-0.8B-Base (0 weights edited — bare 1-line wrong guess):
Qwen3.5-0.8B-Math-Tutor (292,600 static weights edited, 0 gradients — verbatim greedy output):
We need to solve the linear equation: 7x + 9 = 3x + 41.We can solve by isolating x.Subtract 3x from both sides: 7x - 3x + 9 = 41 => 4x + 9 = 41.Subtract 9 from both sides: 4x = 41 - 9 = 32.Divide both sides by 4: x = 32/4 = 8.Thus x = 8.We can also check: 7*8 + 9 = 56 + 9 = 65; 3*8 + 41 = 24 + 41 = 65. Works.
Evaluated on 180 strictly disjoint problems (30 each in arithmetic, fractions_pct, algebra, geometry, gsm8k, and math_comp, zero overlap with probe pool) plus 256 FineWeb-Edu 512-token blocks:
Table with columns: Arm, Changed Weights, Math Acc (plain / chatml), Verified Teaching (plain / chatml), Eq Soundness, Teach Score, FineWeb-Edu NLL| Arm | Changed Weights | Math Acc (plain / chatml) | Verified Teaching (plain / chatml) | Eq Soundness | Teach Score | FineWeb-Edu NLL |
|---|
Qwen/Qwen3.5-0.8B-Base (A0) | 0 | 50.83% (13.33% / 88.33%) | 35.83% (1.11% / 70.56%) | 97.04% | 0.6039 | 2.6202 |
E1 Turn Bridge Only (A1) | 3,068 (0.0004%) | 80.83% (80.00% / 81.67%) | 51.11% (43.33% / 58.89%) | 95.27% | 0.8833 | 2.6204 |
Zero-Leak Pareto (A2) | 623,927 (0.083%) | 78.06% (71.67% / 84.44%) | 58.61% (53.89% / 63.33%) | 97.19% | 0.8978 | 2.6232 |
Qwen3.5-0.8B-Math-Tutor (A3) | 292,600 (0.039%) | 82.78% (80.00% / 85.56%) | 54.44% (42.22% / 66.67%) | 96.23% | 0.8888 | 2.6232 |
How It Works (Zero Gradients)
Every edit is a closed-form algebraic function of the pretrained static weight tensors and BPE token IDs:
E1 — Closed-Form Turn-Embedding Bridge (3,068 scalars):
- In Qwen's BPE vocabulary, digits
'0'..'9' carry no leading space while English openers (' We', ' Let', ' Step') do — causing bare ' ' + 1-line guesses to win right after Answer:.
- Re-anchoring
'Question' (14162) and 'Answer' (15666) in embed_tokens.weight toward Qwen3.5's latent mid-trained tutor circuit (<|im_start|>, user, assistant, <think>, </think>, Step/Solution) at plain_norm_scale = 0.72 respects the Layer-0 un-normalized residual skip connection vs. tied lm_head self-echo threshold, jumping plain math accuracy 6.0× (13.33% -> 80.00%) with +0.0002 nats LM shift.
E2 — Anti-Pedagogical Late-MLP Suppression (L14..L23):
- Damps specialist worksheet-continuation neurons (
"Question 2:"), multi-newline collapse neurons, bare multiple-choice guess neurons, and a 6D CJK exam-leakage subspace orthogonalized against English math vocabulary.
E3 — Pedagogical Step-Starter & Equation-Writer Boost (L13..L23):
- Boosts 32 specialist MLP neurons writing English step initiators, deductive connectors (
"therefore", "thus", "since"), and explicit intermediate equations, plus turn-boundary gate_proj/up_proj key routing on 9 core step starters.
Qwen3.5-0.8B-Math-Tutor preserves exact architectural parity with Qwen/Qwen3.5-0.8B-Base (tie_word_embeddings = True, standard 24-layer Qwen3_5ForConditionalGeneration, compatible with HuggingFace transformers, vLLM, and SGLang):
import torchfrom transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration ckpt = "DJLougen/Qwen3.5-0.8B-Math-Tutor"tok = AutoTokenizer.from_pretrained(ckpt)model = Qwen3_5ForConditionalGeneration.from_pretrained( ckpt, dtype=torch.bfloat16, device_map="auto",).eval() # Works zero-shot with plain "Question: ...\nAnswer:" OR ChatMLprompt = "Question: Solve for x: 7x + 9 = 3x + 41.\nAnswer:"inputs = tok(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=256, do_sample=False) print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Honest Limitations
- Mental-Math Digit Ceiling: Without compiled tool layers, a
0.8B model still fails 8–10 digit long multiplication (20% accuracy) and cannot plot coordinate graphs. Use DJLougen/Qwen3.5-0.8B-Math-GraphCalc when exact bignum/rational/CORDIC/graphing execution is required.
- Tied-Embedding Scratchpad vs. Clean-Stop Tradeoff: Preserving
tie_word_embeddings = True creates a slight Pareto tradeoff between maximum scratchpad self-verification (82.78% math_acc) and occasional literal </think> emissions when skip_special_tokens=False.
License
Apache-2.0 (inherited from Qwen/Qwen3.5-0.8B-Base).