Numbers
Held-out sets share no question with training; the second one uses indirect phrasing ("is it worth upgrading
to the latest Python?", "what's 18% tip on $47.80 split three ways?") and traps such as past-dated questions and
data the user already supplied. Thinking off, egirl, Q8_0 + f16 adapter.
Table with columns: B3-27B, B4-27B | B3-27B | B4-27B |
|---|
| held-out (41): right first move | 32/41 | 41/41 |
| current-info questions searched | 5/10 | 10/10 |
| arithmetic sent to the calculator | 7/10 | 10/10 |
| easy questions answered directly | 10/10 | 10/10 |
| harder set (37): right first move | 28/37 | 36/37 |
| current-info questions searched | 4/10 | 10/10 |
| no-tool questions answered directly | 15/15 | 15/15 |
| user-supplied numbers sent to the calculator | 2/4 | 3/4 |
| ARC / ARC-Easy / TruthfulQA | 64.2 / 85.6 / 39.4 | 64.2 / 85.8 / 39.2 |
With thinking low: 41/41 (B3-27B 37/41). Keyed answers were right where the key was valid; two date questions
("days until Christmas") were keyed for the day before the run and both models answered for the run day.
The one harder-set miss summed user-supplied numbers correctly without the calculator.
Known residuals
- It sometimes writes math in LaTeX (
\(x^2\)). Render accordingly.
- The control pairs' chosen answers were written by B2-9B, not B3-27B. Direct answers came out a little
shorter (mean 104 vs 119 characters on the 15 no-tool questions).
Notes
- Built for egirl, used through its OpenAI API with server-side tools. Other harnesses weren't a goal and
weren't tested.
- Date math (
days_between, add_days, weekday, today() inside calculate) needs an egirl with the
calculate date functions (PR #194); 12 training calls use them.
- Trained with Merlina, ORPO β 0.1, LoRA r16/α32 on all attention
and MLP projections, lr 2e-5 cosine, batch 8, on a DGX Spark (GB10). Stronger runs (lr 5e-5, 3 epochs, r32)
scored the same, so the lightest shipped.
- The adapter was baked into the weights with Kessel. The
mtp.*
and vision tensors are byte-identical to B3-27B.
No GGUF is published. Convert with llama.cpp's convert_hf_to_gguf.py; B3-27B's (B2-27B's) mmproj works.