SmallAICreator

AuroraGPT-ToolFix

README

License: apache-2.0

The problem it fixes

AuroraGPT-Math added ~250k math examples, which diluted the tool-calling signal ~5x in the training mix. The result: tool calls still fired on the exact system-prompt wording used in training, but became unreliable when an app phrased the tool spec differently (e.g. a <tools> JSON block instead of the trained natural-language list). In practice this meant tool-calling broke inside a real on-device chat app.

The fix: prompt-variation augmentation

Every one of the 2,751 tool examples was replayed under 12 different system-prompt wordings — the original phrasing, a <tools> JSON spec block, terse, verbose, bulleted, numbered, XML-ish, generic-assistant, JSON-only, and with no system prompt at all — producing ~33k tool examples. This teaches the model that tool-calling binds to intent, not to one memorized string. Mixed with chat/identity data so nothing else drifts. LoRA r16, 1 epoch, merged.

Measured results

Valid tool calls by system-prompt wording (3 prompts each, locally benchmarked):

Table with columns: system prompt style, AuroraGPT-Math, AuroraGPT-ToolFix
system prompt style	AuroraGPT-Math	AuroraGPT-ToolFix
exact trained wording	3/3	3/3
`<tools>` JSON block	2/3	3/3
terse	3/3	3/3
generic assistant	3/3	3/3
numbered list	3/3	3/3
total	14/15	15/15

Verified working in a real on-device GGUF chat app, where the previous model failed.

Inherited from AuroraGPT-Math: closed-book arithmetic roughly doubled vs the original flagship (5/16 → 10/16 on a 16-question set) via ~250k procedurally generated, correct-by-construction chain-of-thought math examples.

vs LiquidAI LFM2-700M

Same size class, both Q8_0, each model in its own native chat format.

Tool-calling (5 prompts):

Table with columns: AuroraGPT-ToolFix, LFM2-700M
	AuroraGPT-ToolFix	LFM2-700M
valid tool calls	5/5	0/5

LFM2 attempts tool use but emits unparseable pseudo-code (e.g. <tool_call>fetch_url(url="...")</tool_call>) rather than JSON, so no app harness can execute it.

General chat quality (20-question objective set), with AuroraGPT's tools enabled and calculator calls actually executed — i.e. how it's actually deployed:

Table with columns: section, AuroraGPT-ToolFix, LFM2-700M
section	AuroraGPT-ToolFix	LFM2-700M
facts	6/6	6/6
math	4/5	5/5
instruction-following	3/5	5/5
safety (should refuse)	1/2	1/2
over-refusal (should answer)	2/2	2/2
TOTAL

On math, AuroraGPT mostly delegates to the calculator tool rather than computing inline — 4 of 5 math questions were answered by emitting a calculator call and reading back the result. That is the intended design ("own the behavior, rent the facts"), and it is why tools-enabled scoring is the honest measure: with tools disabled the same model scores 12/20, because correct tool calls go unexecuted.

Honest read: LFM2-700M is still better at general chat quality (19/20 vs 16/20). AuroraGPT's advantage is tool-calling, which LFM2 cannot do at all.

Honest limitations

Little/no safety refusal training. On a 2-prompt safety probe it refused only 1/2 — it will comply with some requests it should decline. This is a real gap, not a benchmark artifact. Do not deploy user-facing without a separate safety layer.
General chat quality trails LFM2-700M (16/20 vs 19/20 even with tools enabled). AuroraGPT's edge is tool-calling, not across-the-board quality.
Instruction-following is inconsistent (3/5) — it can miss exact-format constraints like "reply with only yes or no" or "say hello in all caps".
Inline arithmetic is still fragile — it usually routes math to the calculator tool (good), but when it answers inline it can be wrong (e.g. "15% of 80" → 20, correct is 12). Keep the calculator tool available when exactness matters.
False-premise correction is weak — it can confidently agree with a popular myth.
700M capacity limits apply: closed-book knowledge is thin by design — pair with web_search/fetch_url.

Chat format (NOT ChatML)

markdown
<|system|>{system}<|end|><|user|>{user}<|end|><|assistant|>{reply}<|end|>

Tool call (model emits): <tool_call>\n{"name": "...", "arguments": {...}}\n</tool_call> Tool result (feed back as a user turn): <|user|><tool_response>\n{result}\n</tool_response><|end|>

Tool-calling now works with a wide range of system-prompt phrasings, so most app-provided tool specs should trigger it.

Usage (transformers)

python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("SmallAICreator/AuroraGPT-ToolFix")
model = AutoModelForCausalLM.from_pretrained("SmallAICreator/AuroraGPT-ToolFix")
msgs = [{"role": "user", "content": "What's the capital of Burkina Faso? Look it up."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=100)[0][ids.shape[1]:], skip_special_tokens=True))

On-device (llama.cpp / GGUF)

AuroraGPT-ToolFix.Q8_0.gguf (753MB) is included with a tool-capable chat template embedded.

Made by UltraLabs. EOS token is <|end|>.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

SmallAICreator

Model Tree

Base

this model

Input Modalities

Text

Output Modalities