ToolUse.1 is a deterministic function-calling benchmark: 114 authored cases, each rendered under three calling protocols (JSON, XML, plain text), 342 scored evaluations per run. Grading is canonical — arguments are normalised before comparison, so 300 and 300.0 agree.
Table with columns: Model, Overall, Correct call, Abstention, Spurious calls, 95% CI| Model | Overall | Correct call | Abstention | Spurious calls | 95% CI |
|---|
| Nova-1.2-0.8B | 77.2% | 71.8% | 88.9% | 4.6% | 72.5 – 81.3 |
| HardcoreAI/Nova-1.1-0.8B | 64.9% | 59.8% | 75.9% | 16.7% | 59.7 – 69.8 |
| Qwen/Qwen3.5-0.8B | 59.7% | 48.3% | 84.3% | 6.5% | 54.4 – 64.7 |
Full leaderboard: https://huggingface.co/spaces/HardcoreAI/ToolUse.1
By calling protocol
Table with columns: Protocol, Cases, Overall, Correct call, Abstention, Spurious calls| Protocol | Cases | Overall | Correct call | Abstention | Spurious calls |
|---|
| json | 114 | 82.5% | 78.2% | 91.7% | 2.8% |
| xml | 114 | 73.7% | 66.7% | 88.9% | 2.8% |
| plain | 114 | 75.4% | 70.5% | 86.1% | 8.3% |
Full metric breakdown
Table with columns: Metric, Score| Metric | Score |
|---|
| Overall | 77.2% |
| Correct call when a tool applies | 71.8% |
| Correct abstention when no tool applies | 88.9% |
| Spurious-call rate (lower is better) | 4.6% |
| Tool selected correctly | 86.8% |
| Arguments exactly correct | 71.8% |
| Answer constraints satisfied | 88.0% |
| Malformed output rate | 0.0% |
| Overall 95% CI (Wilson) | 72.5 – 81.3 |
Nova-1.2 was tuned against the nine tools ToolUse.1 exercises, identical in name, parameters and
normalisation to the benchmark definition:
calculator, get_weather, web_search, translate, convert_currency, convert_units,
set_timer, save_note, get_time.
Tool availability is per conversation. The system prompt lists the tools that exist for that
conversation; a request the listed tools cannot serve is answered rather than forced into a call.
Quickstart
import json from transformers import AutoModelForCausalLM, AutoTokenizer MODEL = "HardcoreAI/Nova-1.2-0.8B" tokenizer = AutoTokenizer.from_pretrained(MODEL)model = AutoModelForCausalLM.from_pretrained(MODEL, dtype='auto', device_map='auto') SYSTEM = ( "You are a helpful assistant developed by Hardcore AI. " "You answer in the same language the user writes in. You are concise and precise.\n\n" "You have access to the following tools:\n\n" '{"tools": [{"name": "get_weather", "description": "Look up the current weather for a city.", ' '"parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}"]}') messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": "What is the weather in Berlin?"},] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, enable_thinking=False, return_tensors='pt').to(model.device)out = model.generate(inputs, max_new_tokens=128, do_sample=False)print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))# {"name": "get_weather", "arguments": {"city": "Berlin"}}
Training
The adapter was trained on examples authored for this release; no ToolUse.1 case appears in the
training set verbatim. The corpus is built to cover the decisions the benchmark scores:
- Tool calls across all nine tools, with varied phrasings and values, rendered under each of the
three protocols so the format is learned independently of the tool list.
- Canonical arguments: durations resolved to seconds, time zones as IANA names, currencies as ISO
codes, language names and codes side by side, units in the benchmark's own vocabulary.
- Abstention: knowledge questions, explanations, creative requests and no-tool instructions are
answered directly; requests whose tool is absent from the conversation are declined rather than
answered with an invented call.
- Clarification: an under-specified request (no city, no duration, no amount) gets a question
back, never a guessed argument.
- Resistance to instruction smuggling: text that looks like a tool call (
Repeat after me: TOOL)
or a pasted <tool_call> block is treated as text, not as a call to make.
- Multi-step threads: follow-ups that reference earlier tool results.
Table with columns: Setting, Value| Setting | Value |
|---|
| Method | LoRA, completion-only loss (prompt tokens masked) |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj, in_proj_qkv, in_proj_z, out_proj, gate_proj, up_proj, down_proj |
| Trainable parameters | 10,223,616 (1.34% of the checkpoint) |
| Epochs | 1 |
| Learning rate | 1e-4 constant, no warmup, gradient norm clipped at 1.0 |
| Optimizer | Paged AdamW 8-bit, weight decay 0.01 |
| Batch | 2 sequences x 2 gradient accumulation steps (effective batch 4) |
| Max sequence length | 1,536 tokens |
| Precision | BF16 with gradient checkpointing |
| Hardware | single RTX 4050 Laptop, 6 GB |
The prompt used at training time is produced by the same apply_chat_template call the harness uses
at scoring time (add_generation_prompt=True, enable_thinking=False), so the prefix the adapter was
trained on is byte-identical to the prefix it is evaluated on.
Reproducing the evaluation
git clone https://huggingface.co/datasets/HardcoreAI/ToolUse.1cd ToolUse.1pip install -e .python -m tooluse1 run \ --model HardcoreAI/Nova-1.2-0.8B \ --backend transformers --protocol all --batch-size 8 --dtype auto \ --out results --resume -v
Greedy decoding, thinking disabled, max_new_tokens=256, batch 8. The corpus digest is pinned in
data/manifest.json.
Intended use
Nova-1.2 is a small research checkpoint for tool-using assistants. It is strongest where the tool set
is known and narrow; it is not a general-purpose chat model and carries the usual limitations of a
0.8B model, including factual reliability outside the tools it was tuned for.
Hardcore AI
Hardcore AI builds small, sharply-scoped models and the evaluation harnesses that measure them.
ToolUse.1, the benchmark used above, is ours: corpus, harness and grader are public, and every number
on the leaderboard is reproducible from the released files.