canbingol

qwen2.5-3B-Instruct-tool-call-en-mixed-when2call-v2

README

License: apache-2.0

⚠️ Warning

This model was fine-tuned for experimental purposes, as part of an investigation into reducing over-triggered tool calls (i.e., getting the model to correctly abstain when no tool is relevant).

nvidia/When2Call (train_sft) was included in training to improve this behavior. However, this had an unintended side effect: the model's core tool-calling ability degraded significantly. It now frequently asks the user for clarification/input even for simple, complete, unambiguous tool-call requests where all required parameters are already provided.

Do not use this model in production or for tool-calling tasks. It is kept public only as a record of this experiment (see BFCL eval results above — simple_python_acc and multiple_acc collapsed while irrelevance_acc hit ceiling, indicating over-correction rather than genuine improvement).

Qwen2.5-3B-Instruct — Tool Call (EN, Mixed + When2Call 3.3k)

LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for function/tool calling.

Data

Salesforce/xlam-function-calling-60k — 54,000 (tool-call)
HuggingFaceH4/no_robots — 8,000, filtered ≤2,000 chars (no-tool SFT)
nvidia/When2Call (train_sft) — 3,300 (irrelevance/negative signal)
Concatenated, shuffled, formatted via apply_chat_template

LoRA

r=32, lora_alpha=64, lora_dropout=0.05, target_modules=all-linear, bias=none, task_type=CAUSAL_LM

Training

lr=2e-4, epochs=1, per_device_batch=8, grad_accum=16 (effective 128), warmup_ratio=0.03, max_grad_norm=0.3, bf16=True, attn_implementation=sdpa, loss_type=nll

Training Results (step 505, final)

train_loss=0.354, val_loss=0.173, mean_token_accuracy=0.956, runtime≈4h02m (A100)

BFCL Eval Results (inspect_evals/bfcl)

Table with columns: Metric, Score
Metric	Score
accuracy (all)	0.429
irrelevance_acc	1.000
multiple_acc	0.045
parallel_acc	0.595
parallel_multiple_acc	0.640
simple_python_acc	0.090

Note: irrelevance_acc reached 1.000 (perfect abstention), but simple_python_acc (0.090) and multiple_acc (0.045) collapsed even further. Even with a reduced When2Call subset (3.3k), the negative signal causes heavy over-abstention from tool calling.

Comparison to prior runs

Table with columns: Run, accuracy, irrelevance_acc, simple_python_acc, multiple_acc
Run	accuracy	irrelevance_acc	simple_python_acc	multiple_acc
Base Qwen2.5-3B	0.815	0.608	0.953	0.910
EN-only (xlam 60k)	0.705	0.213	0.893	0.865
EN + no_robots	0.730	0.225	0.910	0.875

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

canbingol

Model Tree

Base

Qwen/Qwen2.5-3B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities