Harish241412

qwen2.5-1.5b-toolcalling-dpo

README

License: apache-2.0

Model Details

Table with columns: Property, Value
Property	Value
Base model	Qwen/Qwen2.5-1.5B-Instruct
Parameters	1.5B
Fine-tuning	LoRA + DPO
Dataset	NVIDIA When2Call
Training split	`train_pref`
Final checkpoint	Step 1000
Model format	Merged
Framework	PyTorch, Transformers, PEFT

The model was initially fine-tuned using LoRA and Direct Preference Optimization (DPO). The LoRA adapter was subsequently merged into the base model using merge_and_unload() to produce a standalone model.

Dataset

Training was performed using the train_pref split of NVIDIA's When2Call dataset.

When2Call is designed specifically to evaluate and train LLMs on decisions about when (and when not) to call tools. The dataset includes preference pairs consisting of a chosen and rejected response for a given user request and tool specification.

The train_pref split contains 9,000 preference-training examples with:

Tool specifications
User messages
Chosen responses
Rejected responses

NVIDIA provides both an SFT dataset (train_sft) and a preference dataset (train_pref); this model uses the preference dataset for DPO training.

The When2Call dataset is synthetic and is licensed under CC BY 4.0.

Training Objective

The objective was to improve the model's tool-use decision boundary.

The model learns to distinguish between:

Tool Call — a tool should be invoked to answer the request.
Request for Information — the request can be handled without invoking a tool.
Cannot Answer — the available tools cannot answer the request.

The preference-training setup encourages the model to prefer appropriate responses over incorrect tool-calling behavior.

Evaluation

The fine-tuned model was evaluated against the original Qwen2.5-1.5B-Instruct model on 300 samples.

The evaluation uses the same 300-sample LLM-as-a-judge subset provided by When2Call. NVIDIA's dataset contains a larger 3,652-sample MCQ test set and a 300-sample LLM-as-a-judge subset.

Results

Table with columns: Metric, Qwen2.5-1.5B-Instruct, Qwen2.5-1.5B + DPO
Metric	Qwen2.5-1.5B-Instruct	Qwen2.5-1.5B + DPO
Intent Accuracy	52.7%	74.0%
Tool Precision	40.8%	72.9%
Tool Recall	93.0%	35.0%
Tool F1	56.7%	47.3%
Argument F1	71.9%	67.7%
Unsupported Tool Calls ↓

Key Results

The largest improvement was in unsupported tool calls:

45.0% → 4.3%

This indicates that DPO substantially reduced inappropriate or hallucinated tool invocations.

Intent accuracy also increased:

52.7% → 74.0%

and tool precision increased:

40.8% → 72.9%

However, this came with a substantial reduction in tool recall:

93.0% → 35.0%

and Tool F1:

56.7% → 47.3%

Therefore, the main behavioral change is a shift toward a more conservative, precision-oriented tool-calling policy.

Confusion Matrix

Qwen2.5-1.5B-Instruct

Table with columns: Ground Truth \ Prediction, Tool, Request, Refusal
Ground Truth \ Prediction	Tool	Request	Refusal
Tool Call	93	3	4
Request For Information	76	23	1
Cannot Answer	59	22	19

Qwen2.5-1.5B + DPO

Table with columns: Ground Truth \ Prediction, Tool, Request, Refusal
Ground Truth \ Prediction	Tool	Request	Refusal
Tool Call	35	65	0
Request For Information	12	86	2
Cannot Answer	1	91	8

The confusion matrix shows that DPO significantly reduced the model's tendency to issue tool calls for requests that should not result in a tool invocation.

Tool Calling Example

Tool Definition

python
tools = [{
    "name": "get_stock_price",
    "description": "Fetch real-time stock price for a given ticker symbol.",
    "parameters": {
        "type": "object",
        "properties": {
            "ticker": {
                "type": "string",
                "description": "The ticker symbol (e.g., AAPL, NVDA)"
            }
        },
        "required": ["ticker"]
    }
}]

User

text
Can you check Nvidia's current stock price?

Model Output

text
<tool_call>
{"name": "get_stock_price", "arguments": {"ticker": "NVDA"}}
</tool_call>

Inference

python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

model.eval()

tools = [{
    "name": "get_stock_price",
    "description": "Fetch real-time stock price for a given ticker symbol.",
    "parameters": {
        "type": "object",
        "properties": {
            "ticker": {
                "type": "string",
                "description": "The ticker symbol (e.g., AAPL, NVDA)"
            }
        },
        "required": ["ticker"]
    }
}]

messages = [
    {
        "role": "user",
        "content": "Can you check Nvidia's current stock price?"
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id
    )

response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=False
)

print(response)

Output Post-Processing

During evaluation, model outputs were passed through a lightweight post-processing step to normalize tool-call formatting.

The post-processing:

Extracts JSON containing name and arguments.
Removes duplicate or nested <tool_call> wrappers.
Normalizes the output to:

text
<tool_call>
{"name": "...", "arguments": {...}}
</tool_call>

If no tool-call JSON is detected, the output is treated as a normal conversational response.

The post-processing step is used for format normalization and evaluation and does not generate a tool call that the model did not produce.

Limitations

The main limitation is the precision-recall trade-off introduced by DPO.

The model is considerably better at avoiding unsupported tool calls, but it also misses a larger proportion of valid tool-call opportunities.

Therefore, this model should not be interpreted as universally better than the base model for tool calling. Instead, it demonstrates that preference optimization can strongly shift the tool-use decision policy of a small instruction-tuned model.

The evaluation also uses a relatively small 300-sample subset, so additional evaluation on larger and more diverse tool-calling benchmarks would be useful.

Intended Use

This model is intended for research and experimentation involving:

Tool calling
Function calling
Tool-selection policies
Preference optimization
DPO
Agentic LLM systems
Small language model alignment

It is not intended to be considered production-ready without additional task-specific evaluation.

Future Work

Recover tool-call recall while maintaining low unsupported-call rates
Experiment with DPO hyperparameters
Improve preference-data construction
Compare DPO against SFT
Evaluate larger Qwen models
Evaluate multi-tool selection
Evaluate multi-step tool calling
Improve argument-generation accuracy
Benchmark inference using vLLM
Evaluate on larger tool-calling benchmarks

Base Model

This model is based on:

Qwen/Qwen2.5-1.5B-Instruct

Please refer to the base model for its original capabilities, license, and usage restrictions.

Dataset Citation

This work uses NVIDIA's When2Call dataset.

Ross, Hayley, Ameya Sunil Mahabaleshwarka, and Yoshi Suhara. "When2Call: When (not) to Call Tools." NAACL 2025.

bibtex
@inproceedings{ross-etal-2025-when2call,
    title = "{W}hen2{C}all: When (not) to Call Tools",
    author = "Ross, Hayley  and
      Mahabaleshwarkar, Ameya Sunil  and
      Suhara, Yoshi",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-long.174/",
    doi = "10.18653/v1/2025.naacl-long.174",
    pages = "3391--3409",
    ISBN = "979-8-89176-189-6",
    abstract = "Leveraging external tools is a key feature for modern Language Models (LMs) to expand their capabilities and integrate them into existing systems. However, existing benchmarks primarily focus on the accuracy of tool calling{---}whether the correct tool is called with the correct parameters{---}and less on evaluating when LMs should (not) call tools. We develop a new benchmark, When2Call, which evaluates tool-calling decision-making: when to generate a tool call, when to ask follow-up questions and when to admit the question can{'}t be answered with the tools provided. We find that state-of-the-art tool-calling LMs show significant room for improvement on When2Call, indicating the importance of this benchmark. We also develop a training set for When2Call and leverage the multiple-choice nature of the benchmark to develop a preference optimization training regime, which shows considerably more improvement than traditional fine-tuning. We release the benchmark and training data as well as evaluation scripts."
}

Acknowledgements

Thanks to the NVIDIA When2Call authors and the Qwen team for releasing the dataset and base model used in this experiment.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Harish241412

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table with columns: Property, Value
Property	Value
Base model	Qwen/Qwen2.5-1.5B-Instruct
Parameters	1.5B
Fine-tuning	LoRA + DPO
Dataset	NVIDIA When2Call
Training split	`train_pref`
Final checkpoint	Step 1000
Model format	Merged
Framework	PyTorch, Transformers, PEFT

Dataset

Training was performed using the train_pref split of NVIDIA's When2Call dataset.

The train_pref split contains 9,000 preference-training examples with:

Tool specifications
User messages
Chosen responses
Rejected responses

NVIDIA provides both an SFT dataset (train_sft) and a preference dataset (train_pref); this model uses the preference dataset for DPO training.

The When2Call dataset is synthetic and is licensed under CC BY 4.0.

Training Objective

The objective was to improve the model's tool-use decision boundary.

The model learns to distinguish between:

Tool Call — a tool should be invoked to answer the request.
Request for Information — the request can be handled without invoking a tool.
Cannot Answer — the available tools cannot answer the request.

The preference-training setup encourages the model to prefer appropriate responses over incorrect tool-calling behavior.

Evaluation

The fine-tuned model was evaluated against the original Qwen2.5-1.5B-Instruct model on 300 samples.

The evaluation uses the same 300-sample LLM-as-a-judge subset provided by When2Call. NVIDIA's dataset contains a larger 3,652-sample MCQ test set and a 300-sample LLM-as-a-judge subset.

Results

Table with columns: Metric, Qwen2.5-1.5B-Instruct, Qwen2.5-1.5B + DPO
Metric	Qwen2.5-1.5B-Instruct	Qwen2.5-1.5B + DPO
Intent Accuracy	52.7%	74.0%
Tool Precision	40.8%	72.9%
Tool Recall	93.0%	35.0%
Tool F1	56.7%	47.3%
Argument F1	71.9%	67.7%
Unsupported Tool Calls ↓

Key Results

The largest improvement was in unsupported tool calls:

45.0% → 4.3%

This indicates that DPO substantially reduced inappropriate or hallucinated tool invocations.

Intent accuracy also increased:

52.7% → 74.0%

and tool precision increased:

40.8% → 72.9%

However, this came with a substantial reduction in tool recall:

93.0% → 35.0%

and Tool F1:

56.7% → 47.3%

Therefore, the main behavioral change is a shift toward a more conservative, precision-oriented tool-calling policy.

Confusion Matrix

Qwen2.5-1.5B-Instruct

Table with columns: Ground Truth \ Prediction, Tool, Request, Refusal
Ground Truth \ Prediction	Tool	Request	Refusal
Tool Call	93	3	4
Request For Information	76	23	1
Cannot Answer	59	22	19

Qwen2.5-1.5B + DPO

Table with columns: Ground Truth \ Prediction, Tool, Request, Refusal
Ground Truth \ Prediction	Tool	Request	Refusal
Tool Call	35	65	0
Request For Information	12	86	2
Cannot Answer	1	91	8

The confusion matrix shows that DPO significantly reduced the model's tendency to issue tool calls for requests that should not result in a tool invocation.

Tool Calling Example

Tool Definition

python
tools = [{
    "name": "get_stock_price",
    "description": "Fetch real-time stock price for a given ticker symbol.",
    "parameters": {
        "type": "object",
        "properties": {
            "ticker": {
                "type": "string",
                "description": "The ticker symbol (e.g., AAPL, NVDA)"
            }
        },
        "required": ["ticker"]
    }
}]

User

text
Can you check Nvidia's current stock price?

Model Output

text
<tool_call>
{"name": "get_stock_price", "arguments": {"ticker": "NVDA"}}
</tool_call>

Inference

python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

model.eval()

tools = [{
    "name": "get_stock_price",
    "description": "Fetch real-time stock price for a given ticker symbol.",
    "parameters": {
        "type": "object",
        "properties": {
            "ticker": {
                "type": "string",
                "description": "The ticker symbol (e.g., AAPL, NVDA)"
            }
        },
        "required": ["ticker"]
    }
}]

messages = [
    {
        "role": "user",
        "content": "Can you check Nvidia's current stock price?"
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id
    )

response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=False
)

print(response)

Output Post-Processing

During evaluation, model outputs were passed through a lightweight post-processing step to normalize tool-call formatting.

The post-processing:

Extracts JSON containing name and arguments.
Removes duplicate or nested <tool_call> wrappers.
Normalizes the output to:

text
<tool_call>
{"name": "...", "arguments": {...}}
</tool_call>

If no tool-call JSON is detected, the output is treated as a normal conversational response.

The post-processing step is used for format normalization and evaluation and does not generate a tool call that the model did not produce.

Limitations

The main limitation is the precision-recall trade-off introduced by DPO.

The model is considerably better at avoiding unsupported tool calls, but it also misses a larger proportion of valid tool-call opportunities.

The evaluation also uses a relatively small 300-sample subset, so additional evaluation on larger and more diverse tool-calling benchmarks would be useful.

Intended Use

This model is intended for research and experimentation involving:

Tool calling
Function calling
Tool-selection policies
Preference optimization
DPO
Agentic LLM systems
Small language model alignment

It is not intended to be considered production-ready without additional task-specific evaluation.

Future Work

Recover tool-call recall while maintaining low unsupported-call rates
Experiment with DPO hyperparameters
Improve preference-data construction
Compare DPO against SFT
Evaluate larger Qwen models
Evaluate multi-tool selection
Evaluate multi-step tool calling
Improve argument-generation accuracy
Benchmark inference using vLLM
Evaluate on larger tool-calling benchmarks

Base Model

This model is based on:

Qwen/Qwen2.5-1.5B-Instruct

Please refer to the base model for its original capabilities, license, and usage restrictions.

Dataset Citation

This work uses NVIDIA's When2Call dataset.

Ross, Hayley, Ameya Sunil Mahabaleshwarka, and Yoshi Suhara. "When2Call: When (not) to Call Tools." NAACL 2025.

bibtex
@inproceedings{ross-etal-2025-when2call,
    title = "{W}hen2{C}all: When (not) to Call Tools",
    author = "Ross, Hayley  and
      Mahabaleshwarkar, Ameya Sunil  and
      Suhara, Yoshi",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-long.174/",
    doi = "10.18653/v1/2025.naacl-long.174",
    pages = "3391--3409",
    ISBN = "979-8-89176-189-6",
    abstract = "Leveraging external tools is a key feature for modern Language Models (LMs) to expand their capabilities and integrate them into existing systems. However, existing benchmarks primarily focus on the accuracy of tool calling{---}whether the correct tool is called with the correct parameters{---}and less on evaluating when LMs should (not) call tools. We develop a new benchmark, When2Call, which evaluates tool-calling decision-making: when to generate a tool call, when to ask follow-up questions and when to admit the question can{'}t be answered with the tools provided. We find that state-of-the-art tool-calling LMs show significant room for improvement on When2Call, indicating the importance of this benchmark. We also develop a training set for When2Call and leverage the multiple-choice nature of the benchmark to develop a preference optimization training regime, which shows considerably more improvement than traditional fine-tuning. We release the benchmark and training data as well as evaluation scripts."
}

Acknowledgements

Thanks to the NVIDIA When2Call authors and the Qwen team for releasing the dataset and base model used in this experiment.