SwapniL-pr1117

qwen2.5-0.5b-instructv2-fc

README

License: apache-2.0

Results

Same frozen test sets, same prompt, greedy decoding, same scoring code for all three models.

Table with columns: Metric (%), Base, SFT, DPO (this model) (95% CI)
Metric (%)	Base	SFT	DPO (this model) (95% CI)
Exact match, 2,929 questions ↑	61.6	78.0	80.0 (78.6–81.4)
Function match ↑	89.2	97.3	96.8 (96.2–97.4)
Argument F1 ↑	80.1	90.3	91.2 (90.4–92.0)
Answers `[]` when no tool fits, 300 questions ↑	23.3	0.0	82.3 (77.7 – 86.7)
Wrongly answers `[]` when a tool was needed ↓	0.2	0.0	0.2
Plain chat answered as tool JSON ↓	0.0	0.0	0.0

SFT lifted exact match 61.6% → 78.0%, but xLAM has no "no tool fits" examples, so it learned to always call a tool (abstention 23% → 0%).
DPO brought abstention back to 82% (77% when look-alike tools are present, 88% when the tools are unrelated) and raised exact match to 80.0% (163 questions fixed, 103 broken vs SFT, McNemar p = 0.0003).

Example: "Tell me a joke" with only a get_weather tool → base and SFT call get_weather(city="New York"), this model returns [].

Table with columns: Call pattern, n, Exact match
Call pattern	n	Exact match
single	1,386	84.4%
parallel_same_tool	530	82.5%
multi_distinct_tools	936	73.1%
mixed	77	68.8%

Prompt format (important)

The model was trained on this exact format. It is not Qwen's native <tool_call> format, and it does not read OpenAI-style tools= parameters.

System:

text

You are a function-calling assistant. Your task is to select the appropriate tool and provide the arguments requiredto answer the user's request. Return ONLY valid JSON. The output must be a JSON array of tool calls in this format: [  {    "name": "function_name",    "arguments": {      "argument_name": "value"    }  }] If none of the available tools can answer the request, return an empty array: [] Do not explain your answer.Do not output markdown.Do not invent tools.

User:

text

Available tools: <tools as a JSON list: name, description, parameters> User request: <the request> Return the required tool call(s) as JSON.

Assistant: [{"name": "get_weather", "arguments": {"city": "Pune"}}], or [] if no tool fits.

Usage

transformers

python

import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "SwapniL-pr1117/qwen2.5-0.5b-instructv2-fc"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto") SYSTEM_PROMPT = """<copy the system prompt above>"""tools = [{"name": "get_weather", "description": "Current weather for a city.",          "parameters": {"city": {"type": "str", "description": "City name"}}}]user = (f"Available tools:\n\n{json.dumps(tools, indent=2)}\n\n"        f"User request:\n\nWhat's the weather in Pune?\n\n"        f"Return the required tool call(s) as JSON.") messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True,                                 return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=256)          # greedy is the defaultcalls = json.loads(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))print(calls if calls else "no tool fits this request")

vLLM (OpenAI-compatible server)

bash

vllm serve SwapniL-pr1117/qwen2.5-0.5b-instructv2-fc --max-model-len 4096

Send the same two messages to /v1/chat/completions with temperature=0, then json.loads the reply.

Ollama (GGUF, Q8_0)

bash

ollama run Avahan/qwen2.5_0.5b-instruct-fc-Chat:dpo

No system prompt is baked in: send the system prompt above yourself. (:latest is this DPO model too; :sft is the earlier SFT-only version.)

Decoding

Use greedy decoding. generation_config.json already sets do_sample=False; with OpenAI-compatible servers pass temperature=0. Sampling and repetition penalties hurt JSON accuracy. Cap max_new_tokens (~256-512).

How it behaves

With the system prompt above: answers with a JSON array of tool calls, or [] when none of the tools fit. It's a tool-calling specialist, not a chatbot.
Without it (plain chat): answers in normal text (0% of 50 everyday prompts turned into tool JSON), but chat quality is that of a 0.5B model: don't rely on it for facts. A common pattern (used in the GitHub gateway): send the request with the tool prompt first; if the model returns [], answer it as plain chat.
It does not execute anything. It only writes the call; your code runs the tool. Validate the JSON against your tool schemas before running it, and ask the user before running tools with side effects.

Limitations

Abstention isn't perfect: it still calls a tool in ~18% of no-tool-fits cases, more often when a similar-looking tool exists. Keep a guard (e.g. check that argument values appear in the request).
Most remaining errors are wrong argument values (16% of questions), often optional/default arguments where the training data itself is inconsistent.
Rarely (~0.3%) it loops on long repeated values until max_new_tokens; cap the output length.
Unseen tools: almost every tool in the test set also appears in training, so the score measures new requests for familiar tools. Accuracy on completely new tool schemas may be lower.
Single turn only: it picks the call; it doesn't read tool results or write a final natural-language answer. English only.

Training

SFT: QLoRA (4-bit base) with Unsloth, r=16, alpha=32, all linear layers, 1 epoch on ~48.7k xLAM examples, loss on the answer only. Merged into bf16.
DPO: 1,753 preference pairs built automatically from held-out xLAM prompts (never used in SFT):
- mistake pairs: chosen = correct answer, rejected = the SFT model's own most common wrong answer (sampled 4× at temperature 0.8)
- abstain pairs: tools that can't answer the request; chosen = [], rejected = the call the model invented
- balance pairs: same requests with the right tools; chosen = the call, rejected = [] (so it doesn't learn "always say []")
New LoRA (r=16, alpha=32) on the merged SFT model; reference = the same model with the adapter off. TRL DPO, beta 0.1, RPO alpha 1.0 (keeps correct answers likely), lr 2e-5, 2 epochs. Held-out preference accuracy 0.49 → 0.78. Merged into bf16.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

SwapniL-pr1117

Model Tree

Base

Qwen/Qwen2.5-0.5B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results

Same frozen test sets, same prompt, greedy decoding, same scoring code for all three models.

Table with columns: Metric (%), Base, SFT, DPO (this model) (95% CI)
Metric (%)	Base	SFT	DPO (this model) (95% CI)
Exact match, 2,929 questions ↑	61.6	78.0	80.0 (78.6–81.4)
Function match ↑	89.2	97.3	96.8 (96.2–97.4)
Argument F1 ↑	80.1	90.3	91.2 (90.4–92.0)
Answers `[]` when no tool fits, 300 questions ↑	23.3	0.0	82.3 (77.7 – 86.7)
Wrongly answers `[]` when a tool was needed ↓	0.2	0.0	0.2
Plain chat answered as tool JSON ↓	0.0	0.0	0.0

SFT lifted exact match 61.6% → 78.0%, but xLAM has no "no tool fits" examples, so it learned to always call a tool (abstention 23% → 0%).
DPO brought abstention back to 82% (77% when look-alike tools are present, 88% when the tools are unrelated) and raised exact match to 80.0% (163 questions fixed, 103 broken vs SFT, McNemar p = 0.0003).

Example: "Tell me a joke" with only a get_weather tool → base and SFT call get_weather(city="New York"), this model returns [].

Table with columns: Call pattern, n, Exact match
Call pattern	n	Exact match
single	1,386	84.4%
parallel_same_tool	530	82.5%
multi_distinct_tools	936	73.1%
mixed	77	68.8%

Prompt format (important)

The model was trained on this exact format. It is not Qwen's native <tool_call> format, and it does not read OpenAI-style tools= parameters.

System:

text

You are a function-calling assistant.Your task is to select the appropriate tool and provide the arguments requiredto answer the user's request.Return ONLY valid JSON.The output must be a JSON array of tool calls in this format:[  {    "name": "function_name",    "arguments": {      "argument_name": "value"    }  }]If none of the available tools can answer the request, return an empty array: []Do not explain your answer.Do not output markdown.Do not invent tools.

You are a function-calling assistant. Your task is to select the appropriate tool and provide the arguments requiredto answer the user's request. Return ONLY valid JSON. The output must be a JSON array of tool calls in this format: [  {    "name": "function_name",    "arguments": {      "argument_name": "value"    }  }] If none of the available tools can answer the request, return an empty array: [] Do not explain your answer.Do not output markdown.Do not invent tools.

User:

text

Available tools:<tools as a JSON list: name, description, parameters>User request:<the request>Return the required tool call(s) as JSON.

Available tools: <tools as a JSON list: name, description, parameters> User request: <the request> Return the required tool call(s) as JSON.

Assistant: [{"name": "get_weather", "arguments": {"city": "Pune"}}], or [] if no tool fits.

Usage

transformers

python

import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "SwapniL-pr1117/qwen2.5-0.5b-instructv2-fc"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto") SYSTEM_PROMPT = """<copy the system prompt above>"""tools = [{"name": "get_weather", "description": "Current weather for a city.",          "parameters": {"city": {"type": "str", "description": "City name"}}}]user = (f"Available tools:\n\n{json.dumps(tools, indent=2)}\n\n"        f"User request:\n\nWhat's the weather in Pune?\n\n"        f"Return the required tool call(s) as JSON.") messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True,                                 return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=256)          # greedy is the defaultcalls = json.loads(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))print(calls if calls else "no tool fits this request")

import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "SwapniL-pr1117/qwen2.5-0.5b-instructv2-fc"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto") SYSTEM_PROMPT = """<copy the system prompt above>"""tools = [{"name": "get_weather", "description": "Current weather for a city.",          "parameters": {"city": {"type": "str", "description": "City name"}}}]user = (f"Available tools:\n\n{json.dumps(tools, indent=2)}\n\n"        f"User request:\n\nWhat's the weather in Pune?\n\n"        f"Return the required tool call(s) as JSON.") messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True,                                 return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=256)          # greedy is the defaultcalls = json.loads(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))print(calls if calls else "no tool fits this request")

vLLM (OpenAI-compatible server)

bash

vllm serve SwapniL-pr1117/qwen2.5-0.5b-instructv2-fc --max-model-len 4096

Send the same two messages to /v1/chat/completions with temperature=0, then json.loads the reply.

Ollama (GGUF, Q8_0)

bash

ollama run Avahan/qwen2.5_0.5b-instruct-fc-Chat:dpo

No system prompt is baked in: send the system prompt above yourself. (:latest is this DPO model too; :sft is the earlier SFT-only version.)

Decoding

How it behaves

With the system prompt above: answers with a JSON array of tool calls, or [] when none of the tools fit. It's a tool-calling specialist, not a chatbot.
Without it (plain chat): answers in normal text (0% of 50 everyday prompts turned into tool JSON), but chat quality is that of a 0.5B model: don't rely on it for facts. A common pattern (used in the GitHub gateway): send the request with the tool prompt first; if the model returns [], answer it as plain chat.
It does not execute anything. It only writes the call; your code runs the tool. Validate the JSON against your tool schemas before running it, and ask the user before running tools with side effects.

Limitations

Abstention isn't perfect: it still calls a tool in ~18% of no-tool-fits cases, more often when a similar-looking tool exists. Keep a guard (e.g. check that argument values appear in the request).
Most remaining errors are wrong argument values (16% of questions), often optional/default arguments where the training data itself is inconsistent.
Rarely (~0.3%) it loops on long repeated values until max_new_tokens; cap the output length.
Unseen tools: almost every tool in the test set also appears in training, so the score measures new requests for familiar tools. Accuracy on completely new tool schemas may be lower.
Single turn only: it picks the call; it doesn't read tool results or write a final natural-language answer. English only.

Training

SFT: QLoRA (4-bit base) with Unsloth, r=16, alpha=32, all linear layers, 1 epoch on ~48.7k xLAM examples, loss on the answer only. Merged into bf16.
DPO: 1,753 preference pairs built automatically from held-out xLAM prompts (never used in SFT):
- mistake pairs: chosen = correct answer, rejected = the SFT model's own most common wrong answer (sampled 4× at temperature 0.8)
- abstain pairs: tools that can't answer the request; chosen = [], rejected = the call the model invented
- balance pairs: same requests with the right tools; chosen = the call, rejected = [] (so it doesn't learn "always say []")
New LoRA (r=16, alpha=32) on the merged SFT model; reference = the same model with the adapter off. TRL DPO, beta 0.1, RPO alpha 1.0 (keeps correct answers likely), lr 2e-5, 2 epochs. Held-out preference accuracy 0.49 → 0.78. Merged into bf16.