Sarthak213

doosra-qwen3.5-4b-toolcalls-v2

README

License: apache-2.0

What it does

Who led in dot % during the death overs of the BPL?

Table with columns: Call
	Call
Qwen3.5 4B	`leaderboard(metric="dot_pct", role="bowling", filters={"competition": "BPL", "phase": "death"}, limit=5)`
This model	`leaderboard(metric="dot_pct", role="bowling", filters={"competition": "Bangladesh Premier League", "phase": "death"})`

The base model abbreviates the competition and cuts the table to 5 rows nobody asked for. The fine-tuned model makes the exact call; Doosra runs it, and the answer is written from the result (this one is from the test set):

Rashid Khan is top with 49.07 dot % (Bangladesh Premier League, male, death).

Nasum Ahmed (Khulna Tigers): 47.92

Mohammad Irfan (Rajshahi Royals): 45.61

Files

Table with columns: File
File
`Qwen3.5-4B.Q4_K_M.gguf`	the merged model, 4-bit (2.8 GB), for llama.cpp and Doosra's built-in engine
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (rank 32), to apply to `Qwen/Qwen3.5-4B` yourself
`Qwen3.5-4B.BF16-mmproj.gguf`	the base model's vision projector, unchanged (vision wasn't trained or tested)
`run_log.json`	the training run: settings, losses, timings
`training_loss.png`

Use

In Doosra (3.0 and later): Settings → AI model → Qwen3.5 4B Doosra. The app sends the short prompt and compact tools this model was trained with.

With llama.cpp (any OpenAI-compatible client). The model emits tool calls; your code runs the tools. Give it the system prompt and tool list it was trained with, from the dataset, with thinking off:

bash

llama-server -hf Sarthak213/doosra-qwen3.5-4b-toolcalls-v2:Q4_K_M --jinja --ctx-size 16384

python

import jsonfrom huggingface_hub import hf_hub_downloadfrom openai import OpenAI data = "Sarthak213/doosra-toolcalls"tools = json.load(open(hf_hub_download(data, "tools.json", repo_type="dataset")))system = open(hf_hub_download(data, "system_prompt.txt", repo_type="dataset"), encoding="utf-8").read() client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")reply = client.chat.completions.create(    model="local", temperature=0, tools=tools,    messages=[{"role": "system", "content": system},              {"role": "user", "content": "Who led in dot % during the death overs of the BPL?"}],    extra_body={"chat_template_kwargs": {"enable_thinking": False}})print(reply.choices[0].message.tool_calls)

Send each tool's result back as a tool message and call again; the model finishes with a final_answer call.

Training

Data: Sarthak213/doosra-toolcalls, version 2: 8,000 conversations (6,695 train, 345 validation, 960 test) across 31 kinds of question, single calls and multi-step ones. Questions come from templates over real players, teams, venues and competitions, and 46% were reworded by Qwen3.5 9B; every tool call was executed against the real database, and every answer was written from the results. About 15% of players, teams and venues appear only in the test split, so the test measures names the model never saw. Version 2 adds single-figure questions, about 20 SQL patterns, recovery from a failed call, fixed answer templates, and keeps the end-to-end evaluation's players, grounds and facts out of training (see the dataset card).

Method: LoRA on unsloth/Qwen3.5-4B with Unsloth and TRL, on the language layers (attention and MLP; vision layers frozen), loss on the assistant's turns only (tool calls and answers, not the prompt, question or tool results). Conversations are rendered with Qwen's chat template, the tool list and thinking off, exactly as the app sends them.

Table with columns: Setting
Setting
LoRA	rank 32, alpha 32, dropout 0, all attention and MLP projections (64,929,792 trainable parameters)
Base weights	16-bit (bfloat16), not 4-bit QLoRA
Optimiser	AdamW 8-bit, learning rate 0.0001, cosine schedule, 5 warm-up steps, weight decay 0.01
Batch	16 conversations per step (1 × 16 accumulation), sequences up to 7,424 tokens, above the longest conversation
Length	2 epochs, 838 steps
Hardware	Google Colab, NVIDIA A100-SXM4-40GB (40 GB)
Time	5.2 hours of training in one A100 session (about 22 s a step at batch 1), then a check on held-out questions and the GGUF export; about 31 compute units in all

Loss curves

Validation loss fell from 0.012 (step 100) to 0.0048 (step 500) and ended at 0.0050. Unlike v1, validation loss kept falling into the second epoch (v2's broader data had more to learn) and levelled off around step 500. The adapter at the end of epoch 1 was exported too (…-v2-epoch1); the two tie on the open questions, and this one is faster and more accurate on the auto-graded ones, so it's the one Doosra ships.

Evaluation

Measured with ToolEval, a local evaluation app, each model served by llama.cpp as Doosra runs it (Q4_K_M, thinking off): the base models with the app's full prompt and tool list, the fine-tuned ones with the short prompt they were trained on.

End to end: 40 open questions (Doosra's own evaluation set, written separately from the training templates and kept out of training), each answered through a real tool loop against Doosra's database. 35 are graded automatically on the facts the answer must contain (accepting name variants such as "V Kohli" for "Virat Kohli"); 5 are open-ended (a qualification or definition to choose) and were checked by hand against the database.

End to end (the same 40 questions for every model)

Table with columns: Model, Correct of 40, Auto-graded (35), Reviewed (5), Calls per question, Median s, 90th pct s
Model	Correct of 40	Auto-graded (35)	Reviewed (5)	Calls per question	Median s	90th pct s
Qwen3.5 4B (long prompt)	29	28	1	3.125	30.4	85.1
Qwen3.5 9B (long prompt)	30	27	3	1.95	26.7

It answers more open questions right than both base models while making less than half the base 4B's calls and answering in under half the time. The four auto-graded questions it misses: a comparison that listed only the leader's average (q24), a COUNT(*) where COUNT(DISTINCT player) was needed (q34), and two questions where the definition differs from the reference (Kohli's matches: batting innings 808 against appearances 834; most wins: men's cricket by default against both genders).

Held-out test set (dataset v2: unseen players, teams and venues)

First call of 300 conversations, and every step of 100 with the final answer checked against the tool results.

Table with columns: Model, Right tool, Exact arguments, Conversations all right, Answers when it should, Answers fully grounded
Model	Right tool	Exact arguments	Conversations all right	Answers when it should	Answers fully grounded
Qwen3.5 4B (long prompt)	75%	42%	48%	35%	70%
v2 at epoch 1	100%	94%	97%	100%	100%
v2	100%

Limitations

Doosra's tools only. It learned these 25 tools, their arguments and Doosra's answer style. It isn't a general assistant or a general tool caller, and it expects the short system prompt it was trained with.
Synthetic questions. Training questions come from templates, partly reworded; real users ask in more ways. The end-to-end questions are the check on that.
SQL fine print. It writes valid SQL on Doosra's schema but can miss a rule in the schema notes: asked how many matches were ties, it counted winner IS NULL, which also includes draws and no-results. Check SQL answers to unusual questions.
Names as the data has them. Cricsheet names players as "V Kohli", "JJ Bumrah"; the tools resolve most forms, but unusual spellings may miss.
The data's coverage. Answers are only as complete as Cricsheet: some competitions and teams are thin or missing (for example, there are no Afghanistan matches).

Licence

The model is released under Apache 2.0, like Qwen3.5 4B. The training data's cricket results are from Cricsheet under the Open Data Commons Attribution License.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Sarthak213

Model Tree

Base

Qwen/Qwen3.5-4B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality