Sarthak213

doosra-qwen3.5-4b-toolcalls-b-step350

README

License: apache-2.0

What it does

Who led in dot % during the death overs of the BPL?

Table with columns: Call
	Call
Qwen3.5 4B	`leaderboard(metric="dot_pct", role="bowling", filters={"competition": "BPL", "phase": "death"}, limit=5)`
This model	`leaderboard(metric="dot_pct", role="bowling", filters={"competition": "Bangladesh Premier League", "phase": "death"})`

The base model abbreviates the competition and cuts the table to 5 rows nobody asked for. The fine-tuned model makes the exact call; Doosra runs it, and the answer is written from the result (this one is from the test set):

Rashid Khan is top with 49.07 dot % (Bangladesh Premier League, male, death).

Nasum Ahmed (Khulna Tigers): 47.92

Mohammad Irfan (Rajshahi Royals): 45.61

Files

Table with columns: File
File
`Qwen3.5-4B.Q4_K_M.gguf`	the merged model, 4-bit (2.8 GB), for llama.cpp and Doosra's built-in engine
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (rank 32), to apply to `Qwen/Qwen3.5-4B` yourself
`Qwen3.5-4B.BF16-mmproj.gguf`	the base model's vision projector, unchanged (vision wasn't trained or tested)
`run_log.json`	the training run: settings, losses, timings
`training_loss.png`

Use

In Doosra (3.0 and later): Settings → AI model → Qwen3.5 4B Doosra. The app sends the short prompt and compact tools this model was trained with.

With llama.cpp (any OpenAI-compatible client). The model emits tool calls; your code runs the tools. Give it the system prompt and tool list it was trained with, from the dataset, with thinking off:

bash

llama-server -hf Sarthak213/doosra-qwen3.5-4b-toolcalls-b-step350:Q4_K_M --jinja --ctx-size 16384

python

import jsonfrom huggingface_hub import hf_hub_downloadfrom openai import OpenAI data = "Sarthak213/doosra-toolcalls"tools = json.load(open(hf_hub_download(data, "tools.json", repo_type="dataset")))system = open(hf_hub_download(data, "system_prompt.txt", repo_type="dataset"), encoding="utf-8").read() client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")reply = client.chat.completions.create(    model="local", temperature=0, tools=tools,    messages=[{"role": "system", "content": system},              {"role": "user", "content": "Who led in dot % during the death overs of the BPL?"}],    extra_body={"chat_template_kwargs": {"enable_thinking": False}})print(reply.choices[0].message.tool_calls)

Send each tool's result back as a tool message and call again; the model finishes with a final_answer call.

Training

Data: Sarthak213/doosra-toolcalls, version 1: 7,000 conversations (5,854 train, 306 validation, 840 test) across 29 kinds of question, single calls and multi-step ones. Questions come from templates over real players, teams, venues and competitions, and 43% were reworded by Qwen3.5 9B; every tool call was executed against the real database, and every answer was written from the results. About 15% of players, teams and venues appear only in the test split, so the test measures names the model never saw.

Method: LoRA on unsloth/Qwen3.5-4B with Unsloth and TRL, on the language layers (attention and MLP; vision layers frozen), loss on the assistant's turns only (tool calls and answers, not the prompt, question or tool results). Conversations are rendered with Qwen's chat template, the tool list and thinking off, exactly as the app sends them.

Table with columns: Setting
Setting
LoRA	rank 32, alpha 32, dropout 0, all attention and MLP projections (64,929,792 trainable parameters)
Base weights	16-bit (bfloat16), not 4-bit QLoRA
Optimiser	AdamW 8-bit, learning rate 0.0001, cosine schedule, 5 warm-up steps, weight decay 0.01
Batch	16 conversations per step (2 × 8 accumulation), sequences up to 6,144 tokens (4.5% of conversations were cut short at the end, where the answer is; fixed in v2)
Length	350 steps (about the end of epoch 1) of a 2-epoch run
Hardware	Google Colab, NVIDIA A100-SXM4-40GB (40 GB)
Time	the first 350 steps of run B: about 2 hours of A100 time, including a disconnect at step 195 that resumed from the step-150 checkpoint

Loss curves

Validation loss fell from 0.035 (step 50) to 0.0056 (step 300) and ended at 0.0059.

Evaluation

Measured with ToolEval, a local evaluation app, each model served by llama.cpp as Doosra runs it (Q4_K_M, thinking off): the base models with the app's full prompt and tool list, the fine-tuned ones with the short prompt they were trained on.

End to end: 40 open questions (Doosra's own evaluation set, written separately from the training templates and kept out of training), each answered through a real tool loop against Doosra's database. 35 are graded automatically on the facts the answer must contain (accepting name variants such as "V Kohli" for "Virat Kohli"); 5 are open-ended (a qualification or definition to choose) and were checked by hand against the database.

End to end (the same 40 questions for every model)

Table with columns: Model, Correct of 40, Auto-graded (35), Reviewed (5), Calls per question, Median s, 90th pct s
Model	Correct of 40	Auto-graded (35)	Reviewed (5)	Calls per question	Median s	90th pct s
Qwen3.5 4B (long prompt)	29	28	1	3.125	30.4	85.1
Qwen3.5 9B (long prompt)	30	27	3	1.95	26.7

v1 was near-perfect on its templated tests but lost end to end: it couldn't write SQL (the compact tool list had dropped the schema), answered with a template's headline figure instead of the one asked (sixes, balls faced), made one call and stopped where a second was needed, and had learned answers with no figures from a template bug. Dataset v2 fixed each of these.

Held-out test set (dataset v1: unseen players, teams and venues)

First call of 300 conversations, and every step of 100 with the final answer checked against the tool results.

Table with columns: Model, Right tool, Exact arguments, Conversations all right, Answers when it should, Answers fully grounded
Model	Right tool	Exact arguments	Conversations all right	Answers when it should	Answers fully grounded
Qwen3.5 4B (long prompt)	82%	52%	57%	23%	52%
Qwen3.5 9B (long prompt)	88%	57%	56%	90%	69%
v1, run A	98%

Limitations

Doosra's tools only. It learned these 25 tools, their arguments and Doosra's answer style. It isn't a general assistant or a general tool caller, and it expects the short system prompt it was trained with.
Synthetic questions. Training questions come from templates, partly reworded; real users ask in more ways. The end-to-end questions are the check on that.
SQL fine print. It writes valid SQL on Doosra's schema but can miss a rule in the schema notes: asked how many matches were ties, it counted winner IS NULL, which also includes draws and no-results. Check SQL answers to unusual questions.
Names as the data has them. Cricsheet names players as "V Kohli", "JJ Bumrah"; the tools resolve most forms, but unusual spellings may miss.
The data's coverage. Answers are only as complete as Cricsheet: some competitions and teams are thin or missing (for example, there are no Afghanistan matches).

Licence

The model is released under Apache 2.0, like Qwen3.5 4B. The training data's cricket results are from Cricsheet under the Open Data Commons Attribution License.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Sarthak213

Model Tree

Base

Qwen/Qwen3.5-4B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality