ak3ra

sunflower-qwen3.5-9b-tool-sft-poc

README

License: apache-2.0

Tool contract

The inference host must expose this JSON Schema when applying the chat template:

json
{
  "type": "function",
  "function": {
    "name": "query_external_llm",
    "description": "Send an English prompt to a strong external language model.",
    "parameters": {
      "type": "object",
      "properties": {
        "prompt": {
          "type": "string",
          "description": "The complete request in English."
        }
      },
      "required": ["prompt"]
    }
  }
}

The weights generate the call but do not execute it. The host must:

Parse the generated call.
Send prompt to a local or cloud model.
Append the English response as a role: tool message.
Invoke Sunflower again to generate the localized answer.

How tool calling works

text
Localized user request
        |
        v
Sunflower emits query_external_llm(prompt: string)
        |
        v
Host sends the English prompt to the external model
        |
        v
Host appends query_external_llm(response: string)
        |
        v
Sunflower answers in the user's language

Illustrative Luganda exchange

This example uses the exact interface and message shapes used in training. The wording is illustrative rather than a claimed held-out generation.

User (role: user)

text
Mu tterekero mulimu obusanduuko 17, era buli kasanduuko kalimu eccupa 24.
Eccupa 289 bwe zitundibwa, zisigala mmeka?

Assistant tool call — canonical structured message

json
{
  "role": "assistant",
  "content": "",
  "tool_calls": [
    {
      "type": "function",
      "function": {
        "name": "query_external_llm",
        "arguments": {
          "prompt": "In a warehouse there are 17 boxes, and each box has 24 bottles. After 289 bottles are sold, how many bottles remain?"
        }
      }
    }
  ]
}

Qwen renders and generates the same call in its model-specific textual form:

xml
<tool_call>
<function=query_external_llm>
<parameter=prompt>
In a warehouse there are 17 boxes, and each box has 24 bottles. After 289 bottles are sold, how many bottles remain?
</parameter>
</function>
</tool_call>

Tool result appended by the host (role: tool)

json
{
  "role": "tool",
  "name": "query_external_llm",
  "content": "{\"response\":\"17 × 24 = 408 bottles. 408 − 289 = 119 bottles remain.\"}"
}

Final assistant response (role: assistant)

text
Obusanduuko 17 × eccupa 24 = eccupa 408. Bwe tuggyako eccupa 289,
eccupa 119 ze zisigala.

For a direct request, the model skips the tool exchange and immediately emits a localized assistant response, even though the tool remains available.

Training

Dataset: Sunbird/sunflower-posttrain-data, config sft_tool_calling
Dataset revision: a2cfa0b9b834efe0c013778c6802ddf4ad2a5955
27 languages: 26 Uganda-focused languages plus English
Explicit source-isolated train and test splits
864 training rows and 216 test rows
Target routes balanced 50/50 between tool and direct in each language
25% multi-turn rows
One epoch of full-parameter BF16 SFT over all 9.41B parameters
8-bit AdamW, learning rate 1e-5
Micro-batch 1, effective batch 16, 54 optimizer steps
Maximum sequence length 2,304; longest row 1,692; zero rows dropped

Five languages were excluded because too few translations survived quality control: bfa, kdj, kpz, luc, and mhi.

V2 evaluation

Deterministic greedy evaluation used the source-isolated 216-row test split: 108 tool-positive and 108 direct examples. For valid tool-positive calls, the host replayed the stored English result before asking the model for its final localized answer. No live external API was called.

Table with columns: Metric, Result
Metric	Result
Routing recall	0.7593 (82/108)
Routing precision	0.5503
Routing F1	0.6381
Direct-response specificity	0.3796 (41/108)
Emitted tool calls	156
Valid-call rate	0.9551 (149/156)
Correct tool-name rate	1.0000
English relay-prompt chrF	0.4525
English relay-prompt numeric exact

Diagnostic routing slices

Table with columns: Slice, Result
Slice	Result
Factual-QA tool recall	1.000
Reasoning tool recall	0.964
Multiple-choice tool recall	0.700
Quantitative tool recall	0.537
Direct -> tool recall	0.296
Direct -> direct specificity	0.741
Single-turn direct specificity	0.259

The model learned the tool interface more successfully than the routing policy: almost every emitted call is structurally valid and every valid call uses the correct function, but 67 of 108 direct-labelled examples still emitted a tool call.

Some measured overcalling comes from supervision problems. Broad seed rules labelled requests beginning with words such as write, create, or hello as direct even when the rest of the request asked for code or factual analysis. The test distribution is also skewed toward creative requests and contains no tool_to_direct or tool_to_tool histories. These labels and splits need to be corrected before the next training iteration.

Final-answer chrF is reported for reproducibility but should not be treated as a reliable quality measure for open-ended creative or conversational answers.

Historical v1 result

An earlier dataset revision used a positional 104-row holdout and achieved routing F1 0.9346, direct specificity 0.8868, and valid-call rate 1.0. That result proved the protocol could be learned, but the split was not source-isolated and is not directly comparable with the harder v2 evaluation.

Prompt construction

Use the tokenizer's native chat template, supply the tool schema, and disable thinking:

python
from transformers import AutoTokenizer

model_id = "ak3ra/sunflower-qwen3.5-9b-tool-sft-poc"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

query_external_llm_schema = {
    "type": "function",
    "function": {
        "name": "query_external_llm",
        "description": "Send an English prompt to a strong external language model.",
        "parameters": {
            "type": "object",
            "properties": {"prompt": {"type": "string"}},
            "required": ["prompt"],
        },
    },
}

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "<request in the user's language>"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=[query_external_llm_schema],
    enable_thinking=False,
    tokenize=False,
    add_generation_prompt=True,
)

See sunflower/tool_eval.py and scripts/eval_tool_calling.py for parsing and two-stage replay examples. The full frozen experiment report is in reports/tool-calling-poc/REPORT.md.

Next iteration

The proposed next step is cost-aware reinforcement learning:

text
reward = answer quality - lambda * tool calls

Easy requests can earn the highest reward by answering directly. Difficult requests can pay the small tool-call cost when delegation improves the answer. Before RL, the routing labels, per-task balance, and multi-turn test coverage need to be corrected.

Limitations

This is a proof-of-concept checkpoint, not a production release.
Routing is over-eager and fails the agreed acceptance gates.
The current dataset covers 27 rather than all 32 intended languages.
Some direct/tool labels need policy correction.
The staging repository name is mutable. The evaluated weight revision before this documentation update was 837715f226d8f5bfc94bebaba702558495da4d61.
A deployment host must implement tool execution, safety policy, timeouts, retries, authentication, observability, and cost controls.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider