jgalego

amalia-desenrasca-9b

README

License: apache-2.0

🚀 Usage

The tools go in the system message. The model answers with <tool_call>{...}</tool_call>, or in plain text when no tool fits. Send the tool result back with the tool role.

python

import json from transformers import AutoModelForCausalLM, AutoTokenizer repo = "jgalego/amalia-desenrasca-9b"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto") tools = [    {        "name": "get_weather",        "description": "Estado do tempo atual numa cidade",        "parameters": {            "type": "object",            "properties": {"city": {"type": "string", "description": "Nome da cidade"}},            "required": ["city"],        },    }]system = "Tens acesso \u00e0s ferramentas abaixo. Para usar uma, responde apenas com \u003ctool_call\u003e{\"name\": \"nome\", \"arguments\": {\"argumento\": \"valor\"}}\u003c/tool_call\u003e. Se nenhuma ferramenta servir, diz com educa\u00e7\u00e3o que n\u00e3o podes ajudar."system += "\n<tools>\n" + "\n".join(json.dumps(tool, ensure_ascii=False) for tool in tools) + "\n</tools>"messages = [    {"role": "system", "content": system},    {"role": "user", "content": "Está a chover no Porto?"},]inputs = tokenizer.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)output = model.generate(**inputs, max_new_tokens=128, do_sample=False)print(tokenizer.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True))# Expected: <tool_call>{"name": "get_weather", "arguments": {"city": "Porto"}}</tool_call>

🏋️ Training

Why: AMALIA's chat template has no tool support, and there is no public pt-PT tool-calling data. The tools go in the system message as plain text, and the template renders the tool role as is.
Data: 2,700 conversations from jgalego/function-calling-pt-pt: 1,000 xLAM, 700 ToolACE, 400 Hermes and 600 When2Call refusals. Another 300 generated conversations use the demo's tools for parallel calls, relative dates, percentages, currencies and unsupported forecasts. User turns and final answers are pt-PT; tool schemas, calls and results stay in English.
Pairs: Each assistant turn is one prompt/completion pair. Calls may be parallel. Only When2Call refusals are used because its follow-up questions often ask for information the user already supplied.
Method: LoRA (r=16, alpha=32, all attention and MLP projections) on the bf16 model with Unsloth. 2 epochs, learning rate 2e-4 with a cosine schedule, effective batch size 16. The adapter is merged into the base model.

📊 Evaluation

BFCL pt-PT uses the BFCL v3 single-turn questions translated into pt-PT, with the original functions and answer keys. Scores are shares of questions answered correctly.

Table with columns: Metric, AMALIA, v1, Desenrasca
Metric	AMALIA	v1	Desenrasca
BFCL pt-PT	0.589	0.554	0.775
BFCL English	0.592	0.526	0.762
pt-PT answerable	0.636	0.544	0.743
pt-PT irrelevance	0.484	0.571	0.844
pt-PT parallel	0.528	0.090	0.802

The second evaluation uses 200 held-out Glaive-derived conversations with pt-PT user turns: 96 structured call references and 104 without a structured call.

Table with columns: Metric, AMALIA, v1, Desenrasca
Metric	AMALIA	v1	Desenrasca
Valid call	0.990	0.938	1.000
Right function name	0.990	0.938	1.000
Exact arguments	0.635	0.875	0.844
No call when the reference made none	0.423	0.808	0.654

The last row is reference agreement, not a measure of whether no tool fits. A manual audit of its 104 rows found:

Table with columns: Reference behavior, Rows, Desenrasca made no call
Reference behavior	Rows	Desenrasca made no call
Required details missing	41	29
No available tool fits	36	36
Generic conversation	2	2
A tool fits despite the no-call reference	23	1
City names supplied to coordinate-only tools	2	0

On the 41 missing-detail rows, 23 outputs explicitly request more information. Twelve call a tool instead; eleven use arbitrary defaults or placeholders and one infers Apple's AAPL ticker.

⚠️ Limitations

Missing arguments: The model sometimes invents required values instead of asking the user. This happened on 11 of 41 audited missing-detail requests.
Coordinates: It inferred city-center coordinates in two distance questions whose tools accepted coordinates only.
Translated data: The pt-PT data is machine-translated by AMALIA itself, and nobody has reviewed it.
English arguments: Values like city names follow the English source data, so expect "New York" rather than "Nova Iorque".

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

jgalego

Model Tree

Base

amalia-llm/AMALIA-9B-0626-SFT

Fine-tuned

this model

Input Modalities

Text

Output Modalities