Usage
The prompt must be built with the 12-tool schema and the training system prompt:
You are Aria, a helpful e-commerce assistant. You help customers search for products, track orders, manage returns, answer questions, and resolve issues. Always be polite, concise, and accurate.
utils.py in this repo contains TOOLS_SCHEMA, build_chat_prompt_text
(applies the chat template with the tools) and parse_model_output.
Load the base model in 4-bit and attach this adapter with PeftModel.from_pretrained.
Results
Held-out subset of 300 test examples (120 tool-call, 90 missing-info, 90 no-tool), greedy decoding.
Table with columns: Metric, Base model, This adapter| Metric | Base model | This adapter |
|---|
| valid_format_rate | 0.05 | 1.00 |
| exact_match_rate | 0.003 | 0.643 |
| tool_name_accuracy | 0.083 | 0.625 |
| argument_accuracy | 0.008 | 0.308 |
| no_tool_accuracy | 0.00 | 0.867 |
| missing_info_accuracy | 0.00 | 0.867 |
Limitations
- Only 120 tool-call examples were evaluated (of 248 available), so differences of a few points are within noise.
- Even with the right tool, arguments often differ from the reference, and exact-match scoring counts free-text differences as errors.
- It sometimes outputs
NO_TOOL_CALL on simple search or add-to-cart requests.
- It expects the system prompt above; a different prompt can change behaviour.
- About 7% of order-ID targets refer to IDs that are not in the prompt, because conversation history was not included.
- A DPO variant was tried and not released: it improved nothing measurable on arguments and lost the no-tool and missing-info behaviour.
Training
QLoRA (4-bit NF4), r=16, alpha=32, 300 steps, learning rate 2e-4, on 5,541 filtered
examples from Full-Ecom-Chatbot-Dataset. Mixed training targets: 1,022 tool calls,
2,000 missing-info, 3,000 no-tool (the last two capped for balance).