Model Details
- Developed by: Matthew Hans
- Model type: Qwen2.5-7B-Instruct causal language model with LoRA adapter fine-tuning
- Base model:
Qwen/Qwen2.5-7B-Instruct
- Training base:
unsloth/Qwen2.5-7B-Instruct-bnb-4bit
- Fine-tuning method: QLoRA with rank-stabilized LoRA (rsLoRA)
- Language: English
- Output format: Hermes FC v1 tool call blocks
- Release: v5a (May 2026)
Intended Use
The model is intended to sit behind a UPS MCP server and produce one structured tool call for each user request. A downstream orchestrator should parse the <tool_call_> block, validate arguments against the MCP tool schema, and then decide whether to call the UPS API.
<tool_call_>
{"name": "track_package", "arguments": {"inquiryNumber": "1Z999AA10123456784"}}
</tool_call_>
The system prompt should include the complete set of UPS MCP tool definitions in Hermes-compatible JSON:
<tools>
[{"type": "function", "function": {"name": "track_package", "parameters": {...}}}]
</tools>
Table with columns: Tool, Category| Tool | Category |
|---|
track_package | Tracking |
validate_address | Address validation |
rate_shipment | Rating |
create_shipment | Shipping |
void_shipment | Shipping |
recover_label |
Evaluation
Held-out accuracy (122 test cases, disjoint from training)
Table with columns: Metric, Score| Metric | Score |
|---|
| Tool selection accuracy | 100% |
| Parameter completeness | 100% |
| JSON validity rate | 100% |
| Format compliance | 100% |
| Payload structural accuracy | 99.06% |
Stress-test suite (~2,000+ inference calls across 9 axes)
Four graders run in parallel on every case:
- Shape grader — required keys present in the tool call
- Value-grounded grader — argument values match what the prompt said
- Nested-structural grader — deep paths populated correctly
- Schema-typed grader — types and enums match the MCP schema
Confidence intervals are Wilson 95%.
Table with columns: Axis, n, Shape, Value-grounded, Silent hallucination| Axis | n | Shape | Value-grounded | Silent hallucination |
|---|
| Strict replay (templated) | 900 | 95.22% | 93.78% | 1.78% |
| OOD-lexicon (disjoint vocab) | 360 | 99.17% | 98.06% | 1.11% |
| Paraphrase (voice/email/sms/typo) | 360 | 96.67% | 68.61% | 28.06% |
Changes from previous release (v3 → v5a)
Table with columns: Axis, v3, v5a, Delta| Axis | v3 | v5a | Delta |
|---|
| Held-out accuracy | 98.5% | 100% | +1.5pp |
| Sibling shape | 89.2% | 88.33% | −0.9pp |
| Multi-turn flow | 66.7% | 80.0% | +13.3pp |
| Multi-turn step | 93.8% | 96.27% | +2.5pp |
Training Data
The dataset is synthetic, generated from UPS MCP server tool schemas.
Table with columns: Split, Samples| Split | Samples |
|---|
| Train | ~1,100 |
| Eval | ~120 |
| Test | 122 |
| Total | ~1,342 |
The v5a training data includes:
- Full replay of the original training set
- 20 targeted augmentation examples focused on multi-turn workflow steps and negative/refusal boundaries
- Oversampling of initially weak tools (3x):
rate_shipment, get_time_in_transit, upload_paperless_document, get_service_center_facilities, get_landed_cost_quote
No UPS API credentials or private customer shipment data are included.
Training Procedure
Table with columns: Setting, Value| Setting | Value |
|---|
| Method | QLoRA SFT |
| Base checkpoint for training | models/ups-tools-v3-merged (v3 champion, merged) |
| LoRA rank | 16 |
| LoRA alpha | 16 |
| rsLoRA | Enabled |
| Target modules | q_proj, k_proj, v_proj, o_proj, , , |
Hardware and Software
Training was run locally on an NVIDIA GeForce RTX 3090 with 24 GB VRAM.
Table with columns: Component, Version| Component | Version |
|---|
| Python | 3.12.3 |
| PyTorch | 2.10.0+cu128 |
| Unsloth | 2026.2.1 |
| PEFT | 0.18.1 |
| TRL | 0.24.0 |
Artifacts
Table with columns: Artifact, Path| Artifact | Path |
|---|
| LoRA adapter | Repo root: adapter_config.json, adapter_model.safetensors, tokenizer files |
| GGUF Q4_K_M export | ups-tools-v5a-Q4_K_M.gguf |
| Ollama Modelfile | Modelfile |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "matt-hans93/ups-tools-qwen2.5-7b"
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
Ollama
Download ups-tools-v5a-Q4_K_M.gguf and Modelfile, then create a local Ollama model:
ollama create ups-tools-qwen25 -f Modelfile
The Modelfile uses a low temperature and enough output budget for nested JSON payloads:
FROM ./ups-tools-v5a-Q4_K_M.gguf
PARAMETER temperature 0.1
PARAMETER top_p 0.9
PARAMETER num_predict 2048
PARAMETER stop "<|im_end|>"
Limitations and Risks
- Paraphrase fragility. Value-grounded accuracy drops to 68.6% on paraphrased prompts (terse SMS-style is worst at ~57%). Free-form text interfaces will see higher error rates.
- Sibling-tool confusion. On adversarial prompts with similar tools, shape accuracy is 88.3%. The most confused pairs are
get_political_divisions vs find_locations and rate_shipment vs create_shipment.
- Incomplete prompts. The model fabricates missing fields instead of asking for clarification. Production orchestrators must validate arguments against the MCP schema before execution.
rate_shipment cross-field combinations. The model can pair UPS service codes with incompatible packaging types. Validate service↔packaging compatibility before submitting.- Multi-step workflow cascading. Step accuracy is 96.3% but full-workflow accuracy is 80% — early errors cascade. Validate each turn in multi-turn flows.
- Specialization to 18 UPS MCP tools. Do not expect generalization to unrelated APIs without additional evaluation.
Out-of-Scope Use
Do not use this model as an autonomous authority for shipping purchases, billing decisions, customs declarations, legal compliance, or cancellation actions. It should generate candidate MCP tool calls only; production systems should validate, log, and gate execution.
License
Apache 2.0. The base Qwen2.5 model is also released under Apache 2.0.