Chloemp

smollm2-1.7b-instruct-xlam-lora

README

License: apache-2.0

Quick start

The adapter is ~72 MB; the base model is downloaded separately. The system prompt format matters — it is the one used in training.

python

import json, torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer ADAPTER  = "Chloemp/smollm2-1.7b-instruct-xlam-lora"REVISION = "988889136d65e50bad0ec9dd974dfbf227752a2a" tok = AutoTokenizer.from_pretrained(ADAPTER, revision=REVISION)base = AutoModelForCausalLM.from_pretrained(    "HuggingFaceTB/SmolLM2-1.7B-Instruct", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, ADAPTER, revision=REVISION).merge_and_unload().eval() tools = json.dumps([    {"name": "get_weather",     "description": "Get the current weather for a location.",     "parameters": {"city": {"description": "City name", "type": "str"},                    "unit": {"description": "celsius or fahrenheit", "type": "str", "default": "celsius"}}},    {"name": "search_flights",     "description": "Search flights between two cities on a date.",     "parameters": {"origin": {"description": "Departure city", "type": "str"},                    "destination": {"description": "Arrival city", "type": "str"},                    "date": {"description": "YYYY-MM-DD", "type": "str"}}},]) messages = [    {"role": "system", "content": f"You are a function-calling assistant. Available tools:\n{tools}"},    {"role": "user", "content": "What's the weather in Lyon in fahrenheit, and find me a flight from Paris to Lisbon on 2026-11-03."},] inputs = tok.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) with torch.no_grad():    out = model.generate(**inputs, max_new_tokens=512, do_sample=False,                         pad_token_id=tok.pad_token_id or tok.eos_token_id) print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))# [{"name": "get_weather", "arguments": {"city": "Lyon", "unit": "fahrenheit"}},#  {"name": "search_flights", "arguments": {"origin": "Paris", "destination": "Lisbon", "date": "2026-11-03"}}]

Greedy decoding is what was evaluated. For batched generation, set tok.padding_side = "left" — a decoder generates from the last position, and right padding makes that a <pad>.

What it is good at

Table with columns: Level, Score
Level	Score
1. Valid JSON	99.75%
2. Schema `[{name, arguments}]`	99.75%
3. No hallucinated tool	99.71%
4. Correct call count	98.83%
5. Correct tool names	98.29%
6. Correct argument values (= exact match)	83.47%

Structure is essentially solved: you can parse its output without a fallback path. What remains is argument values. The same model prompted rather than fine-tuned produces a valid schema only 52.65% of the time — format compliance is what the fine-tune buys.

Training

LoRA r=16, α=32, dropout 0.05 on all linear layers (18.1M trainable of 1.73B, 1.05%), lr 2e-4, 3% warmup, 1 epoch (3,563 steps), effective batch 16 (8 × 2 accumulation), bf16, gradient checkpointing, max length 2048, on one A100. Trained in two jobs (interrupted at step 1500, resumed from the full Hub checkpoint).

Identical in every respect to smollm2-1.7b-xlam-lora except the starting checkpoint. Starting from Instruct rather than the base model is worth +8.4 points (paired McNemar, 234 vs 34 discordant pairs, p ≈ 7e-38).

How it was evaluated

2,395 held-out examples from the same dataset, after removing 105 contaminated rows (93 queries appeared on both sides of the split). Greedy decoding, left padding, max_new_tokens=512 (the longest reference in the eval set is 410 tokens), batch size 32, bf16 on one A100. Scores attach to the pinned revision above, not to main.

Scoring is six nested levels, each strictly harder than the last, so a failure is located rather than just counted. Level 6 is exact match after normalising types (5 == "5"), key order and call order.

Table with columns: Model, Exact match (n=2,395), 95% CI
Model	Exact match (n=2,395)	95% CI
smollm2-1.7b-instruct-xlam-lora	83.47%	[81.92, 84.90]
smollm2-1.7b-xlam-lora	75.11%	[73.34, 76.81]
smollm2-1.7b-xlam-qlora	72.61%	[70.79, 74.36]
SmolLM2-1.7B-Instruct, few-shot prompting, no fine-tune	39.58%

For reference, Qwen3-235B-A22B-Instruct prompted with the same format block scores 78.33% on a fixed 300-example subset — statistically indistinguishable from the Instruct-based LoRA on those same 300 (80.00%, paired McNemar p = 0.50).

Limitations

Known tools only. A random split of xLAM means nearly every tool in the eval set also appears in training. These numbers measure reliable calling of a known toolset on new phrasing. On 20 hand-written queries against four tools absent from xLAM, the fine-tunes score 60–70% against 80% for the un-tuned Instruct model: fine-tuning on xLAM costs some generalisation to unseen tools.
Irrelevance detection is unmeasured. Nothing here tests whether the model abstains when no available tool answers the request. Expect it to call something anyway.
Single-turn, English, xLAM-style schemas. No multi-turn, no tool results fed back.
Ground truth is imperfect, and now measured. An LLM-judge audit flagged exact-match failures whose reference cannot be derived from the request (a city, an ID, a ticker appearing nowhere in the prompt). 83 flagged cases were then reviewed by hand, blind to the judge's verdict: its precision is 80.0% on self-consistent flags [67.0, 88.8] but only 27.3% on self-contradictory ones [15.1, 44.2]. Correcting the raw 20.5% by those rates gives 15.0% of exact-match failures [12.7, 16.6] — about one in seven — or 4.2% of all held-out examples. Only flagged cases were reviewed, so that is a floor, not a ceiling: unusable references the judge missed would push it higher. True accuracy is therefore somewhat higher than exact match suggests.

License and attribution

Weights follow the base model's Apache-2.0 licence. Training data is Salesforce/xlam-function-calling-60k, CC-BY-4.0 — attribute Salesforce AI Research if you use anything derived from it.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Chloemp

Model Tree

Base

HuggingFaceTB/SmolLM2-1.7B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quick start

The adapter is ~72 MB; the base model is downloaded separately. The system prompt format matters — it is the one used in training.

python

import json, torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer ADAPTER  = "Chloemp/smollm2-1.7b-instruct-xlam-lora"REVISION = "988889136d65e50bad0ec9dd974dfbf227752a2a" tok = AutoTokenizer.from_pretrained(ADAPTER, revision=REVISION)base = AutoModelForCausalLM.from_pretrained(    "HuggingFaceTB/SmolLM2-1.7B-Instruct", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, ADAPTER, revision=REVISION).merge_and_unload().eval() tools = json.dumps([    {"name": "get_weather",     "description": "Get the current weather for a location.",     "parameters": {"city": {"description": "City name", "type": "str"},                    "unit": {"description": "celsius or fahrenheit", "type": "str", "default": "celsius"}}},    {"name": "search_flights",     "description": "Search flights between two cities on a date.",     "parameters": {"origin": {"description": "Departure city", "type": "str"},                    "destination": {"description": "Arrival city", "type": "str"},                    "date": {"description": "YYYY-MM-DD", "type": "str"}}},]) messages = [    {"role": "system", "content": f"You are a function-calling assistant. Available tools:\n{tools}"},    {"role": "user", "content": "What's the weather in Lyon in fahrenheit, and find me a flight from Paris to Lisbon on 2026-11-03."},] inputs = tok.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) with torch.no_grad():    out = model.generate(**inputs, max_new_tokens=512, do_sample=False,                         pad_token_id=tok.pad_token_id or tok.eos_token_id) print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))# [{"name": "get_weather", "arguments": {"city": "Lyon", "unit": "fahrenheit"}},#  {"name": "search_flights", "arguments": {"origin": "Paris", "destination": "Lisbon", "date": "2026-11-03"}}]

Greedy decoding is what was evaluated. For batched generation, set tok.padding_side = "left" — a decoder generates from the last position, and right padding makes that a <pad>.

What it is good at

Table with columns: Level, Score
Level	Score
1. Valid JSON	99.75%
2. Schema `[{name, arguments}]`	99.75%
3. No hallucinated tool	99.71%
4. Correct call count	98.83%
5. Correct tool names	98.29%
6. Correct argument values (= exact match)	83.47%

Training

How it was evaluated

Table with columns: Model, Exact match (n=2,395), 95% CI
Model	Exact match (n=2,395)	95% CI
smollm2-1.7b-instruct-xlam-lora	83.47%	[81.92, 84.90]
smollm2-1.7b-xlam-lora	75.11%	[73.34, 76.81]
smollm2-1.7b-xlam-qlora	72.61%	[70.79, 74.36]
SmolLM2-1.7B-Instruct, few-shot prompting, no fine-tune	39.58%

Limitations

Known tools only. A random split of xLAM means nearly every tool in the eval set also appears in training. These numbers measure reliable calling of a known toolset on new phrasing. On 20 hand-written queries against four tools absent from xLAM, the fine-tunes score 60–70% against 80% for the un-tuned Instruct model: fine-tuning on xLAM costs some generalisation to unseen tools.
Irrelevance detection is unmeasured. Nothing here tests whether the model abstains when no available tool answers the request. Expect it to call something anyway.
Single-turn, English, xLAM-style schemas. No multi-turn, no tool results fed back.
Ground truth is imperfect, and now measured. An LLM-judge audit flagged exact-match failures whose reference cannot be derived from the request (a city, an ID, a ticker appearing nowhere in the prompt). 83 flagged cases were then reviewed by hand, blind to the judge's verdict: its precision is 80.0% on self-consistent flags [67.0, 88.8] but only 27.3% on self-contradictory ones [15.1, 44.2]. Correcting the raw 20.5% by those rates gives 15.0% of exact-match failures [12.7, 16.6] — about one in seven — or 4.2% of all held-out examples. Only flagged cases were reviewed, so that is a floor, not a ceiling: unusable references the judge missed would push it higher. True accuracy is therefore somewhat higher than exact match suggests.

License and attribution

Weights follow the base model's Apache-2.0 licence. Training data is Salesforce/xlam-function-calling-60k, CC-BY-4.0 — attribute Salesforce AI Research if you use anything derived from it.

smollm2-1.7b-instruct-xlam-lora

README

Quick start

What it is good at

Training

How it was evaluated

Limitations

Links

License and attribution

Explore FriendliAI today

README

Quick start

What it is good at

Training

How it was evaluated

Limitations

Links

License and attribution