Chloemp

smollm2-1.7b-xlam-lora

README

License: apache-2.0

Quick start

python

import json, torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer ADAPTER  = "Chloemp/smollm2-1.7b-xlam-lora"REVISION = "8868bcd45fb4942552222602f539148799d6e593" tok = AutoTokenizer.from_pretrained(ADAPTER, revision=REVISION)base = AutoModelForCausalLM.from_pretrained(    "HuggingFaceTB/SmolLM2-1.7B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, ADAPTER, revision=REVISION).merge_and_unload().eval() tools = json.dumps([    {"name": "get_weather",     "description": "Get the current weather for a location.",     "parameters": {"city": {"description": "City name", "type": "str"},                    "unit": {"description": "celsius or fahrenheit", "type": "str", "default": "celsius"}}},]) messages = [    {"role": "system", "content": f"You are a function-calling assistant. Available tools:\n{tools}"},    {"role": "user", "content": "What's the weather in Lyon in fahrenheit?"},] inputs = tok.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) with torch.no_grad():    out = model.generate(**inputs, max_new_tokens=512, do_sample=False,                         pad_token_id=tok.pad_token_id or tok.eos_token_id) print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))# [{"name": "get_weather", "arguments": {"city": "Lyon", "unit": "fahrenheit"}}]

The tokenizer shipped here carries the chat template the model was trained with (ChatML, eos <|im_end|>), borrowed from SmolLM2-1.7B-Instruct — the base model has no template of its own. Greedy decoding is what was evaluated; for batched generation set tok.padding_side = "left".

Scores by level

Table with columns: Level, Score
Level	Score
1. Valid JSON	99.21%
2. Schema `[{name, arguments}]`	99.21%
3. No hallucinated tool	98.87%
4. Correct call count	97.04%
5. Correct tool names	95.62%
6. Correct argument values (= exact match)	75.11%

An LLM-judge audit of these predictions puts accuracy at 84.55% once references that cannot be derived from the request are excluded — computed from the judge's raw flags; the human-review correction below shrinks the unusable share and moves this figure down by roughly one point. The judge agreed with 99.89% of the exact matches it was asked to re-grade, which is what makes that figure usable.

Training

LoRA r=16, α=32, dropout 0.05 on all linear layers (18.1M trainable of 1.73B, 1.05%), lr 2e-4, 3% warmup, 1 epoch (3,563 steps), effective batch 16 (8 × 2 accumulation), bf16, gradient checkpointing, max length 2048. One A100, 1h45, 9.48 GB peak.

How it was evaluated

2,395 held-out examples from the same dataset, after removing 105 contaminated rows (93 queries appeared on both sides of the split). Greedy decoding, left padding, max_new_tokens=512 (the longest reference in the eval set is 410 tokens), batch size 32, bf16 on one A100. Scores attach to the pinned revision above, not to main.

Scoring is six nested levels, each strictly harder than the last, so a failure is located rather than just counted. Level 6 is exact match after normalising types (5 == "5"), key order and call order.

Table with columns: Model, Exact match (n=2,395), 95% CI
Model	Exact match (n=2,395)	95% CI
smollm2-1.7b-instruct-xlam-lora	83.47%	[81.92, 84.90]
smollm2-1.7b-xlam-lora	75.11%	[73.34, 76.81]
smollm2-1.7b-xlam-qlora	72.61%	[70.79, 74.36]
SmolLM2-1.7B-Instruct, few-shot prompting, no fine-tune	39.58%

For reference, Qwen3-235B-A22B-Instruct prompted with the same format block scores 78.33% on a fixed 300-example subset — statistically indistinguishable from the Instruct-based LoRA on those same 300 (80.00%, paired McNemar p = 0.50).

Limitations

Known tools only. A random split of xLAM means nearly every tool in the eval set also appears in training. These numbers measure reliable calling of a known toolset on new phrasing. On 20 hand-written queries against four tools absent from xLAM, the fine-tunes score 60–70% against 80% for the un-tuned Instruct model: fine-tuning on xLAM costs some generalisation to unseen tools.
Irrelevance detection is unmeasured. Nothing here tests whether the model abstains when no available tool answers the request. Expect it to call something anyway.
Single-turn, English, xLAM-style schemas. No multi-turn, no tool results fed back.
Ground truth is imperfect, and now measured. An LLM-judge audit flagged exact-match failures whose reference cannot be derived from the request (a city, an ID, a ticker appearing nowhere in the prompt). 83 flagged cases were then reviewed by hand, blind to the judge's verdict: its precision is 80.0% on self-consistent flags [67.0, 88.8] but only 27.3% on self-contradictory ones [15.1, 44.2]. Correcting the raw 20.5% by those rates gives 15.0% of exact-match failures [12.7, 16.6] — about one in seven — or 4.2% of all held-out examples. Only flagged cases were reviewed, so that is a floor, not a ceiling: unusable references the judge missed would push it higher. True accuracy is therefore somewhat higher than exact match suggests.

License and attribution

Weights follow the base model's Apache-2.0 licence. Training data is Salesforce/xlam-function-calling-60k, CC-BY-4.0 — attribute Salesforce AI Research if you use anything derived from it.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Chloemp

Model Tree

Base

HuggingFaceTB/SmolLM2-1.7B

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quick start

python

import json, torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer ADAPTER  = "Chloemp/smollm2-1.7b-xlam-lora"REVISION = "8868bcd45fb4942552222602f539148799d6e593" tok = AutoTokenizer.from_pretrained(ADAPTER, revision=REVISION)base = AutoModelForCausalLM.from_pretrained(    "HuggingFaceTB/SmolLM2-1.7B", dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, ADAPTER, revision=REVISION).merge_and_unload().eval() tools = json.dumps([    {"name": "get_weather",     "description": "Get the current weather for a location.",     "parameters": {"city": {"description": "City name", "type": "str"},                    "unit": {"description": "celsius or fahrenheit", "type": "str", "default": "celsius"}}},]) messages = [    {"role": "system", "content": f"You are a function-calling assistant. Available tools:\n{tools}"},    {"role": "user", "content": "What's the weather in Lyon in fahrenheit?"},] inputs = tok.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) with torch.no_grad():    out = model.generate(**inputs, max_new_tokens=512, do_sample=False,                         pad_token_id=tok.pad_token_id or tok.eos_token_id) print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))# [{"name": "get_weather", "arguments": {"city": "Lyon", "unit": "fahrenheit"}}]

Scores by level

Table with columns: Level, Score
Level	Score
1. Valid JSON	99.21%
2. Schema `[{name, arguments}]`	99.21%
3. No hallucinated tool	98.87%
4. Correct call count	97.04%
5. Correct tool names	95.62%
6. Correct argument values (= exact match)	75.11%

Training

How it was evaluated

Table with columns: Model, Exact match (n=2,395), 95% CI
Model	Exact match (n=2,395)	95% CI
smollm2-1.7b-instruct-xlam-lora	83.47%	[81.92, 84.90]
smollm2-1.7b-xlam-lora	75.11%	[73.34, 76.81]
smollm2-1.7b-xlam-qlora	72.61%	[70.79, 74.36]
SmolLM2-1.7B-Instruct, few-shot prompting, no fine-tune	39.58%

Limitations

Known tools only. A random split of xLAM means nearly every tool in the eval set also appears in training. These numbers measure reliable calling of a known toolset on new phrasing. On 20 hand-written queries against four tools absent from xLAM, the fine-tunes score 60–70% against 80% for the un-tuned Instruct model: fine-tuning on xLAM costs some generalisation to unseen tools.
Irrelevance detection is unmeasured. Nothing here tests whether the model abstains when no available tool answers the request. Expect it to call something anyway.
Single-turn, English, xLAM-style schemas. No multi-turn, no tool results fed back.
Ground truth is imperfect, and now measured. An LLM-judge audit flagged exact-match failures whose reference cannot be derived from the request (a city, an ID, a ticker appearing nowhere in the prompt). 83 flagged cases were then reviewed by hand, blind to the judge's verdict: its precision is 80.0% on self-consistent flags [67.0, 88.8] but only 27.3% on self-contradictory ones [15.1, 44.2]. Correcting the raw 20.5% by those rates gives 15.0% of exact-match failures [12.7, 16.6] — about one in seven — or 4.2% of all held-out examples. Only flagged cases were reviewed, so that is a floor, not a ceiling: unusable references the judge missed would push it higher. True accuracy is therefore somewhat higher than exact match suggests.

License and attribution

Weights follow the base model's Apache-2.0 licence. Training data is Salesforce/xlam-function-calling-60k, CC-BY-4.0 — attribute Salesforce AI Research if you use anything derived from it.

smollm2-1.7b-xlam-lora

README

Quick start

Scores by level

Training

How it was evaluated

Limitations

Links

License and attribution

Explore FriendliAI today

README

Quick start

Scores by level

Training

How it was evaluated

Limitations

Links

License and attribution