PursuitOfDataScience

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ToolCall-LoRA

README

License: openmdw-1.1

Why this finetune exists

Nemotron-3.5-Lightning is built for long-running autonomous agents, and it is already willing to call tools — probed on held-out agentic trajectories it emits a well-formed call 97.7% of the time when one is needed. The problem is which call. It picks the correct function only 77.3% of the time, gets the full argument set exactly right 54.5% of the time, and fires a tool at an irrelevant request 18.5% of the time.

For an agent loop those three numbers are the ones that matter: a confidently malformed call costs a wasted turn, and a call made when none was warranted costs a wrong action. This adapter targets exactly those, trained on real multi-turn tool trajectories.

Results

Held-out Toucan-1.5M trajectories, n=200, greedy decoding, both arms measured at the same 1536-token generation budget. The validation split is deduplicated against training by normalised opening prompt.

Table with columns: metric, base, + adapter, change
metric	base	+ adapter	change
tool-name accuracy	77.3%	92.4%	↑ +15.2pp better
argument exact match	54.5%	72.7%	↑ +18.2pp better
schema valid (name exists, required args present)	81.1%	94.7%	↑ +13.6pp better
abstained when no tool applies	81.5%	92.6%	↑ +11.1pp better
named the right tool anywhere in its output	78.0%	92.4%	↑ +14.4pp better
emitted a well-formed call when one was needed	97.7%	94.7%	↓ -3.0pp worse
hit the generation ceiling mid-answer	2.3%	0.0%	↓ -2.3pp better

The one regression is honest and small: the adapter emits a call slightly less often (−3.0pp). It became more selective, and the selectivity is what buys the +18.2pp on argument exact-match and +11.1pp on abstention. Every call it does make is far likelier to be the right one, correctly parameterised.

Does it break anything?

A tool-calling finetune damages a general model in one characteristic way: it starts emitting call syntax when no tools were offered at all. That is invisible to every metric above, so it is measured directly — GSM8K prompts with no tools in the context, counting any tool-call markup in the output.

Table with columns: metric, base, + adapter, change
metric	base	+ adapter	change
GSM8K accuracy (no tools in context)	75.3%	78.7%	↑ +3.3pp better (n.s.)
tool-syntax intrusion	0.0%	0.0%	— +0.0pp

Paired significance on the GSM8K delta — McNemar: 9 fixed by the adapter, 4 broken by it, exact two-sided p = 0.2668 -- NOT significant at 0.05. The reasoning change is not significant; it is reported so the absence of damage is visible, not as an improvement.

Usage

This adapter targets the bf16 weights, not the NVFP4 checkpoint as shipped. quant_method: "modelopt" has no loader in transformers 5.6 or 5.15, so the published checkpoint cannot be loaded into a trainable — or adaptable — model at all. dequantize_nemotron_nvfp4.py in this repository rebuilds it:

bash

# ~10 min on CPU, ~20 GiB RAM, produces ~61 GiBpython dequantize_nemotron_nvfp4.py     --src /path/to/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4     --dst /path/to/nemotron-3.5-lightning-bf16

Compare the dequant_manifest.json it writes against the one in this repository to confirm you rebuilt the same tensors. The nibble unpacking is bit-exact against compressed-tensors' reference implementation.

python

from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained(    "/path/to/nemotron-3.5-lightning-bf16", dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "PursuitOfDataScience/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ToolCall-LoRA")tok = AutoTokenizer.from_pretrained("/path/to/nemotron-3.5-lightning-bf16") msgs = [{"role": "user", "content": "What's the weather in Chicago?"}]tools = [{"type": "function", "function": {    "name": "get_weather",    "description": "Current weather for a city.",    "parameters": {"type": "object",                   "properties": {"city": {"type": "string"}},                   "required": ["city"]}}}]ids = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True,                              return_tensors="pt").to(model.device)print(tok.decode(model.generate(ids, max_new_tokens=512)[0, ids.shape[1]:]))

The model answers in its native ChatML <tool_call><function=…> syntax. The adapter does not change the interface, only the reliability of using it.

Training

Table

Data	3,200 Toucan-1.5M trajectories (400 steps x 8 examples)
Hardware	1x H200 141GB, 149 minutes
LoRA	r=16, alpha=32, dropout=0.05, 13.9M trainable (0.044%)
Targets	mamba `in_proj`/`out_proj`, the 6 attention blocks, and the shared expert
LR	1e-4 cosine to 1e-5, warmup 20, grad-clip 1.0
Sequence	max 4096 tokens, micro-batches capped by padded token count
Loss	assistant spans only

The 23x128 routed experts are deliberately not adapted — they hold 29 of the 30B parameters, and adapting them would defeat the point of a light adapter on a 3B-active model. Validation loss fell 0.5341 → 0.4965 over the run.

Data preparation

Three defects in the raw corpus are filtered, because each teaches the opposite of the goal:

7.4% of records leak the corpus generator's own <tool_call>{...}</tool_call> JSON into assistant prose — a third syntax this model's template never uses. Lifted into structured calls.
4.2% of gold calls name a tool that was never offered; 114 are literally named unknown. Dropped — supervising those teaches exactly the hallucination this adapter is meant to remove.
The reasoning traces interleave the agent's planning with blocks where the data generator invents tool results. Only the agent's own passes are kept; an agent must never be trained to hallucinate observations.

Validation is deduplicated against training by normalised opening prompt. Toucan's subsets overlap by construction, so distinct ids are not distinct problems, and a naive split leaks 4.6% of validation.

Provenance and licensing

Base model NVIDIA-Nemotron-3.5-Lightning-30B-A3B, OpenMDW-1.1. That licence requires any redistribution to retain a copy of it and all origin notices.
Training data Toucan-1.5M, Apache-2.0.
Toucan is synthetic: every trajectory used here was generated by MiniMax-M2.5. The behaviour this adapter installs is distilled from that model's outputs and inherits its conventions.

Limitations

Trained on 3,200 examples for 400 steps. Validation loss was still falling, so this is a bounded run, not a converged finetune.
The evaluation scores the first tool call of a trajectory. It does not measure multi-turn task completion.
Measured only on Toucan-derived data. Gains against a corpus generated by MiniMax-M2.5 are partly gains at matching that model's conventions; this adapter has not been evaluated on an independent tool-calling benchmark.
Single seed. Sampling error is roughly ±6.9 points per cell at n=200.
The MTP (multi-token-prediction) head survives dequantisation but is not exercised by this recipe; speculative decoding was not re-measured.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

PursuitOfDataScience

Model Tree

Base

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Adapter

this model

Input Modalities

Text

Output Modalities