How to use
1. Download and install
pip install -U huggingface_hub # gives the `hf` commandhf auth login # only needed while the repo is privatehf download CobuildX-AI/tool-select-judge-2b --revision v0.1 --local-dir tool-select-judge-2bcd tool-select-judge-2bpip install -r requirements.txt
2. Use it in your code
ToolJudge (in judge.py, part of this repo) loads the model once and scores tool calls:
from judge import ToolJudge judge = ToolJudge() # loads Qwen/Qwen3.5-2B + this adapterjudge.score( tools=[{"name": "search_hotels", "description": "Search hotels in a city"}, {"name": "book_hotel", "description": "Book a specific hotel room"}], conversation=[{"role": "user", "content": "Find me a hotel in Paris for next week"}], picked=["book_hotel"], # tool name(s) the agent called; [] = no tool called)# {'answer': 0, 'confidence': 0.97, 'probability_right': 0.03, 'latency_ms': 210}
tools: the tools the agent had at this step. conversation: every message before the call, with
roles user, assistant, assistant_tool_call (an earlier call) and tool (a tool result).
- Score a whole trace by calling
score once per tool-call step (or score_batch).
judge.py builds every prompt with prompt.py, the exact layout the model was trained on.
3. Optional: check your setup
python example.py scores 4 sample tool calls and prints the answers, so you can confirm the install works.
Results
Table with columns: tool-select-judge-2b v0.1, Jev 1.13 | tool-select-judge-2b v0.1 | Jev 1.13 |
|---|
| Held-out test accuracy (3,904 calls) | 95.6% | 89.8% |
| Multi-turn accuracy (1,028 calls) | 93.0% | 80.0% |
| TRBench subset accuracy (1,910 calls) | 83.7% | 84.1% |
| Calls to tools that do not exist, caught (395 calls) | 98.0% | 47.3% |
All tests are 50% right / 50% wrong and were never used in training. Results were measured with
prompts of up to 2,048 tokens; much longer prompts are slower and have not been measured.
TRBench is a subset, not the full benchmark. From the 11,932 pairs in
RioLee/TRBench-BFCL we removed 7,232 where the
wrong answer used the same tool with different arguments (not a tool-choice mistake) or had formatting
issues, and 3,745 duplicates. That leaves 955 unique wrong tool choices plus 955 correct reference
answers. These scores are not comparable to full-TRBench scores.
Licence
Apache-2.0.