Model summary
Soro-Kev-0.8B is a specialized decision model for AI agent tool routing, schema normalization, and execution boundary control. It reads one document (the state: user prompt, conversational history, and available tool schemas) and a set of typed questions about it, and returns a calibrated probability distribution over the options supplied with each question, in a single forward pass and without generating text.
It is built directly on top of Jared Palmer's Kev pointer architecture, addressing an explicit limitation documented in the original Kev-0.8B model card ("Tool-call routing: its When2Call accuracy is below chance (0.133). Do not use it for tool-call routing"). Soro-Kev-0.8B is trained specifically on multi-protocol tool catalogs (Model Context Protocol, REST/OpenAPI, Agent Skills, and Python signatures), raising tool routing accuracy to 85.67% (1,028/1,200) on held-out MCP & agent benchmarks, and 83.00% (83/100) on BFCL v3 tool selection.
It consists of a LoRA adapter (r=16,α=32) and a pointer head on Qwen/Qwen3.5-0.8B-Base. It runs in ~124.2 ms on an NVIDIA GeForce RTX 4050 Laptop GPU (1,563 MB peak VRAM, PyTorch 2.6.0). It is intended for edge deployment, local MCP desktop sidecars, and high-frequency routing pipelines where generative LLM token generation latency is prohibitive.
Model details
Table with columns: Field, Detail| Field | Detail |
|---|
| Developer | Keyush Nisar (github.com/nisaral/Soro-Boro) |
| Model type | Decision model: causal language-model backbone run prefill-only, with a pointer head over candidate options |
| Backbone | Qwen/Qwen3.5-0.8B-Base: 24 layers (18 Gated DeltaNet linear attention, 6 full attention); frozen |
| Adapter | LoRA, rank 16, α 32, on attention, MLP, and DeltaNet projections (11.3M parameters) |
| Head | Pointer head: hidden-state projections score each option's closing token against the question's final token (h) |
Soro-Kev-0.8B evaluates a state and any number of named questions across three typed categories:
Table with columns: Type, Options, Output, Intended Purpose| Type | Options | Output | Intended Purpose |
|---|
choice (tool_route) | 1–255 candidate tool schemas + __abstain__ | Probability per tool, top tool, and conformal prediction set S | Selects the single correct tool to execute, or abstains |
choice (arg_mapping) | Schema property keys + __drop__ | Probability distribution over candidate property names | Maps unrecognized LLM argument keys to target schema parameters |
Each question is evaluated as its own forward path branching from the shared state hidden representation; questions cannot cross-contaminate one another, and the state prefix is computed once and cached.
Intended Uses
- AI Agent Tool Routing: Selecting tools from catalogs of 10 to 100+ schemas in sub-100ms without consuming context tokens on JSON generation.
- Model Context Protocol (MCP) Sidecars: Routing queries to local tools exposed by MCP servers over stdio or SSE.
- Conformal Action Boundary: Generating calibrated prediction sets (S) where ∣S∣=1 triggers
EXECUTE, ∣S∣∈[2,8] triggers targeted ASK clarification, and __abstain__ safely halts execution.
- : Running alongside small edge LLMs on consumer laptops, Apple Silicon, or 4 GB GPUs.
Out-of-Scope Uses
- Text generation, conversation, or creative writing: The model runs in prefill-only mode and has no generative token decode loop.
- Arbitrary general classification without tools: Out of domain on tasks unlike its training families (e.g. sentiment classification or narrative analysis), where general models like Kev-0.8B or Kev-4B should be used.
- Fully autonomous execution of high-risk actions: Financial transfers, database drops, or administrative actions should always require operator confirmation (
ASK / human-in-the-loop).
- Languages other than English: Non-English tool descriptions and prompts are not validated.
How to Use
1. High-Level Python SDK (Soro Harness)
from soro.harness.pipeline import SoroHarness # Load checkpoint from Hugging Faceharness = SoroHarness.from_pretrained("nisaralll/soro-kev-0.8b") @harness.tool(description="Check account balance for customer", side_effect="read_only")def get_balance(customer_id: str) -> dict: return {"customer_id": customer_id, "balance_usd": 1420.50} @harness.tool(description="Issue partial or full refund for settled order", side_effect="mutation")def refund_order(order_id: str, amount: float) -> dict: return {"order_id": order_id, "refunded": amount} # Sub-100ms Conformal Routingdecision = harness.route("Refund customer $45.00 for order ORD-9921")print(decision.action) # "EXECUTE"print(decision.selected_tool) # "refund_order"print(decision.candidates) # ["refund_order"] (Prediction set |S| = 1)
2. Low-Level In-Process Scorer
from soro.routing.kev_local import LocalKevScorer scorer = LocalKevScorer(run="nisaralll/soro-kev-0.8b", backend="neural")scorer.load() state = "User Request: Check available disk storage in /var/log"questions = { "tool_route": { "type": "choice", "instructions": "Which tool should execute for this request?", "criteria": { "get_disk_usage": "Return total, used, and available disk storage for path.", "list_processes": "List active system processes and PIDs.", "__abstain__": "None of the listed tools fit." } }} result = scorer.evaluate_turn(state, questions)print(result["answers"]["tool_route"]) # "get_disk_usage"print(result["probabilities"]["tool_route"])
Training Data & Target Reconciliation
Soro-Kev-0.8B was trained on the nisaralll/soro-universal-tool-harness dataset (train.jsonl), consisting of exactly 23,192 multi-task state rows. Each row contains exactly 5 question instances evaluated concurrently in a single forward pass, yielding 115,960 machine-counted supervision targets:
Table with columns: Question Head, Question Type, Rows Present, Ground Truth Breakdown, Purpose & Learning Task| Question Head | Question Type | Rows Present | Ground Truth Breakdown | Purpose & Learning Task |
|---|
tool_route | choice | 23,192 | 23,192 valid closed-candidate selections + __abstain__ | Primary tool routing decision across multi-protocol candidates |
tool_route_perm | choice | 23,192 |
Evaluation suites (val_external_human.jsonl with 1,200 held-out queries, and val_zeroshot.jsonl with 2,220 queries across 55 unseen packages) are strictly disjoint from training:
- Package Overlap: 0 packages (0 / 5 external, 0 / 55 zero-shot).
- Exact Query Overlap: 0 queries across all splits.
- Near-Duplicate Queries: 0 pairs at 3-gram Jaccard ≥0.75, 1 pair at dense embedding cosine ≥0.88.
- Tool Name Overlap: 4 generic tools (
check_service_status, list_processes, reverse_geocode, transfer_to_human_agents) account for 162 rows (13.50%) in external holdout; 1,038 rows (86.50%) feature strictly disjoint tools whose names never appear in train.jsonl. Both subsets are reported separately below.
- Audit Verification: Committed in and .
Evaluation Results
All models and baselines were evaluated on an NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM, PCIe 4.0 x4, PyTorch 2.6.0, CUDA 12.4). Conformal prediction metrics report empirical coverage and prediction set size (∣S∣) over 100 random splits (Ncal=500, nominal coverage 1−α=0.90).
1. Decision Model & Retrieval Baseline Comparison (Full 1,200 External Rows across 115 Unique Base Queries)
Note on Query Stratification: The 1,200 evaluations comprise 115 unique base queries tested across ~10 format/distractor views
(1,095 reference MCP rows + 105 τ2 persona-dialogue rows). Soro-Kev-4B achieves 96.89% (1,061/1,095) on reference MCP tools
and 24.76% (26/105) on τ2, where 64.56% of misses are calibrated abstentions or preliminary status checks.
Table with columns: Method / Architecture, Overall (1,200), Non-Abstain (1,065), Abstain Refusal (135)*, Strictly Disjoint Tools (1,038), Shared Tool Names (162), Conformal Coverage (1−α=0.90), Mean Set Size ∥S∥| Method / Architecture | Overall (1,200) | Non-Abstain (1,065) | Abstain Refusal (135)* | Strictly Disjoint Tools (1,038) | Shared Tool Names (162) | Conformal Coverage (1−α=0.90) | Mean Set Size ∥S∥ |
|---|
| BM25 (Calibrated τ) |
*Note on Abstention: The 135 abstain rows represent 15 unique base queries evaluated across 9 format/distractor views each (15×9=135). Denominators strictly reconcile: 1,065+135=1,200. Numerators strictly reconcile: e.g. 0.8B gets 926+102=1,028; 4B gets 964. Dedicated adversarial refusal is evaluated on the 33-row BFCL v3 Irrelevance benchmark (Section 3).
Pointer Network Guarantee: Restricts candidate selection strictly to the closed candidate set + __abstain__ (0% invalid or invented tool-name generation). The empirical misclassification rate (1−Top-1 Accuracy) is 14.33% (172/1200) on external holdouts.
Note on Queries & Schemas: Queries share structural template distributions with training while all 55 tool schemas
and packages are strictly disjoint (0 package overlap with the 319 training packages).
Table with columns: System, Overall Zero-Shot (2,220 rows), Clean Menus (2,168 rows), Collision Distractors (52 rows)| System | Overall Zero-Shot (2,220 rows) | Clean Menus (2,168 rows) | Collision Distractors (52 rows) |
|---|
| BM25 (Calibrated Abstention) | 48.60% (1079/2220) | 49.40% (1071/2168) | 15.38% (8/52) |
| Dense Cosine (Calibrated Abstention) | 46.04% (1022/2220) | 46.63% (1011/2168) | 21.15% (11/52) |
| Cross-Encoder (Calibrated Abstention) | 50.54% (1122/2220) | 51.48% (1116/2168) | 11.54% (6/52) |
| Untrained Option Logits (Zero-shot) | 47.48% (1054/2220) |
3. Berkeley Function-Calling Leaderboard (BFCL v3 Diagnostic, 100 rows)
Table with columns: System, Multiple Tools (34), Live Multiple (33), Irrelevance / Abstention (33), Overall Diagnostic (100), Latency (ms)| System | Multiple Tools (34) | Live Multiple (33) | Irrelevance / Abstention (33) | Overall Diagnostic (100) | Latency (ms) |
|---|
| BM25 (Calibrated Abstention) | 94.12% (32/34) | 57.58% (19/33) | 36.36% (12/33) | 63.00% (63/100) | 0.1 ms |
| Dense Cosine (Calibrated Abstention) | 97.06% (33/34) | 60.61% (20/33) | 33.33% (11/33) | 64.00% (64/100) | 8.5 ms |
|
*Hardware & Latency Note (RTX 4050 6GB Laptop GPU, Windows WDDM):
- Architecture Difference (Prefill vs. Autoregressive Decode): Soro-Kev outputs routing + key-mapping decisions scoring candidate tools and an explicit
__abstain__ anchor via a dual linear pointer head in 1 single prefill forward pass (124.2 ms for 0.8B, 648.6 ms for 4B) with 0 generated decode tokens. In contrast, generative models (xLAM-2-1b-fc-r, Hammer2.1-1.5b) generate full JSON tool calls with argument values (~27–32 decode tokens), requiring 1 prefill pass plus 24-32 sequential autoregressive decode steps (one forward pass per token).
- In-VRAM vs. Sustained / Paged Latency: In clean isolated VRAM bursts,
Arch-Router-1.5B achieves a median latency of 1,212.9 ms (~24 tokens generated). xLAM-2-1b-fc-r (mean prompt 406 tokens, mean gen 27 tokens) achieves 7,106.9 ms median in clean VRAM (initial queries: ~1,458-2,549 ms; laptop GPU thermal throttling at 85C settles to ~7.1s). Hammer2.1-1.5b (generating full markdown JSON arrays of ~32 tokens) settles to 10,564.5 ms median. When multiple generative models were evaluated sequentially without intermediate process garbage collection, Windows WDDM silently paged KV caches into Shared GPU Memory across PCIe, increasing latencies to ~10.4s-11.5s.
- Dense Cosine & Neural Baselines: All 100 BFCL rows for Dense Cosine were evaluated directly using
SentenceTransformer / forward passes on CUDA timed with , yielding a genuine hardware median latency of (6.3-60.4 ms range, 64/100 correct).
Limitations and Trade-Offs
- No Generative Argument Synthesis: Soro-Kev normalizes parameter keys and checks argument completeness; it does not author free-text string values. In production pipelines, arguments are drafted by a fast parameter extractor, normalized via
arg_mapping, and guarded by missing_required before execution.
- Single-Turn Model: Soro-Kev evaluates single routing turns. Multi-tool sequential chaining ("call tool A, then tool B") is an architectural runtime design and has not yet been benchmarked on multi-step traces.
- Edge Model Capacity: With 0.8B parameters, out-of-domain transfer on complex multi-hop semantic reasoning is lower than Soro-Kev-4B (85.67% vs 90.58% on external holdouts).
- Abstention Sensitivity: On BFCL irrelevance slices where queries closely resemble registered tools, Soro-Kev-0.8B abstains 51.52% (17/33) of the time, compared to 75.76% (25/33) for Soro-Kev-4B. For strict zero-tolerance out-of-domain rejection, use Soro-Kev-4B.
Provenance and Reproducibility
Citations
If you use Soro-Kev or Universal Tool Harness in your research or production systems, please cite the following publications:
@software{nisar2026sorokev, author = {Nisar, Keyush}, title = {Soro-Kev: System-One Conformal Tool Router and Execution Boundary}, year = {2026}, publisher = {GitHub}, howpublished = {\url{https://github.com/nisaral/Soro-Boro}}, note = {Release v0.2.0}} @misc{palmer2026kev08b, author = {Palmer, Jared}, title = {Kev-0.8B: a calibrated decision model on Qwen3.5-0.8B}, year = {2026}, howpublished = {\url{https://huggingface.co/jaredpalmer/kev-0.8b}}, note = {Kev 1.0}} @inproceedings{shi2025toolret, author = {Shi, Zhengliang and Gao, Shen and Chen, Xiuying and Wang, Lingzhi and Pei, Jiahuan and Guan, Xinyu and Ren, Zhaochun}, title = {Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2025}, pages = {11624--11639}, year = {2025}, url = {https://aclanthology.org/2025.findings-acl.647/}} @inproceedings{mishra2026mcpsemantic, author = {Mishra, Amritansh and Panditrao, Yash and Parab, Shlok and Sharma, Achintya and Mishra, Payal and Saha, Ishani}, title = {Scalable Semantic Routing and QoS-Aware Provider Selection for MCP Agents}, booktitle = {Procedia Computer Science, International Conference on Machine Learning and Data Engineering (ICMLDE 2026)}, year = {2026}, howpublished = {\url{https://github.com/amrittmishra/mcp-semantic-tool-routing-benchmark}}} @misc{mo2025livemcpbench, author = {Mo, Guozhao and others}, title = {LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?}, howpublished = {Preprint}, year = {2025}} @inproceedings{ross2025when2call, author = {Ross, Hayley and Mahabaleshwarkar, Ameya Sunil and Suhara, Yoshi}, title = {When2Call: When (not) to Call Tools}, booktitle = {Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)}, year = {2025}}