Model summary
Soro-Kev-4B is a flagship decision model for AI agent tool routing, multi-protocol schema normalization, and execution boundary control. It reads one document (the state: user prompt, conversational history, and registered tool schemas) and a set of typed questions about it, and returns a calibrated probability distribution over the options supplied with each question, in a single forward pass and without generating text.
It is built directly on top of Jared Palmer's Kev pointer architecture, addressing an explicit limitation documented in the original Kev model cards ("Tool-call routing: its When2Call accuracy is below chance (0.133). Do not use it for tool-call routing"). Soro-Kev-4B is fine-tuned specifically on the multi-protocol Universal Tool Harness, raising tool routing accuracy to 90.58% (1,087/1,200) on held-out MCP & agent benchmarks, and 91.00% (91/100) on BFCL v3 tool selection.
It consists of a LoRA adapter (r=16,α=32) and a pointer head on Qwen/Qwen3-4B. In 4-bit NF4 quantization, it operates in ~4,048 MB VRAM with a median prefill latency of ~648.6 ms on an NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM, PCIe Gen 4, PyTorch 2.6.0, CUDA 12.4). It is intended for production AI agents, complex Model Context Protocol (MCP) tool catalogs (50+ tools), and backend API routing engines where generative LLM JSON hallucinations and latency overhead cannot be tolerated.
Model details
Table with columns: Field, Detail| Field | Detail |
|---|
| Developer | Keyush Nisar (github.com/nisaral/Soro-Boro) |
| Model type | Decision model: causal language-model backbone run prefill-only, with a pointer head over candidate options |
| Backbone | Qwen/Qwen3-4B: 36 layers, multi-head attention; frozen |
| Adapter | LoRA, rank 16, α 32, on attention projections and MLP layers |
| Head | Pointer head (d=2560): hidden-state projections score each option's closing token against the question's final token () |
Soro-Kev-4B evaluates a state and any number of named questions across three typed categories:
Table with columns: Type, Options, Output, Intended Purpose| Type | Options | Output | Intended Purpose |
|---|
choice (tool_route) | 1–255 candidate tool schemas + __abstain__ | Probability per tool, top tool, and conformal prediction set S | Selects the single correct tool to execute, or abstains |
choice (arg_mapping) | Schema property keys + __drop__ | Probability distribution over candidate property names | Maps unrecognized LLM argument keys to target schema parameters |
Each question is evaluated as its own forward path branching from the shared state hidden representation; questions cannot cross-contaminate one another, and the state prefix is computed once and cached.
Intended Uses
- Production AI Agent Tool Routing: Selecting tools from large catalogs (50+ schemas) in single-pass inference without prompt token generation bloat.
- Model Context Protocol (MCP) Server Gateways: High-accuracy routing across heterogeneous tool protocols (MCP JSON, REST APIs, GraphQL, Python callables).
- Conformal Action Boundary: Generating calibrated prediction sets (S) where ∣S∣=1 triggers
EXECUTE, ∣S∣∈[2,8] triggers targeted ASK clarification, and __abstain__ safely halts execution.
Out-of-Scope Uses
- Text generation, conversation, or creative writing: The model runs in prefill-only mode and has no generative token decode loop.
- Open-ended text synthesis: The model selects and normalizes schemas; it does not author free-form documents.
- Fully autonomous execution of high-risk actions: Critical financial or destructive operations should always require operator confirmation (
ASK / human-in-the-loop).
- Languages other than English: Non-English tool descriptions and prompts are not validated.
How to Use
1. High-Level Python SDK (Soro Harness)
from soro.harness.pipeline import SoroHarness # Loads directly from Hugging Face in NF4 quantization (4 GB VRAM)harness = SoroHarness.from_pretrained("nisaralll/soro-kev-4b") @harness.tool(description="Look up customer order details", side_effect="read_only")def get_order_status(order_id: str) -> dict: return {"order_id": order_id, "status": "shipped", "tracking": "TRK-9810"} @harness.tool(description="Cancel a warehouse shipment", side_effect="mutation")def cancel_shipment(order_id: str) -> dict: return {"order_id": order_id, "cancelled": True} # Fast Conformal Routingdecision = harness.route("Where is my package for order ORD-1092?")print("Decision Action:", decision.action) # "EXECUTE"print("Selected Tool:", decision.selected_tool) # "get_order_status"print("Confidence:", decision.probabilities.get("get_order_status")) # Execute with Automatic Argument Coercionresult = harness.execute("get_order_status", {"order_id": "ORD-1092"})print("Execution Result:", result.data)
2. Low-Level In-Process Scorer
from soro.routing.kev_local import LocalKevScorer scorer = LocalKevScorer(run="nisaralll/soro-kev-4b", backend="neural")scorer.load() state = "User Request: Check available disk storage in /var/log"questions = { "tool_route": { "type": "choice", "instructions": "Which tool should execute for this request?", "criteria": { "get_disk_usage": "Return total, used, and available disk storage for path.", "list_processes": "List active system processes and PIDs.", "__abstain__": "None of the listed tools fit." } }} result = scorer.evaluate_turn(state, questions)print(result["answers"]["tool_route"]) # "get_disk_usage"
Training Data & Target Reconciliation
Soro-Kev-4B was trained on the nisaralll/soro-universal-tool-harness dataset (train.jsonl), consisting of exactly 23,192 multi-task state rows. Each row contains exactly 5 question instances evaluated concurrently in a single forward pass, yielding 115,960 machine-counted supervision targets:
Table with columns: Question Head, Question Type, Rows Present, Ground Truth Breakdown, Purpose & Learning Task| Question Head | Question Type | Rows Present | Ground Truth Breakdown | Purpose & Learning Task |
|---|
tool_route | choice | 23,192 | 23,192 valid closed-candidate selections + __abstain__ | Primary tool routing decision across multi-protocol candidates |
tool_route_perm | choice | 23,192 |
Evaluation suites (val_external_human.jsonl with 1,200 held-out queries, and val_zeroshot.jsonl with 2,220 queries across 55 unseen packages) are strictly disjoint from training:
- Package Overlap: 0 packages (0 / 5 external, 0 / 55 zero-shot).
- Exact Query Overlap: 0 queries across all splits.
- Near-Duplicate Queries: 0 pairs at 3-gram Jaccard ≥0.75, 1 pair at dense embedding cosine ≥0.88.
- Tool Name Overlap: 4 generic tools (
check_service_status, list_processes, reverse_geocode, transfer_to_human_agents) account for 162 rows (13.50%) in external holdout; 1,038 rows (86.50%) feature strictly disjoint tools whose names never appear in train.jsonl. Both subsets are reported separately below.
- Audit Verification: Committed in and .
Evaluation Results
All models and baselines were evaluated on an NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM, PCIe 4.0 x4, PyTorch 2.6.0, CUDA 12.4). Conformal prediction metrics report empirical coverage and prediction set size (∣S∣) over 100 random splits (Ncal=500, nominal coverage 1−α=0.90).
1. Decision Model & Retrieval Baseline Comparison (Full 1,200 External Rows across 115 Unique Base Queries)
Note on Query Stratification: The 1,200 evaluations comprise 115 unique base queries tested across ~10 format/distractor views
(1,095 reference MCP rows + 105 τ2 persona-dialogue rows). Soro-Kev-4B achieves 96.89% (1,061/1,095) on reference MCP tools
and 24.76% (26/105) on τ2, where 64.56% of misses are calibrated abstentions or preliminary status checks.
To investigate why classical retrievers fail on tool catalogs, all retrievers were evaluated with clean schema text parsing (unwrapping JSON/REST docstrings) and calibrated with an explicit abstention threshold τabstain tuned on calibration splits:
Table with columns: Method / Architecture, Overall (1,200), Non-Abstain (1,065), Abstain Refusal (135)*, Strictly Disjoint Tools (1,038), Shared Tool Names (162), Conformal Coverage (1−α=0.90), Mean Set Size ∥S∥| Method / Architecture | Overall (1,200) | Non-Abstain (1,065) | Abstain Refusal (135)* | Strictly Disjoint Tools (1,038) | Shared Tool Names (162) | Conformal Coverage (1−α=0.90) | Mean Set Size ∥S∥ |
|---|
| BM25 (Calibrated τ) |
*Note on Abstention: The 135 abstain rows represent 15 unique base queries evaluated across 9 format/distractor views each (15×9=135). Denominators strictly reconcile: 1,065+135=1,200. Numerators strictly reconcile: e.g. 0.8B gets 926+102=1,028; 4B gets 964. Dedicated adversarial refusal is evaluated on the 33-row BFCL v3 Irrelevance benchmark (Table 5).
Pointer Network Guarantee: The dual pointer head restricts selection strictly to candidate keys in context + __abstain__. It generates 0% invalid or invented tool names by construction. True misclassification rate (1−Top-1 Accuracy) is 9.42% (113/1200) on external holdouts.
Hand-review of 20 failure cases reveals three systematic root causes for classical retriever weakness:
- Directional Semantic Inversion: Queries like "Give me the street address" require
reverse_geocode (takes coordinates → outputs address). geocode_location (takes address → outputs coordinates) contains the keyword "address" in its input parameters, causing BM25 and Dense retrievers to pick the wrong direction.
- Lexical Synonym Gaps: User prompts like "What is the current RAM consumption?" or "Change the file type of this image" have 0 token overlap with tools named
get_memory_usage ("Returns memory...") or convert_image_format ("Converts image format..."). Keyword matching scores 0.0 and falls back to incorrect tools or premature abstention.
- Prompt Instruction Noise: Real user dialogue prompts (e.g. from Tau-2) include conversational instructions ("If the agent suggests actions that don't fix it, express mild frustration..."). Incidental words like "suggest", "read", or "plan" trigger false matches against unrelated tools like
read_environment_variable or change_plan.
3. Per-Package Accuracy Table (External Holdout Breakdown)
Table with columns: Package, Domain / Origin, Rows, BM25, Dense Cosine, Cross-Encoder, Untrained Logits, Soro-Kev-0.8B, Soro-Kev-4B| Package | Domain / Origin | Rows | BM25 | Dense Cosine | Cross-Encoder | Untrained Logits | Soro-Kev-0.8B | Soro-Kev-4B |
|---|
mcp_benchmark/system_mcp | OS & Process Management | 370 | 64.59% (239/370) | 55.68% (206/370) | 63.24% (234/370) | 66.22% (245/370) | 87.57% (324/370) | |
*Note on Tau-2 persona simulation: Across the 1,095 reference MCP tools (system, geo, media), Soro-Kev-4B achieves 96.89% (1,061/1,095). On the 105 Tau-2 multi-turn dialogue instructions, Soro-Kev correctly refuses to blindly fire backend mutation tools without customer identifiers: 64.56% (51/79) of Soro-Kev-4B's misses were explicit abstentions (21 rows) or preliminary read/status lookups (30 rows, e.g. check_service_status, get_device_details) rather than premature account mutations.
Note on Queries & Schemas: Queries share structural template distributions with training while all 55 tool schemas
and packages are strictly disjoint (0 package overlap with the 319 training packages).
Table with columns: System, Overall Zero-Shot (2,220 rows), Clean Menus (2,168 rows), Collision Distractors (52 rows)| System | Overall Zero-Shot (2,220 rows) | Clean Menus (2,168 rows) | Collision Distractors (52 rows) |
|---|
| BM25 (Calibrated Abstention) | 48.60% (1079/2220) | 49.40% (1071/2168) | 15.38% (8/52) |
| Dense Cosine (Calibrated Abstention) | 46.04% (1022/2220) | 46.63% (1011/2168) | 21.15% (11/52) |
| Cross-Encoder (Calibrated Abstention) | 50.54% (1122/2220) | 51.48% (1116/2168) | 11.54% (6/52) |
| Untrained Option Logits (Zero-shot) | 47.48% (1054/2220) |
5. Berkeley Function-Calling Leaderboard (BFCL v3 Diagnostic, 100 rows)
Table with columns: System, Multiple Tools (34), Live Multiple (33), Irrelevance / Abstention (33), Overall Diagnostic (100), Latency (ms)| System | Multiple Tools (34) | Live Multiple (33) | Irrelevance / Abstention (33) | Overall Diagnostic (100) | Latency (ms) |
|---|
| BM25 (Calibrated Abstention) | 94.12% (32/34) | 57.58% (19/33) | 36.36% (12/33) | 63.00% (63/100) | 0.1 ms |
| Dense Cosine (Calibrated Abstention) | 97.06% (33/34) | 60.61% (20/33) | 33.33% (11/33) | 64.00% (64/100) | 8.5 ms |
|
*Hardware & Latency Note (RTX 4050 6GB Laptop GPU, Windows WDDM):
- Architecture Difference (Prefill vs. Autoregressive Decode): Soro-Kev outputs routing + key-mapping decisions scoring candidate tools and an explicit
__abstain__ anchor via a dual linear pointer head in 1 single prefill forward pass (124.2 ms for 0.8B, 648.6 ms for 4B) with 0 generated decode tokens. In contrast, generative models (xLAM-2-1b-fc-r, Hammer2.1-1.5b) generate full JSON tool calls with argument values (~27–32 decode tokens), requiring 1 prefill pass plus 24-32 sequential autoregressive decode steps (one forward pass per token).
- In-VRAM vs. Sustained / Paged Latency: In clean isolated VRAM bursts,
Arch-Router-1.5B achieves a median latency of 1,212.9 ms (~24 tokens generated). xLAM-2-1b-fc-r (mean prompt 406 tokens, mean gen 27 tokens) achieves 7,106.9 ms median in clean VRAM (initial queries: ~1,458-2,549 ms; laptop GPU thermal throttling at 85C settles to ~7.1s). Hammer2.1-1.5b (generating full markdown JSON arrays of ~32 tokens) settles to 10,564.5 ms median. When multiple generative models were evaluated sequentially without intermediate process garbage collection, Windows WDDM silently paged KV caches into Shared GPU Memory across PCIe, increasing latencies to ~10.4s-11.5s.
- Dense Cosine & Neural Baselines: All 100 BFCL rows for Dense Cosine were evaluated directly using
SentenceTransformer / forward passes on CUDA timed with , yielding a genuine hardware median latency of (6.3-60.4 ms range, 64/100 correct).
- Agent Skills (
SKILL.md): 100.0% (29/29)
- GraphQL RPC: 96.8% (30/31)
- REST / OpenAPI: 96.2% (25/26)
- Mixed / Heterogeneous Catalogs: 94.6% (35/37)
- MCP JSON-RPC: 94.4% (34/36)
- Python Signatures: 87.8% (36/41)
Limitations and Trade-Offs
- No Generative Argument Synthesis: Soro-Kev routes tools, normalizes parameter keys, and validates completeness; it does not author free-text string values. In production pipelines, arguments are drafted by a fast parameter extractor, normalized via
arg_mapping, and guarded by missing_required before execution.
- Single-Turn Model: Soro-Kev evaluates single routing turns. Multi-tool sequential chaining ("call tool A, then tool B") is an architectural runtime design and has not yet been benchmarked on multi-step traces.
- Hardware & VRAM: Soro-Kev-4B requires ~4,048 MB VRAM in NF4 4-bit quantization. For edge devices with <2 GB VRAM, deploy Soro-Kev-0.8B (~1,563 MB, ~124.2 ms).
- Option Presentation Order: While option order is shuffled during training, slight score variances (<1.5%) can occur across permutations. Evaluate with averaged permutations for safety-critical tasks.
Provenance and Reproducibility
Citations
If you use Soro-Kev or Universal Tool Harness in your research or production systems, please cite the following publications:
@software{nisar2026sorokev, author = {Nisar, Keyush}, title = {Soro-Kev: System-One Conformal Tool Router and Execution Boundary}, year = {2026}, publisher = {GitHub}, howpublished = {\url{https://github.com/nisaral/Soro-Boro}}, note = {Release v0.2.0}} @misc{palmer2026kev08b, author = {Palmer, Jared}, title = {Kev-0.8B: a calibrated decision model on Qwen3.5-0.8B}, year = {2026}, howpublished = {\url{https://huggingface.co/jaredpalmer/kev-0.8b}}, note = {Kev 1.0}} @inproceedings{shi2025toolret, author = {Shi, Zhengliang and Gao, Shen and Chen, Xiuying and Wang, Lingzhi and Pei, Jiahuan and Guan, Xinyu and Ren, Zhaochun}, title = {Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2025}, pages = {11624--11639}, year = {2025}, url = {https://aclanthology.org/2025.findings-acl.647/}} @inproceedings{mishra2026mcpsemantic, author = {Mishra, Amritansh and Panditrao, Yash and Parab, Shlok and Sharma, Achintya and Mishra, Payal and Saha, Ishani}, title = {Scalable Semantic Routing and QoS-Aware Provider Selection for MCP Agents}, booktitle = {Procedia Computer Science, International Conference on Machine Learning and Data Engineering (ICMLDE 2026)}, year = {2026}, howpublished = {\url{https://github.com/amrittmishra/mcp-semantic-tool-routing-benchmark}}} @misc{mo2025livemcpbench, author = {Mo, Guozhao and others}, title = {LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?}, howpublished = {Preprint}, year = {2025}} @inproceedings{ross2025when2call, author = {Ross, Hayley and Mahabaleshwarkar, Ameya Sunil and Suhara, Yoshi}, title = {When2Call: When (not) to Call Tools}, booktitle = {Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)}, year = {2025}}