At a glance
- 83.4% end-to-end accuracy on 554 held-out tasks (math, knowledge,
science, code, long context, charts) at 0.42x the cost of always using the
strongest agent.
- Beats every single agent used alone (best: Always
glm-5.3-flash at 79.2%) and the
strongest agent alone (78.5% at 1.00x).
- Learns capability routing from data: charts -> the only vision agent, very long inputs -> the long-context agents.
- Does not beat hand-written capability rules overall: Capability rules ->
glm-5.3-flash reaches
85.7% at 0.13x. See Results and Findings.
- 100.0% of outputs on the test set are valid route JSON.
{"domain": "math", "difficulty": "easy|medium|hard", "strategy": "single|ensemble", "agents": ["glm-5.3-flash"]}
Agent pool
Table with columns: agent, model, context, image input (in this pool), relative cost / token| agent | model | context | image input (in this pool) | relative cost / token |
|---|
gemma-3-1b | google/gemma-3-1b-it | 32k | no | 0.02 |
kimi-vl | Kimi-VL-A3B-Thinking | 8k | yes | 0.05 |
nemotron-lightning | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 | 128k | no | 0.1 |
|
Agents are reached through OpenAI-compatible chat-completions endpoints. "Relative cost" is a per-token weight used
to compute routing cost (roughly relative serving cost), not a vendor price.
Quick start
import jsonfrom huggingface_hub import hf_hub_downloadfrom transformers import AutoModelForCausalLM, AutoTokenizerimport importlib.util repo = "podstack/Shaktimaan"spec = importlib.util.spec_from_file_location("common", hf_hub_download(repo, "common.py"))common = importlib.util.module_from_spec(spec); spec.loader.exec_module(common) tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo)task = "Write a Python function that returns the n-th Fibonacci number."msgs = [{"role": "system", "content": common.ROUTER_SYSTEM}, {"role": "user", "content": common.router_view(task, has_image=False)}]enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)out = model.generate(**enc, max_new_tokens=96, do_sample=False)print(json.loads(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True)))
Always build the input with common.router_view(task, has_image) - the router was trained on that exact format (a
[input ~Nk tokens; 1 image attached] header plus the first 5,000 and last 1,500 characters of the task) with
common.ROUTER_SYSTEM as the system prompt.
End-to-end orchestration
orchestrate.py routes with Shaktimaan, drops agents that cannot take the input, then calls the chosen agent(s) and
aggregates ensembles:
export SHAKTIMAAN_ENDPOINTS='{"kimi-vl": "http://host-a:8000/v1", "nemotron-lightning": "http://host-b:8000/v1", ...}'python orchestrate.py "If a train travels 120 km in 1.5 hours, what is its average speed?"python orchestrate.py --code "Write a function that checks whether a string is a palindrome."python orchestrate.py --image chart.png "What was the revenue in Q2 2021?"
Training data
4989 train / 554 test tasks across eight sources. Every capable agent attempted every
task once and was graded automatically; the label is the cheapest reliable agent that solved the task (or an
ensemble when none did). Full details, solve rates per agent and the label policy are in the
dataset card.
Table with columns: source, category, train, test, grading, license| source | category | train | test | grading | license |
|---|
| GSM8K | math word problems | 895 | 105 | numeric match | MIT |
| MATH | competition math | 894 | 106 | normalised LaTeX / numeric match |
Training
Table with columns: setting, value| setting | value |
|---|
| base model | Qwen2.5-0.5B-Instruct (494M parameters) |
| method | full fine-tune, supervised; loss on the route JSON only |
| epochs | 3 |
| learning rate | 2e-5, cosine schedule, 20 warm-up steps |
| effective batch size | 32 |
| max sequence length | 3,072 tokens |
| precision | bf16 |
| final eval loss | 0.0484 |
Results
Held-out test set: 554 tasks the router never saw. A policy's accuracy is whether the agent it picks
actually solved the task. Shaktimaan's 31 ensemble decisions were executed live (all agents
plus aggregator called and graded); single-agent decisions use the recorded outcomes. Cost is the sum of
relative cost x output tokens, normalised so that always using nemotron-ultra = 1.00.
Table with columns: policy, accuracy, relative cost (always-Ultra = 1.00)| policy | accuracy | relative cost (always-Ultra = 1.00) |
|---|
| Shaktimaan v1.0 | 83.4% | 0.424 |
Capability rules -> glm-5.3-flash | 85.7% | 0.133 |
Capability rules -> qwen3.8-27b | 85.4% | 0.403 |
Capability rules -> nemotron-ultra | 82.7% | 1.001 |
| Always |
Capability rules are a hand-written baseline: image -> kimi-vl; input too long for the default agent -> the
cheapest agent with a large enough context window; otherwise the default agent. Oracle is the hindsight-best
training label (an upper bound; its ensembles are scored as the aggregator's solo answer).
Accuracy by source
Table with columns: source, n, Shaktimaan, rules -> glm, rules -> qwen, always Ultra, oracle| source | n | Shaktimaan | rules -> glm | rules -> qwen | always Ultra | oracle |
|---|
| ARC-Challenge | 75 | 98.7% | 98.7% | 98.7% | 98.7% | 100.0% |
| ChartQA | 31 | 74.2% | 74.2% | 74.2% | 0.0% | 74.2% |
Routes chosen on the test set
Table with columns: route chosen by Shaktimaan, test tasks| route chosen by Shaktimaan | test tasks |
|---|
nemotron-lightning | 218 |
glm-5.3-flash | 192 |
kimi-vl | 80 |
glm-5.3-flash,qwen3.8-27b,nemotron-ultra | 32 |
nemotron-ultra | 31 |
qwen3.8-27b |
Findings
- Routing beats any single model. Shaktimaan is 4.1
points above the best single agent and 4.9
points above always using the strongest (and most expensive) one, at
0.42x its cost.
- Capability routing is learned from data. On long-context tasks Shaktimaan matches the oracle
(59.4%) and beats the capability rules (46.9%),
sending inputs to the agent most likely to solve them rather than merely one that fits. On charts it routes to the
only vision agent, as the rules do.
- Simple capability rules are stronger overall (85.7% at
0.13x). On ordinary text and code tasks the mid-size agents are close to
interchangeable, and which one solves a particular task is largely sampling noise that a router cannot predict from
the prompt. Shaktimaan sometimes picks the cheaper
nemotron-lightning or kimi-vl where the rule's fixed
glm-5.3-flash default is slightly more reliable, and its ensembles (chosen for hard tasks) solved
11 of the 31 hard tasks they were sent, at a high cost.
- Practical recommendation: for this pool, apply capability rules first and use Shaktimaan where its learned
judgement helps most - choosing among long-context agents and flagging hard tasks.
Limitations
- Routes only among the six agents above, with their context windows and image support; a different pool needs new
data and retraining.
- Labels come from a single sample per agent and inherit grading noise (e.g. answer extraction from free text).
- Not trained on agentic / tool-use, multi-turn or open-ended writing tasks; English only.
est_tokens uses a conservative characters/3 estimate, so borderline inputs may be routed to a larger-context
agent than strictly needed.
Files
Table with columns: file, purpose| file | purpose |
|---|
model.safetensors, config.json, tokenizer files | the router |
common.py | agent pool, prompts, router_view, capability checks, graders |
orchestrate.py | end-to-end runtime (route -> call agents -> aggregate) |
gen.py | builds tasks and collects graded agent outcomes |
build.py |
License
Model weights: Apache-2.0 (as the base model). Training data: see the
dataset card (CC-BY-SA-4.0; ChartQA rows GPL-3.0, not redistributed).