Evaluation
Worker under orchestrator, concurrency 64; BrowseComp uses four TP1 replicas, DR9K uses TP1 native DP4. BrowseComp has 150 questions (50 per difficulty); DR9K has 256 (128 frontier + 128 challenge). This is not solo-model accuracy. Repository manual protocol uses normalized exact matching, identical-answer precedents and full-answer agent review, not the canonical Gemini judge. Empty and failed episodes count wrong.
Table with columns: Orchestrator, BrowseComp, DR9K| Orchestrator | BrowseComp | DR9K |
|---|
| MiniMax-M2.7 | 103/150 | 112/256 |
| DeepSeek-V4-Flash | 99/150 | 152/256 |
| Nemotron Ultra | 99/150 | 137/256 |
| MiMo-V2.5 | 102/150 | 149/256 |
| Nemotron Super | 96/150 | 128/256 |
| Inkling-Small | 106/150 | 159/256 |
Full raw/SFT report includes exact latency settings, difficulty scores, role token counts and measured versus theoretical prefix reuse. BrowseComp traces and DR9K traces.
Use the per-benchmark chat templates and evaluator in the report artifacts to reproduce worker behavior. Generic chat generation does not reproduce the search tools, budgets or orchestrator setup.
Grading erratum (2026-09-18): MiMo and Inkling DR9K each gain one answer under the existing accepted Bretton Woods Agreement alias rule. Predictions and weights unchanged.