Results
Decision Index edition 0.3 public-only index: 47.20, versus 44.09 for the published 9B v2, a +3.11 point change. These are self-run public results with the unchanged official scorer. The board computes the full score with private parts after submission. This is not an accepted full leaderboard score.
Table with columns: Family, Previous 9B v2, v2.1, Delta| Family | Previous 9B v2 | v2.1 | Delta |
|---|
| Knowledge & Reasoning | 29.22 | 32.89 | +3.67 |
| Language Understanding | 47.90 | 52.24 | +4.34 |
| Retrieval & Classification | 43.60 | 49.56 | +5.96 |
| Tools & Automation | 66.68 | 65.84 | -0.84 |
| Arts & Human Taste | 32.36 | 32.37 | +0.01 |
Tools declined 0.84 points. The biggest index-benchmark losses are When2Call (-7.44 points) and Humicroedit (-7.30). SGD/SGD-X fell to 0.00, down 28.14 points; this benchmark is display-only and is not included in the index. The aggregate gain does not remove these regressions. No statistical significance is claimed.
All 140,178 scored requests succeeded, with zero failures and full coverage across all five families. The run has 140,620 successful raw requests including 442 scoring exclusions, 37 index benchmarks and 43 reported benchmarks. Full comparison: eval/public-comparison.csv.
Table with columns: In-process request latency, RTX PRO 6000, v2.1| In-process request latency, RTX PRO 6000 | v2.1 |
|---|
| Median | 49.9 ms |
| p95 | 209.2 ms |
| Mean | 97.1 ms |
These device-synchronized timings include rendering, tokenization and label-logit reads, exclude model loading, and use shared-prefix caching and batching. They do not estimate CPU latency.
Run it anywhere
Use the standalone renderer and label-logit scorer shipped in scripts/:
pip install torch transformers==5.17.0 huggingface_hubhf download frontier-infra/jebadiah-9b-v2-1 --local-dir jebpython jeb/scripts/decide_standalone.py --model jeb --request jeb/scripts/example-request.json
For CPU add --device cpu; for Apple silicon add --device mps. The bf16 weights occupy about 19 GB before runtime overhead. --no-temperatures returns the raw distribution. temperatures.json contains the v2.1 fitted per-type calibration, not the previous release's calibration. The standalone server exposes /v1/systemone and /v1/decide; follow the linked local-run guide and explicitly select this repository.
How it decides
The shipped prompt renderer is the training renderer. It applies the chat template with thinking off, reads fp32 candidate-label logits at the last prompt position and applies the temperature for the question type. Thresholds belong to the caller. Score and noul answers are interpreted using the typed question's rubric.
Training
LoRA rank 16, alpha 32 on all linear projections, learning rate 1e-4, one epoch, effective batch size 8, seed 17. Base revision: c202236235762e1c871ad0ccb60c8ee5ba337b9a. The merged checkpoint and original training provenance are shipped unchanged. Per-type temperatures are choice 1.0467, noul 1.0284 and score 0.6805 (hard-label fit for score).
Training data
The disclosure-safe manifest summary is data-summary.json. Training has 16,622 records and 21,190 question presentations; calibration has 280 records and 512 questions. Training presentations include 11,708 choice, 3,696 noul and 5,786 score questions. Source names, licences and source-pool counts are listed below; pool counts are provenance counts before presentation selection and must not be summed as final split counts.
Whole-family overlap exclusion was confirmed by two independent reconstructions. Both reconstructed checks found zero direct hits over all 16,902 retained train/calibration records, with all 22 excluded families absent and none of the original 42 flagged record IDs retained. The protected public suite includes all 140,620 requests, including scoring exclusions. These checks do not prove semantic independence or remove possible base-model pretraining overlap. Private diagnostic sources from later experiments are not included.
Table with columns: Source, Licence, Source-pool records, Source-pool questions| Source | Licence | Source-pool records | Source-pool questions |
|---|
| 12ml/e-CARE | BSD 2-Clause License | 986 | 986 |
| agent_action_safety | Apache-2.0 | 687 | 687 |
| allenai/qasc | cc-by-4.0 | 885 | 885 |
| allenai/quartz | cc-by-4.0 | 827 | 827 |
Limitations
English-focused, text-only training and calibration. Training prompts are capped at 2,048 tokens. Calibration was fitted on the training distribution; refit for a new domain. Prefer questions with at most 20 options for ordinary served integrations. Image decisions and multi-hop reasoning are untested. Earlier Jevals and JevBench scores belong to earlier weights; the v2.1 GGUF agreement check is documented on its card; see the linked MLX cards for their build and agreement details.
Versioning and license
v2.1 uses a new repository; previous releases remain separately addressable. Weights and base: Apache-2.0. Source data retains its own licences and attribution requirements. See LICENSE and data-summary.json.
Support: help@myagents.email
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt.