Results
Decision Index edition 0.3 public-only index: 57.03, versus 55.11 for the previous 27B, a +1.92 point change. These are self-run public results with the unchanged official scorer. The board computes the full score with private parts after submission. This is not an accepted full leaderboard score.
Table with columns: Family, Previous 27B, v2.1, Delta| Family | Previous 27B | v2.1 | Delta |
|---|
| Knowledge & Reasoning | 40.78 | 43.12 | +2.34 |
| Language Understanding | 59.66 | 63.55 | +3.89 |
| Retrieval & Classification | 53.89 | 56.70 | +2.81 |
| Tools & Automation | 78.13 | 76.03 | -2.10 |
| Arts & Human Taste | 40.73 | 42.04 | +1.31 |
Tools declined 2.10 points. When2Call fell 6.38 points (71.77 to 65.39), and home appliance simulator fell 3.41 (72.73 to 69.32). The aggregate gain does not remove these regressions. No statistical significance is claimed.
All 140,178 scored requests succeeded, across 37 index benchmarks and 43 reported benchmarks; all five family coverages are 1.0. The frozen public corpus and payload hashes match the previous run. Full comparison: eval/public-comparison.csv.
Table with columns: In-process request latency, RTX PRO 6000, Previous, v2.1| In-process request latency, RTX PRO 6000 | Previous | v2.1 |
|---|
| Median | 110.2 ms | 106.5 ms |
| p95 | 603.3 ms | 599.5 ms |
| Mean | 252.2 ms | 247.3 ms |
These device-synchronized timings include rendering, tokenization and label-logit reads, exclude model loading, and use shared-prefix caching and batching. Individual requests can exceed 1000 ms. They do not estimate CPU latency.
Run it anywhere
Use the standalone renderer and label-logit scorer shipped in scripts/:
pip install torch transformers==5.17.0 huggingface_hubhf download frontier-infra/jebadiah-27b-v2-1 --local-dir jebpython jeb/scripts/decide_standalone.py --model jeb --request jeb/scripts/example-request.json
For CPU add --device cpu; for Apple silicon add --device mps. The bf16 weights occupy about 56 GB before runtime overhead. --no-temperatures returns the raw distribution. temperatures.json contains the v2.1 fitted per-type calibration, not the previous release's calibration. The standalone server exposes /v1/systemone and /v1/decide; follow the linked local-run guide and explicitly select this repository.
How it decides
The shipped prompt renderer is the training renderer. It applies the chat template with thinking off, reads fp32 candidate-label logits at the last prompt position and applies the temperature for the question type. Thresholds belong to the caller. Score and noul answers are interpreted using the typed question's rubric.
Training
LoRA rank 16, alpha 32 on all linear projections, learning rate 1e-4, one epoch, effective batch size 8, seed 17. Base revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. The merged checkpoint and original training provenance are shipped unchanged. Per-type temperatures are choice 1.0608, noul 1.0405 and score 0.6524 (hard-label fit for score).
Training data
The disclosure-safe manifest summary is data-summary.json. Training has 16,622 records and 21,190 question presentations; calibration has 280 records and 512 questions. Training presentations include 11,708 choice, 3,696 noul and 5,786 score questions. Source names, licences and source-pool counts are listed below; pool counts are provenance counts before presentation selection and must not be summed as final split counts.
Whole-family overlap exclusion was confirmed by two independent reconstructions. Both reconstructed checks found zero direct hits over all 16,902 retained train/calibration records, with all 22 excluded families absent and none of the original 42 flagged record IDs retained. The protected public suite includes all 140,620 requests, including scoring exclusions. These checks do not prove semantic independence or remove possible base-model pretraining overlap. Private diagnostic sources from later experiments are not included.
Table with columns: Source, Licence, Source-pool records, Source-pool questions| Source | Licence | Source-pool records | Source-pool questions |
|---|
| 12ml/e-CARE | BSD 2-Clause License | 986 | 986 |
| agent_action_safety | Apache-2.0 | 687 | 687 |
| allenai/qasc | cc-by-4.0 | 885 | 885 |
| allenai/quartz | cc-by-4.0 | 827 | 827 |
Limitations
English-focused, text-only training and calibration. Training prompts are capped at 2,048 tokens. Calibration was fitted on the training distribution; refit for a new domain. Prefer questions with at most 20 options for ordinary served integrations. Image decisions and multi-hop reasoning are untested. Earlier Jevals and JevBench scores belong to earlier weights; the v2.1 GGUF and MLX builds were checked for agreement with these weights (see their cards).
Versioning and license
v2.1 uses a new repository; previous releases remain separately addressable. Weights and base: Apache-2.0. Source data retains its own licences and attribution requirements. See LICENSE and data-summary.json.
Support: help@myagents.email
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt.