Use
git clone https://github.com/shibing624/train-jev.git && cd train-jevpip install -r requirements.txtpython -m trainjev.serve --base Qwen/Qwen3.8-27B \ --adapter shibing624/train-jev-27b-lora --port 8020
The first run downloads the backbone, adapter, head.pt and saved temperatures. Plain PEFT loading does not apply the decision head or calibration; use train-jev, not a generative-chat pipeline. Pin this release with --adapter_revision v0.4.0. See the serving guide for the API.
Evaluation
Ours are the current weights. Other models are historical references, not rerun or newly matched in this round.
Table with columns: Evaluation, train-jev-4b-lora, train-jev-27b-lora, Kev-4B (historical), autotrust/JEV-27B-VL (historical), TypeSafe Jev v1.13 (historical API)| Evaluation | train-jev-4b-lora | train-jev-27b-lora | Kev-4B (historical) | autotrust/JEV-27B-VL (historical) | TypeSafe Jev v1.13 (historical API) |
|---|
| train-jev-pk accuracy, 2,494 questions ↑ | 76.03% | 79.52% | 63.50% | 73.47% | – |
| train-jev-pk ECE15 ↓ | 0.0930 | 0.0901 | 0.159 | 0.107 | – |
| Classification PK accuracy, 1,300 questions ↑ | 0.6523 | 0.6938 | 0.640 | 0.690 | 0.700 |
| Classification PK ECE10 ↓ | 0.0622 | 0.0377 | 0.083 | 0.094 | 0.130 |
| Kev transfer-v4 accuracy, 656 scored questions ↑ | 0.8201 | 0.8476 | 0.817 | 0.852 | 0.857 |
| Kev transfer-v4 ECE10 ↓ | 0.0588 | 0.0681 | 0.042 | 0.097 | – |
| Decision Index historical sample, 6,000 requests ↑ | 43.84 | 57.54 | 38.4 | 54.1 | 59.7 |
| Decision Index full public panel, 110,087 requests ↑ | 42.00 | 57.01 | – | – | – |
Exact metrics, prior-release comparison, benchmark regressions and evidence hashes are in metrics.json; protocols and context are in the results guide.
- PK uses 22-task macro accuracy and per-task upper-inclusive ECE15; classification uses pooled upper-inclusive ECE10. Transfer evaluated 764 original requests and scores 656 clean knowable questions, preserving Kev normalization and its ECE10 boundaries.
- The historical sample has 6,000 requests / 47,463 child answers, 41 input catalogs and 35 scoring benchmarks. Full public evaluation has 110,087 requests / 272,229 children over 37 scoring benchmarks. Six unindexed diagnostics were not requested: canonical suite43
complete=false is retained. These are composite DI scores, not accuracy or the private Full leaderboard.
- Prior-release controls are original stored predictions and CPU recompositions, not new same-stack inference. Old classification wide choices used tournament; current weights use native single-pass readout.
- 4B classification and transfer ECE also worsen. Negative DI benchmark deltas are retained in
metrics.json, not hidden by aggregate gains. Historical official/API values without matching raw proof are contextual references, not new verified runs.
Training
Rank16 / alpha32 LoRA and the 24-slot head are trained; the backbone is frozen. Initial Jev-distribution distillation is followed by gold/rule refinement and 12,947 admitted tasksource rows across nine sources, including 2,692 native soft targets. Sources include CommonsenseQA, CosmosQA, DynaSent rounds 1/2, WNUT17, LexGLUE CaseHOLD, Fig-QA, QuaRTz and Amazon Counterfactual English.
This release selects step 512 on a 2,048-step scheduler, seed17, batch128. The tasksource stage replaces 32 same-kind choice slots per batch; earlier gold mixing occurs before replacement and is not the final all-draw fraction. Actual 0–512 exposures are recorded in training_recipe.json, not estimated from pool proportions. Calibration is retained byte-identically and was never refitted to these results.
The training guide explains the reusable public-data workflow. Processed training data, generators, optimizer/RNG and raw evaluation predictions are not bundled; exact one-command training reproduction is not provided.
Limitations and attribution
- PK includes training-source and shared-rule-family items with previous selection exposure; it is not a wholly unseen-source independent final test.
- After generation/verification, a schema probe displayed native labels in two PK provenance records. They were not used for generation, oracles, screening, selection or tuning, but strict zero-label-access was not fully preserved. No 100% contamination-free claim is made.
- No new 27B final, SDK, MemWrite8 or vision test is claimed. Earlier 4B evidence is archival; API support does not establish new-weight quality or GPU/HTTP bitwise equivalence.
- Only choice rows A–P are trained; the extra 240 rows for wide choices are untrained base readouts. Confidence is not a universal correctness guarantee.
Adapter/head and code: Apache-2.0. Upstream data retains its own terms and attribution; see LICENSE and THIRD_PARTY_ATTRIBUTIONS.txt. Independent work, not affiliated with TypeSafe AI, autotrust or upstream authors.