Use
git clone https://github.com/shibing624/train-jev.git && cd train-jevpip install -r requirements.txtpython -m trainjev.serve --base Qwen/Qwen3.5-4B \ --adapter shibing624/train-jev-4b-lora --port 8020
The first run downloads the backbone, LoRA, head.pt and saved temperatures. This is a custom decision adapter, not a chat/generation model: plain PEFT loading does not apply the decision head or calibration. Pin this release with --adapter_revision v0.4.0. API examples are in the serving guide.
Evaluation
Ours are the current weights. Other models are historical references, not rerun or newly matched in this round.
Table with columns: Evaluation, train-jev-4b-lora, train-jev-27b-lora, Kev-4B (historical), autotrust/JEV-27B-VL (historical), TypeSafe Jev v1.13 (historical API)| Evaluation | train-jev-4b-lora | train-jev-27b-lora | Kev-4B (historical) | autotrust/JEV-27B-VL (historical) | TypeSafe Jev v1.13 (historical API) |
|---|
| train-jev-pk accuracy, 2,494 questions ↑ | 76.03% | 79.52% | 63.50% | 73.47% | – |
| train-jev-pk ECE15 ↓ | 0.0930 | 0.0901 | 0.159 | 0.107 | – |
| Classification PK accuracy, 1,300 questions ↑ | 0.6523 | 0.6938 | 0.640 | 0.690 | 0.700 |
| Classification PK ECE10 ↓ | 0.0622 | 0.0377 | 0.083 | 0.094 | 0.130 |
| Kev transfer-v4 accuracy, 656 scored questions ↑ | 0.8201 | 0.8476 | 0.817 | 0.852 | 0.857 |
| Kev transfer-v4 ECE10 ↓ | 0.0588 | 0.0681 | 0.042 | 0.097 | – |
| Decision Index historical sample, 6,000 requests ↑ | 43.84 | 57.54 | 38.4 | 54.1 | 59.7 |
| Decision Index full public panel, 110,087 requests ↑ | 42.00 | 57.01 | – | – | – |
Exact metrics, prior-release comparison, benchmark regressions and evidence hashes are in metrics.json; protocols and context are in the results guide.
- PK uses 22-task macro accuracy and per-task upper-inclusive ECE15; classification uses pooled upper-inclusive ECE10. Transfer evaluated 764 original requests and scores 656 clean knowable questions, preserving Kev normalization and its ECE10 boundaries.
- The historical sample has 6,000 requests / 47,463 child answers, 41 input catalogs and 35 scoring benchmarks. Full public evaluation has 110,087 requests / 272,229 children over 37 scoring benchmarks. Six unindexed diagnostics were not requested: canonical suite43
complete=false is retained. These are composite DI scores, not accuracy or the private Full leaderboard.
- Prior-release controls are original stored predictions and CPU recompositions, not new same-stack inference. Old classification wide choices used tournament; current weights use native single-pass readout.
- Improvements are not universal. For 27B, transfer accuracy drops 85.8232% → 84.7561% and ECE rises 0.031566 → 0.068090. Negative DI benchmark deltas are retained, not hidden by the aggregate score.
Training
Rank16 / alpha32 LoRA and the 24-slot head are trained; the backbone is frozen. Initial Jev-distribution distillation is followed by gold/rule refinement and 12,947 admitted tasksource rows across nine sources, including 2,692 native soft targets. Sources include CommonsenseQA, CosmosQA, DynaSent rounds 1/2, WNUT17, LexGLUE CaseHOLD, Fig-QA, QuaRTz and Amazon Counterfactual English.
This release selects step 1,024 on a 2,048-step scheduler, seed17, batch128. The tasksource stage replaces 32 same-kind choice slots per batch; earlier gold mixing occurs before replacement and is not the final all-draw fraction. Actual phase exposures are recorded in training_recipe.json; calibration bytes and temperatures were not refitted for these evaluations.
The training guide explains the reusable public-data workflow. Processed training data, generators, optimizer/RNG and raw evaluation predictions are not bundled; exact one-command training reproduction is not provided.
Limitations and attribution
- PK includes training-source and shared-rule-family items with previous selection exposure; it is not a wholly unseen-source independent final test.
- After generation/verification, a schema probe displayed native labels in two PK provenance records. They were not used for generation, oracles, screening, selection or tuning, but strict zero-label-access was not fully preserved. No 100% contamination-free claim is made.
- New vision quality was not re-evaluated. Earlier 4B final/SDK/MemWrite8 evidence is archival; no new 27B SDK or MemWrite8 test is claimed. API support is not model-quality validation.
- Only choice rows A–P are trained; the extra 240 rows for wide choices are untrained base readouts. Confidence is not a universal correctness guarantee.
Adapter/head and code: Apache-2.0. Upstream data retains its own terms and attribution; see LICENSE and THIRD_PARTY_ATTRIBUTIONS.txt. Independent work, not affiliated with TypeSafe AI, autotrust or upstream authors.