Results
The Oct 21 test file, 6,035 rows, scored per row with calibrated answers, gold = the human / source label. Every row that
shared a state or a group with this file was removed from training first (5,645 rows).
Table with columns: Test, Rows, This model, Qwen3.5-4B-Judge (Oct 21, human labels), Jev 1.14| Test | Rows | This model | Qwen3.5-4B-Judge (Oct 21, human labels) | Jev 1.14 |
|---|
| RAG claim checks | 1,873 | 89.1% | 87.9% | 82.1% |
| Safety decisions | 2,162 | 89.8% | 90.7% | 82.3% |
| RAG + safety | 4,035 | 89.5% | 89.4% | 82.2% |
| MedHallu PubMed answers, at the 0.5 cut | 2,000 | 76.8% | 78.4% | 67.9% |
| MedHallu, real answer ranked above the fake one | 1,000 pairs | 862 | 855 | 916 |
- On RAG + safety it matches the Oct 21 judge (3,611 vs 3,609 rows right) while being trained on a broad mix of 20
task families instead of those two sets alone.
- Jev 1.14 ranks real PubMed answers above fake ones better (916 vs 862). Jev's default cut is strict on PubMed, so
don't read 76.8% vs 67.9% on its own. Tune the threshold for your data.
- Jev 1.14's answers are the ones from the Oct 21 run (TypeSafe, no fallbacks), same state, instructions and criteria.
On the 1M's own test split (1,927 rows, 15 families), the families with enough rows:
Table with columns: Family, Rows, Accuracy| Family | Rows | Accuracy |
|---|
| Intent routing (banking) | 300 | 90.7% |
| Conversation domain routing | 278 | 91.7% |
| Pairwise response preference | 169 | 69.8% |
| Principle fulfilment | 125 | 78.4% |
| When to call a tool | 300 | 50.0% |
when_to_call_tool was not in its training sample and sits at chance: don't use it for that. Families with 12 test rows
or 100% on what may be a single class say nothing and are left out.
Use
from unsloth import FastDecisionModel model, tokenizer = FastDecisionModel.from_pretrained("MaziyarPanahi/Qwen3.5-4B-Judge-1M", max_seq_length=4096, load_in_4bit=False)FastDecisionModel.for_inference(model)question = {"decision": {"type": "choice", "instructions": "Is every factual claim in `assistant_response` supported by `authoritative_context`?", "criteria": {"true": "Every factual claim in the response is supported by the context.", "false": "The response contains at least one unsupported or contradicted claim."}}}state = '{"authoritative_context": "The clinic opens at 8 a.m. on weekdays and is closed on Sundays.", '\ '"assistant_response": "The clinic is open every day from 8 a.m."}'print(FastDecisionModel.predict(model, tokenizer, state, question))
Training
- Data: 39,992 rows sampled from the
choice and noul decisions of OpenDecisionMix-1M, capped at 2,505 per task
family, states up to 5,000 characters, seed 3407. 39,592 train, 400 held out to calibrate. Labels are the source labels;
every jev_* column was removed from the training copy.
- Recipe: Unsloth
FastDecisionModel, bf16 LoRA r=16, lr 2e-4, batch 8 x 4, max length 4,096.
- Hardware: one A100 on Hugging Face Jobs, 1 h 40 min of training, stopped by a time budget at 0.97 of the epoch
(about 5.30atthelisted2.50/h).
Limits
- Not a clinical model. PubMed answers are a transfer test, not medical validation. It must not decide anything
about a patient.
- Home advantage. The RAG and safety test rows come from the same sources as its training rows. Jev 1.14 answers
zero-shot. The Oct 21 judge has the same advantage, so that column is the fair comparison.
- Long inputs are untested: training states were at most 5,000 characters, and the model reads at most 4,096 tokens.
Credit