jbrashear

jebadiah-4b-v0

README

License: apache-2.0

Results

Measured by us with AINode's bench, one logit read per question, the same rendered prompt for every model. Accuracy is the share of questions whose top label is the human label. Decision Score is Jevals' metric: 100 = perfect, 0 = guessing the label base rates, below 0 = worse than that. These are our numbers on the public suites, not rows on the Jevals board.

Table with columns: Set (questions), Type, Accuracy, Floor (majority label), Decision Score (Jevals), ECE, temperatures applied, Flips over identical repeats
Set (questions)	Type	Accuracy	Floor (majority label)	Decision Score (Jevals)	ECE, temperatures applied	Flips over identical repeats
Jevals PubMedQA (300)	noul	87.7	62.0	62.0	0.021	0.3%
Jevals Banking77 (300, 77 options)	choice	68.7	1.3	53.8	0.110	0.7%
Jevals HelpSteer2 helpfulness (300, 5 levels)	score	36.3	41.7	-20.2	0.439 (raw 0.249)	0.3%
Nimble held-out eval (324)	mixed	77.5	17.6	59.2	0.082	0.0%
Kev transfer-v4 test (764)	mixed	82.5	21.5	63.6	0.062	0.3%
Kev decision-v7 test (1,440)	mixed	78.5	20.3	66.8	0.118	n/a
typed-decisions test (2,000)	mixed	79.8	15.3	59.0	0.077	0.1%

Nimble's 13 public human-labelled subsets (3,880 questions), macro accuracy: 73.0 (per subset: aegis2 80.0, boolq 87.0, civil_comments 79.3, helpsteer2 33.3, massive-de-DE 82.6, massive-en-US 84.6, multinli 83.6, paws 79.2, pubmedqa 68.0, squad2 73.6, summeval-consistency 79.2, summeval-relevance 45.4, vitaminc-dev 73.6). For scale, Bespoke's published table puts Nimble-9B at 74.8 and Jev at 76.0 on the same subsets with their scorer, and the untrained 4B base at 65.7 with ours.

Where it is weak: the score type on human rubrics. HelpSteer2 helpfulness and summeval-relevance sit at their floors, and the fitted score temperature (0.40, a sharpening learned from soft teacher targets) makes HelpSteer2's calibration worse rather than better. That is the first thing v1 changes.

Every eval record (per question: option keys, probabilities, pick, label, repeat, option order) is in eval/, with eval/results.json carrying the full metric set (Brier, NLL, ECE raw and fitted, per-question floors, flips with their top-2 gaps) and the resolved training configuration.

How it decides

The prompt is AINode's own decide rendering (ainode.api.decide.build_messages, source commit e5c08938, prompt_source_sha256 d2660ebe... in prompt_contract.json), through the base chat template with thinking off. The question's state, instructions and options are rendered as structured text; the option labels are single tokens (68 of them); the model's answer is the distribution over those label tokens at the last prompt position, read in fp32 from the last hidden state and then temperature scaled per type (temperatures.json: choice 0.95, noul 1.09, score 0.40, fitted on a held-out calibration slice of the training pool). Nothing is generated.

The supported way to run it is AINode, which renders the prompt exactly as trained:

bash

curl -sS https://<your-ainode>/v1/systemone -H "Authorization: Bearer <key>" -H "Content-Type: application/json" -d '{  "model": "jebadiah/jebadiah-4b-v0",  "state": {"ticket": "Customer says the invoice total does not match the quote."},  "questions": [    {"id": "route", "type": "choice", "instructions": "Which team should take this ticket?", "options": ["billing", "support", "sales"]},    {"id": "urgent", "type": "noul", "instructions": "The customer is blocked from working."}  ]}'

Standalone with transformers and peft (the label tokens, rendering and fp32 read must match the contract; see prompt_contract.json and the AINode source for the renderer):

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = "Qwen/Qwen3.5-4B-Base"tok = AutoTokenizer.from_pretrained(base, revision="1001bb4d826a52d1f399e183466143f4da7b741b")model = AutoModelForCausalLM.from_pretrained(base, revision="1001bb4d826a52d1f399e183466143f4da7b741b", torch_dtype=torch.bfloat16, device_map="cuda")model = PeftModel.from_pretrained(model, "jbrashear/jebadiah-4b-v0").eval() messages = [{"role": "user", "content": "<the AINode decide rendering of state, question and options>"}]ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to("cuda")with torch.no_grad():    h = model(ids, output_hidden_states=True).hidden_states[-1][0, -1].float()labels = ["A", "B", "C"]                      # the option labels the rendering assignedlabel_ids = [tok.encode(l, add_special_tokens=False)[0] for l in labels]W = model.get_output_embeddings().weight[label_ids].float()probs = torch.softmax((W @ h) / 0.95, dim=0)   # choice temperature from temperatures.json

Thresholds belong to the caller: act on a high probability, confirm or escalate on a middle one, hand a low one to a person or a bigger model. The model never refuses; policy is built from decisions.

Training

Objective: cross-entropy over the candidate option-label logits at the answer position, with the source's gold distribution as the target where it provides one (typed-decisions) and the hard label otherwise. No text generation is trained.
Adapter: LoRA r=16, alpha=32, dropout 0.05 on every linear projection including the Gated DeltaNet projections (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj, plus the attention and MLP projections), 32.5M trainable parameters. bf16, SDPA attention, micro-batch 8, learning rate 1e-4, one epoch, 1,384 steps, gradient checkpointing.
Data (public only): LocalLLaMA/typed-decisions train (Apache-2.0, revision ea930645) and the Kev v7 training sources whose licenses permit derived weights (BoolQ, MNLI, DBpedia14 and Kev's contrastive and composition sets, Kev suites revision a88f56db). 11,072 training and 272 calibration questions after a 95/5 split by question family. Nimble's train set is excluded (no license stated); the Jevals test sources are excluded from training. No private data of any kind.
Compute: one NVIDIA H100 PCIe 80 GB, 39 min training, 23 min evaluation, torch 2.11, PEFT 0.21.

Limitations

Single-hop judgments only. A question that hides a chain of inference should be split into hops.
The score type is not yet trained on human helpfulness rubrics; treat score outputs on such rubrics as uncalibrated until v1.
AINode caps a choice question at 20 options; Banking77's 77 options were scored with an extended single-token alphabet for the benchmark only.
English data. Massive's German subset scores well, but nothing else was checked.
Calibration was fitted on the training distribution. Refit the temperatures on your own data before trusting a threshold.

Versioning and license

v0 is frozen. Later versions land as jebadiah-<size>-v<N> repositories. The adapter is Apache-2.0, the base model is Apache-2.0. Evaluation data: Jevals suite 0.1.0 (CC-BY-4.0, "Jevals (jevals.com), release 2026-09-18"), Nimble public subsets (Bespoke Labs), Kev test sets and typed-decisions test, each under its own license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

jbrashear

Model Tree

Base

Qwen/Qwen3.5-4B-Base

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality