What a System 1 model is, and why answer this way
Psychologists separate fast, intuitive judgement (System 1) from slow, step-by-step reasoning
(System 2). TypeSafe AI uses the name System One model for models built around the first kind:
the possible answers are decided in advance, and the model makes a single quick judgement over them
with a probability attached, rather than writing an answer out word by word. CareJev-Omni is a
System 1 model for clinical records, images and signals, and answering this way has its own
advantages:
- The answer is already structured. The question comes with its allowed answers and the model
returns a probability for each one; there is no prose to read, parse or validate, and every answer
is one of the options asked about.
- Its confidence tracks how often it is right. When it says 80 %, it is correct about 80 % of the
time: across all held-out test questions, stated confidence and actual accuracy differ by under 2
points on average (expected calibration error 0.018). Higher confidence means more likely correct:
on the half of questions it is most sure about, accuracy is 0.897, against 0.741 over all questions
(see the reliability diagrams below). A confidence threshold can therefore decide which answers to
use as they are and which to pass to a slower process — a reasoning model or a person.
- Position cannot steer it. The order in which the answers are listed does not change the result.
- It is fast and predictable. One forward pass per question, with no generation loop and no
variable-length output, and the same interface for text, X-rays, ECGs and heart sounds.
The framing also sets the limit: a fast judgement suits screening many questions and flagging the
uncertain ones in research pipelines. It does not replace deliberate clinical reasoning, and this
model is not validated for any decision about a real patient (see the notice below).
Research use only — not a medical device
This model is released for research purposes only. It is not a medical device, has not been
reviewed or cleared by any regulator, and has not been validated on any live clinical population.
Do not use it for diagnosis, triage, treatment, trial eligibility, resource allocation or any
other decision about a real person. It is distributed "as is", without warranty of any kind,
express or implied, and without any representation of fitness for a particular purpose; the
authors accept no liability for any use of it or of its outputs. Anyone evaluating it does so
at their own risk and remains responsible for complying with the licences below and with all
applicable law and research-ethics requirements. Its outputs
are probability estimates from an incomplete picture of a patient, carry the limitations listed
under Limitations, and must never substitute for the judgement of a qualified clinician.
Results — held-out test split (patient-disjoint)
Every row is the same test items under an identical protocol for all systems: same prompts, same
option lists, same split. base is the untouched google/gemma-4-12B-it scored by its next-token distribution
over the option numbers (it placed ≥ 99 % of its mass on valid options). Jev-Omni, when present, is the published
akhilaaa3/Jev-Omni checkpoint run through its own loader
on the same items. majority is the majority-class rate. The best system in each row is in bold;
bootstrap 95 % CIs (500 resamples) are given for the pooled figures. 6 sources measured here are not tabulated — a model given only the patient's sex and age answers them as well as this one, so per-source numbers there would not measure what they appear to measure; they remain in the totals and in the run artifacts.
By task family
15,629 held-out test questions.
Table with columns: Task family, majority, CareJev, base, Jev-Omni| Task family | majority | CareJev | base | Jev-Omni |
|---|
| 12-lead ECG (rendered) (15) | 0.737 | 0.792 | 0.587 | 0.529 |
| Heart sounds (3) | 0.694 | 0.700 | 0.429 | 0.404 |
| Chest X-ray | 0.490 | 0.870 | 0.694 | 0.571 |
CareJev 95 % CI: pooled 0.741 (0.734–0.747); mean over sources 0.739 (0.728–0.750).
Ranking quality (AUROC, yes/no questions, positive = "Yes")
Table with columns: Task family, CareJev, base, Jev-Omni| Task family | CareJev | base | Jev-Omni |
|---|
| 12-lead ECG (rendered) (15) | 0.812 | 0.672 | 0.656 |
| Heart sounds (3) | 0.626 | 0.495 | 0.543 |
| ICU outcomes (SUPPORT2) (4) | 0.829 | 0.650 | 0.651 |
| EHR demo (eICU / MIMIC-IV) (5) | 0.650 | 0.571 | 0.508 |
Calibration (expected calibration error, 10 equal-width bins; lower is better)
Table with columns: Task family, CareJev, base, Jev-Omni| Task family | CareJev | base | Jev-Omni |
|---|
| 12-lead ECG (rendered) (15) | 0.062 | 0.331 | 0.197 |
| Heart sounds (3) | 0.148 | 0.472 | 0.250 |
| Chest X-ray | 0.044 | 0.268 | 0.187 |
| Sleep staging (rendered PSG) | 0.066 | 0.491 | 0.145 |
Family rows weight each question by its test items (so accuracy is the pooled family accuracy); AUROC
is the mean over the family's yes/no questions; the number in brackets is how many questions a family
has (the datasets are credited under Data). The checkpoint was selected on the dev split only; the test split was never used for selection or
calibration.
One-shot external benchmarks (never trained on, run once)
Every set below was converted to the typed-decision format and scored once, after the internal
test, for CareJev, the untouched base model and the published Jev-Omni (through its own loader) on
identical items. Nothing was tuned on them. They answer two different questions: the knowledge sets
(MedQA, MMLU, PubMedQA, DecisionBench) ask what the fine-tune kept or lost of the base model's general
ability; the shift sets (VQA-RAD, MedMNIST, CinC-2016, PTB-XL) ask whether the clinical training
transfers to other hospitals, devices and question wordings.
Accuracy
Table with columns: benchmark, majority, CareJev, base, Jev-Omni| benchmark | majority | CareJev | base | Jev-Omni |
|---|
| MedQA-USMLE (4-option) | 0.277 | 0.738 | 0.687 | 0.701 |
| MMLU medical (8 subjects) | 0.304 | 0.821 | 0.821 | 0.812 |
| PubMedQA (yes/no/maybe) | 0.552 | 0.746 | 0.732 | 0.738 |
DecisionBench's own scoring (state-macro accuracy · confidence − accuracy gap): CareJev medium 0.821 (-14 pp), hard 0.555 (+2 pp) · base medium 0.771 (+19 pp), hard 0.551 (+37 pp) · Jev-Omni medium 0.882 (+1 pp), hard 0.643 (+15 pp)
Calibration (ECE, lower is better)
Table with columns: benchmark, CareJev, base, Jev-Omni| benchmark | CareJev | base | Jev-Omni |
|---|
| MedQA-USMLE (4-option) | 0.054 | 0.209 | 0.107 |
| MMLU medical (8 subjects) | 0.038 | 0.161 | 0.100 |
| PubMedQA (yes/no/maybe) | 0.029 | 0.244 | 0.171 |
| DecisionBench medium | 0.146 | 0.198 | |
PTB-XL: compare AUROC, not accuracy. We render the PTB-XL ECGs in the 12-lead paper layout this model was trained on. That layout drags the base model's accuracy down (hypertrophy 0.176, against 0.856 on a plainer strip layout) without changing how well it ranks cases, so accuracy would overstate the margin. AUROC does not depend on a decision threshold, and CareJev has the highest AUROC on all 5 tasks, against either layout:
Table with columns: PTB-XL task (AUROC), CareJev, base, Jev-Omni, base, strip layout| PTB-XL task (AUROC) | CareJev | base | Jev-Omni | base, strip layout |
|---|
| conduction disturbance | 0.665 | 0.603 | 0.634 | 0.605 |
| hypertrophy | 0.664 | 0.442 | 0.447 | 0.603 |
| myocardial infarction | 0.781 | 0.663 | 0.653 | 0.670 |
Caveats that belong next to those numbers: MedQA and MMLU are almost certainly in the base model's
pre-training; MedMNIST-Pneumonia draws on the same paediatric CXR collection as the "viral pneumonia"
class of our COVID-19 Radiography training data, so its number may be optimistic; and on CinC-2016 heart sounds
no model beats the trivial baseline: answering "normal" to everything scores 0.800, this model
scores 0.617 with an AUROC of 0.542, and the base model and Jev-Omni score 0.200 because they answer
"abnormal" to almost every recording. Being better than the other two there is not the same as being
useful: the honest reading is that the heart-sound pathway does not transfer to this collection, and
it is the open item for the next run.
Further analyses
Post-hoc calibration
Fitted on the dev split: a global temperature (T = 1.036), with per-source temperature / yes-bias terms where they helped. Not shipped. The fit was validated on the single-prompt path (test NLL 0.6377 to 0.6284), but the released path averages over option rotations, which already smooths the probabilities: it measures an expected calibration error of 0.018 without any post-hoc term. Shipping a correction that was never validated on the path we release would make the tables describe a configuration nobody runs, so the loader returns the raw head probabilities and every number on this card is measured that way. The fit is kept with the run artifacts for the record.
Table with columns: before, after (fitted, not shipped) | before | after (fitted, not shipped) |
|---|
| dev NLL (fit set) | 0.6740 | 0.6738 |
| test NLL | 0.6377 | 0.6284 |
| test ECE | 0.0218 | 0.0096 |
| test mean-source NLL | 0.6127 | 0.6084 |
| test accuracy | 0.7375 | 0.7389 |
Selective prediction (does confidence rank errors?)
Accuracy when only the most confident share of test questions is answered; a model whose
confidence is informative gains accuracy as coverage falls.
15,629 test questions (text 5,847, image 8,835, audio 947).
Table with columns: group, acc @ 50 %, acc @ 80 %, acc @ 100 %| group | acc @ 50 % | acc @ 80 % | acc @ 100 % |
|---|
| all | 0.897 | 0.815 | 0.741 |
| text | 0.868 | 0.752 | 0.670 |
| image | 0.921 | 0.846 | 0.791 |
| audio | 0.829 | 0.748 | 0.711 |
Beyond accuracy (balanced accuracy and score questions)
On skewed questions accuracy can sit at the majority rate while the model still separates the
classes; balanced accuracy (the mean of per-class recall) shows that. For score questions the
interface returns the expected level E[level] = Σ i·p_i.
Table with columns: Task family, CareJev, base, Jev-Omni| Task family | CareJev | base | Jev-Omni |
|---|
| 12-lead ECG (rendered) (15) | 0.630 | 0.570 | 0.546 |
| Heart sounds (3) | 0.515 | 0.501 | 0.505 |
| Chest X-ray | 0.798 | 0.457 | 0.434 |
| Sleep staging (rendered PSG) | 0.651 | 0.225 | 0.255 |
Score questions, mean absolute error of the expected level: eicu_demo_los 1.40 levels (majority 1.77); mimic4_demo_los 2.31 levels (majority 2.38); support2_los 1.53 levels (majority 1.57).
Sub-groups (SUPPORT2, as recorded in the state)
2,739 SUPPORT2 test questions (in-hospital death, 2- and 6-month survival); groups with fewer than 30 questions are not shown.
Table with columns: group, accuracy, AUROC, ECE| group | accuracy | AUROC | ECE |
|---|
| sex: female | 0.758 | 0.841 | 0.019 |
| sex: male | 0.790 | 0.869 | 0.030 |
| age: <50 | 0.803 | 0.893 | 0.043 |
| age: 50-64 | 0.791 | 0.862 | 0.034 |
| age: 65-79 |
Latency of the released path
Table with columns: path, questions, ms / question, input tokens / s| path | questions | ms / question | input tokens / s |
|---|
| noul (Yes/No), one prompt | 40 | 307 | 925 |
| score, 10 levels, released path (8 rotations, one batch) | 40 | 2287 | 154 |
| score, 10 levels, single prompt (permutations=0) | 40 | 338 | 1042 |
| prefix cache (opt-in), 3-4 questions per state, single prompt each | 40 | 183 | - |
Measured on NVIDIA GB10.
Data
CareJev-Omni was trained only on publicly available, open-access medical datasets: rendered 12-lead
ECGs, chest X-rays, heart-sound recordings, sleep recordings, ICU outcome records, de-identified EHR
demo data, medical exam questions and clinical transcriptions. No credentialed data and no protected
health information were used, and evaluation uses a patient-disjoint held-out split.
Attribution: PhysioNet/CinC Challenge 2020, CirCor DigiScope, Sleep-EDF, eICU-CRD demo and
MIMIC-IV demo (PhysioNet); COVID-19 Radiography Database (CC BY 4.0); BMD-HS (CC BY 4.0); SUPPORT2
(Vanderbilt / UCI); MedMCQA (MIT); Medical Transcriptions (CC0). Each dataset's own licence and terms
continue to apply.
Quick start
pip install -r requirements.txt # torch, torchvision, transformers==5.17.0, peft, ...# audio input also needs ffmpeg on PATH
torchvision is required even for text-only use: transformers' Gemma 4 image processor imports it at
module load. This repo is self-contained: the frozen base weights, the adapter applied on top at load and the decision head. No network access is needed after download. The low-rank delta is deliberately not folded into the weights: merging it into bf16 leaves accuracy and NLL intact but costs about half of this model's calibration and flips 3.4 % of answers (measured, 300 test items), and calibration is the property this card is about. python example.py (shipped in this repo) prints one answer of each question type.
Every file's SHA-256 is in sha256.json; to check a download:
python -c "import json,hashlib,pathlib;[print(k, hashlib.sha256(pathlib.Path(k).read_bytes()).hexdigest()==v) for k,v in json.load(open('sha256.json')).items()]"
from huggingface_hub import snapshot_downloadimport sys; sys.path.insert(0, snapshot_download("nevermindai/carejev-omni", allow_patterns=["*.py"]))from carejev_omni_loader import load_carejev_omni # same contract as Jev-Omni's load_jev_omni clf = load_carejev_omni("nevermindai/carejev-omni")print(clf.predict( state="71-year-old male. Admission type: Emergency. ... DAY-3 PHYSIOLOGY: mean arterial pressure 55 mmHg; ...", question="Will this patient die before being discharged from this hospitalization?", options=["Yes", "No"],))# {'prediction': 'Yes', 'prediction_index': 0, 'confidence': 0.71, 'probabilities': {'Yes': 0.71, 'No': 0.29}}
The inference code ships in this repo (carejev_omni_loader.py and the carejev_omni/ package it
imports), so pip install -r requirements.txt is all a fresh environment needs.
How to use
Once loaded, clf.predict(...) answers one question about one patient state, and
clf.predict_many(...) answers several questions about the same state.
Inputs
Table with columns: argument, type, meaning| argument | type | meaning |
|---|
state | str | what is known about the patient, written as text (notes, values, context) |
question | str | the question to answer |
options | list[str] | the allowed answers: 2 to 256, all different |
|
Output: a plain, JSON-serialisable dict:
{ "prediction": "Yes", "prediction_index": 0, "confidence": 0.71, "probabilities": {"Yes": 0.71, "No": 0.29}}
probabilities holds every option, in the order you passed them, summing to 1. prediction is the
most probable option, prediction_index its position in your list and confidence its probability.
A threshold on confidence is how you decide which answers to use as they are and which to pass on
for review.
The same structure comes back for every question type (values below are illustrative). Choice:
{ "prediction": "Sepsis", "prediction_index": 0, "confidence": 0.82, "probabilities": {"Sepsis": 0.82, "Heart failure": 0.09, "Pulmonary embolism": 0.06, "Diabetic ketoacidosis": 0.03}}
Score (ordered levels): the most probable level, plus every level's probability:
{ "prediction": "8 to 14 days", "prediction_index": 8, "confidence": 0.34, "probabilities": {"Less than 1 day": 0.0, "1 day": 0.01, "2 days": 0.02, "3 days": 0.04, "4 days": 0.06, "5 days": 0.08, "6 days": 0.09, "7 days": 0.11, "8 to 14 days": 0.34, "More than 14 days": 0.25}}
predict_many: a list of the same dicts, one per question, in question order:
[ {"prediction": "No", "prediction_index": 1, "confidence": 0.77, "probabilities": {"Yes": 0.23, "No": 0.77}}, {"prediction": "Yes", "prediction_index": 0, "confidence": 0.58, "probabilities": {"Yes": 0.58, "No": 0.42}}]
Examples
state = "PATIENT: 68-year-old male. ... DAY-3 PHYSIOLOGY: mean arterial pressure 72 mmHg; ..." # yes / no: read the probability of "Yes"out = clf.predict(state=state, options=["Yes", "No"], question="Will this patient die before being discharged from this hospitalization?")p_yes = out["probabilities"]["Yes"] # choice: one of several optionsout = clf.predict(state=state, question="Which single problem best explains this presentation?", options=["Sepsis", "Heart failure", "Pulmonary embolism", "Diabetic ketoacidosis"])answer, confidence = out["prediction"], out["confidence"] # score: ordered levels; the expected level is the probability-weighted positionlevels = ["Less than 1 day", "1 day", "2 days", "3 days", "4 days", "5 days", "6 days", "7 days", "8 to 14 days", "More than 14 days"]out = clf.predict(state=state, question="How long will this patient's hospital stay be in total?", options=levels, qtype="score")expected_level = sum(i * out["probabilities"][o] for i, o in enumerate(levels)) # an image (here a chest X-ray) or a heart-sound recording (audio needs ffmpeg on PATH)clf.predict(state="Frontal chest radiograph.", question="What does this chest X-ray most likely show?", options=["COVID-19", "Normal", "Lung opacity", "Viral pneumonia"], media="cxr.png", modality="image")clf.predict(state="Phonocardiogram: digital stethoscope recording of heart sounds.", question="Is a heart murmur audible in this recording?", options=["Yes", "No"], media="pcg.wav", modality="audio") # several questions about the same state: a list of the same dicts, in question orderouts = clf.predict_many(state=state, questions=[ ("Will this patient die before being discharged from this hospitalization?", ["Yes", "No"]), ("Will this patient be alive 6 months (180 days) after study entry?", ["Yes", "No"]),])
Questions phrased like the training data work best: the examples above use that wording, and the
model card's task families show what it was trained on. predict_many(..., use_prefix_cache=True)
encodes the state once for all the questions; it is faster but scores each question with a single
prompt in your option order, like permutations=0. Every number on this card uses the default path.
Limitations
- Trained on open, mostly small datasets; SUPPORT2 (1989–94, five US hospitals) dominates the
text portion. Not validated on any live clinical population.
- Signal "images" are our own renderings; performance depends on that rendering.
- Heart-sound questions whose label age and sex alone predict as well as this model (
bmdhs_ar, bmdhs_as, bmdhs_mr, bmdhs_normal) are not reported: a per-source number there would not measure the recording. The heart-sound rows that are tabulated (bmdhs_ms, circor_murmur, circor_outcome) do not have that shortcut, and on the external CinC-2016 collection no model beats the trivial answer.
- Video is supported by the architecture but received no training data.
- eICU / MIMIC-IV demo sources are tiny (≤ 100 patients); their numbers are noisy.
- Calibration is reported on the internal test split only; distribution shift will degrade it.
- On the SUPPORT2
outcomes, a gradient-boosted model fitted on the raw numeric columns available at study entry is
level with this model on the three binary outcomes (AUROC within 0.01, accuracy within 0.01) and
clearly better on the length-of-stay score question, with slightly lower NLL on all four. Per-source
classical models also win on a few other sources (notably the tiny eICU demo tasks and CirCor
murmur). This model is for the case where the evidence is mixed — prose, rendered signals, sounds —
and one interface has to cover all of it, not for replacing a well-fitted tabular risk score.
Relation to other work
Independent open model implementing the typed-decision interface described by TypeSafe AI's Jev;
not affiliated with or trained on outputs of TypeSafe AI, Jev, Jev-Omni, or Google. Architecture
and prompt follow Jev-Omni for drop-in compatibility.
Evaluation protocol
- Splits are patient-disjoint and were fixed before any model saw the data.
- The test split is touched once per model, after training; selection uses dev only.
- Baselines share the exact prompt text and option lists (the base model's answer is read from its
next-token distribution over the option numbers).
- Published comparisons are to the untouched base model and to the other released model implementing
this interface. Non-neural baselines (TF-IDF / tabular / pixel / MFCC models fitted per source) were
measured and are kept with the run artifacts, but are not part of the published tables: they are
bespoke per task and per modality, so they do not answer the question this card is about, which is
what one general model does across every source. See Limitations for where such a model is the
better tool.
- Intervals: non-parametric bootstrap over test examples, 500 resamples, percentile 95 % CI.
- Calibration terms are fitted on dev and reported on test before/after.
- Every number on this card is rendered directly from the evaluation outputs of this checkpoint.
Licence
The base weights are Gemma 4, published by Google under
Apache-2.0 (licence). Our own contributions —
decision head, loader code and this card — are Apache-2.0 as well, so the whole release is Apache-2.0. Dataset licences are listed above and remain separate. Both licences disclaim warranties
and limit liability; nothing here grants any additional warranty or indemnity, and the research-use
notice at the top of this card applies to every part of the release. Card generated 2026-10-04 from run
artifacts.