onatm

gev-e2b

README

License: apache-2.0

Use

With access to the Gev source, run uv sync --locked in its checkout (add --extra mlx on Apple Silicon). Save this as request.json:

json

{"state":"Order #1 arrived damaged.","questions":{"route":{"type":"choice","instructions":"Which team handles this?","criteria":{"billing":"Payments","support":"Product issues"}}}}

bash

uv run gev predict onatm/gev-e2b --input request.json

The response includes questions.route.answer, questions.route.probabilities (one per option), and the serving temperature. Choice, yes/no (noul), and ordinal (score) questions are supported. A Hugging Face text-generation or PEFT-only pipeline cannot serve this model.

Evaluation

Clean-question scores from the saved reports. Accuracy is at raw T=1 (temperature scaling does not change the winning answer); Brier and ECE are lower-is-better. A dash means calibrated metrics were not recorded for that split.

The served columns use the saved temperature T=1.6245.

Table with columns: Suite / split, Clean n, Accuracy, Brier raw, Brier served, ECE raw, ECE served
Suite / split	Clean n	Accuracy	Brier raw	Brier served	ECE raw	ECE served
decision-v7/development	1,264	0.7975	0.2847	-	0.0628	-
transfer-v4/development	656	0.6113	0.4990	0.4735	0.1163	0.0522
decision-v7/test	1,200	0.8292	0.2457	0.2336	0.0661	0.0170
transfer-v4/test	656

decision-v7 contains held-out questions from the training source families; transfer-v4 contains new sources and held-out policy structures. Development was for model selection; these reports describe seed 0. Temperature was fitted on the separate decision-v7/calibration split, not on either test split.

The decision-v7 test report predates the temperature fit. Its served metrics were computed afterward from saved raw logits, without rerunning inference or fitting on test.

Known limits

This is one seed, not a multi-seed study. The new-source test is substantially harder than the trained-source test; evaluate on your own decisions before use.

For changed-answer contrastive pairs, both answers were correct in 12/64 pairs.
With a none-of-the-above option present, 14/36 questions were correct.

Model and provenance

Base: google/gemma-4-E2B at revision d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f; frozen base weights are not in this repo.
Architecture: LoRA rank 16, alpha 32 on the text decoder and a 256-wide pointer head. Each question is scored independently.
Recipe: seed 0, 2 epochs, 12,576 training records, 3,144 steps, MLX/BF16. The checkpoint stores the fitted temperature in gev.json.
Training data SHA-256: 7ed5254b5cb5291baefaceb09edf7e13110258211518c8038f4a12c11bd628ad; each saved evaluation report also records its split's SHA-256.
Training suite: jaredpalmer/kev-suites (decision-v7, pinned and verified by hash).

Training uses option permutation, none-of-the-above and distractor augmentation, and contrastive pairs. The architecture follows Jev's Architecture Unmasked; the data and evaluation protocol are adapted from Kev. The detailed run reports and code are in the private Gev repository.

License

The adapter and pointer head are apache-2.0; check the separately loaded base model and dataset licenses as well.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

onatm

Model Tree

Base

google/gemma-4-E2B

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality