At a glance
Table with columns: AUBIN, reference | AUBIN | reference |
|---|
| Typed decisions (Laya's benchmark) | 77.55 · #1 | meraGPT 76.8 · Laya 76.65 · Jev 72.7 |
| Jev's own published split | 8 of 8 metrics | accuracy · Brier · ECE · NLL, both splits |
| Kev suites, sources Kev never saw | 89.8 | Kev-27B 89.6 |
| Kev suites, Kev's own training sources | 87.08 · NLL 0.43 | Kev-9B 87.4 · Kev-27B 87.0 · Kev-4B 86.5 |
| Visual computer use (ScreenSpot) | 69.3 (E4B-Screen) · 66.7 (12B base, fp32) | SeeClick 53.4 · CogAgent 47.4 · UI-TARS-7B 89.5 |
| Web agent (Mind2Web, cross-domain step success) | 45.0 (31B) · 43.5 (12B) | MindAct-XL 39.6 · GPT-4 26.4 |
| Real-time control (100 unseen episodes) | 92% · 0 lava deaths | best rule + same shield 89% |
| FPS play (ViZDoom), learning while it plays | 17.3 kills / episode | same model without learning 15.6 · scripted rule 18.8 (30 episodes; gain not yet significant) |
All rows are measured; protocols, sample sizes and every variant tried are in
reports/AUBIN_RESULTS_2026-10-03.md.
Where AUBIN is behind (Kev-9B on Kev's training sources; the strongest dedicated GUI-grounding models), it says so.
The family
Table with columns: model, ability, key result| model | ability | key result |
|---|
| AUBIN-Omni | one model, every ability: decide · click · web · act · learn | every ability in E4B, 12B and 31B |
| AUBIN-31B | typed decisions, strongest | Kev unseen sources 89.8 |
| AUBIN-12B | typed decisions | typed-decisions 77.55 (#1) |
| AUBIN-E4B-v3 |
Quick start
pip install "git+https://huggingface.co/emrevrg/AUBIN-12B#subdirectory=code"
from aubin import Aubin, AubinOmni, AubinEngine ai = Aubin("emrevrg/AUBIN-12B")ai.decide("Billed twice for March, refund or we cancel.", {"team": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "payments", "tech": "bugs"}}}) omni = AubinOmni.from_hub("emrevrg/AUBIN-Omni", size="E4B") # one base model, every ability (lazy-loaded)omni.click(screenshot, "open settings") # -> {"x": ..., "y": ...} on a 0-1000 scale
aubin serve --model emrevrg/AUBIN-12B --learn memory.npz # HTTP · OpenAI-compatible · Laya/Jev-compatible /v1/systemone · MCP
More: INTEGRATION.md (LangChain, LlamaIndex, CrewAI, TypeScript, Docker, AUBIN Engine).
AUBIN-12B — AUBIN by Norovox
At a glance
- Beats Jev on Jev's own published split, all 8 numbers: accuracy 0.863 vs 0.857 and 0.862 vs 0.845; NLL 0.409 vs 0.701; Brier, ECE lower too.
- AUBIN Trio (adds Phi-4), 8/8 vs Jev: accuracy 0.872 vs 0.857 (new sources) and 0.851 vs 0.845 (trained sources); ECE 0.018 / 0.032 vs 0.049 / 0.067.
- AUBIN Quad (adds Phi-4 + Mistral-Small; chosen by the rule), 8/8 vs Jev: accuracy 0.869 vs 0.857 (new sources) and 0.856 vs 0.845 (trained sources); ECE 0.030 / 0.022 vs 0.049 / 0.067.
- 22x fewer confidently-wrong answers (AUBIN-31B, one pass, trained-source split: 0.24% vs Jev 5.22%).
- 0 errors on held-out policy rules it never trained on (Jev: 14 / 176).
- Locked test, never-trained sources: 0.898, the best in Kev's table (Kev-27B 0.896).
- Computer use (Mind2Web), AUBIN-31B with no Mind2Web training beats GPT-4 on 9 of 9 numbers (element acc / op F1 / step success): Cross-Task 48.0 / 81.0 / 40.0 vs 41.6 / 60.6 / 36.2; Cross-Website 41.0 / 82.0 / 33.5 vs 35.8 / 51.1 / 30.1; Cross-Domain 54.0 / 86.5 / 48.5 vs 37.1 / 46.5 / 26.4. GPT-4 in the paper chose from 10 candidates, AUBIN from 50.
- Also beats the fine-tuned MindAct (Flan-T5-XL, trained on Mind2Web) on all three numbers: Cross-Domain (54.0 / 86.5 / 48.5 vs 42.1 / 66.5 / 39.6).
- Real-time control (command-following game, 100 unseen episodes, success rate): AUBIN-E4B-Control 95% with the safety shield (0 lava deaths, 385 ms/move on a free T4), 58% with no shield at all; AUBIN-12B-Control 92% with the safety shield (0 lava deaths, 856 ms/move on a free T4), 74% with no shield at all. Rule baselines: greedy 58%, greedy + same shield 89%.
Typed questions in, calibrated probabilities out. AUBIN is Norovox's open-base model family: strong open-weight
bases, adapted and calibrated for structured decisions (choice / yes-no / score). Fast path = one forward pass;
when the model is unsure (top-2 log-prob gap < 4), it writes a short rationale and re-scores
(think-when-uncertain, 26-32% of questions). Open weights, runs locally/offline in 4-bit on a single free 16 GB GPU (T4).
Head-to-head with Jev — on Jev's own published split
Jev's published numbers (jaredpalmer/kev runs/jev-*-v4/report.json, "clean")
are on the development split of Kev's public suites, variant == clean (656 new-source + 1264 trained-source
questions). AUBIN is scored on exactly the same questions (code/jev_compare.py). Bold = better than Jev.
Table with columns: model, new sources acc, Brier ↓, ECE ↓, NLL ↓, trained sources acc, Brier ↓, ECE ↓, NLL ↓| model | new sources acc | Brier ↓ | ECE ↓ | NLL ↓ | trained sources acc | Brier ↓ | ECE ↓ | NLL ↓ |
|---|
| Jev (hosted, published report) | 0.857 | 0.2110 | 0.049 | 0.701 | 0.845 | 0.2367 | 0.067 | 0.776 |
| AUBIN-12B full precision (think) | 0.857 |
AUBIN Duo-SC (31B self-consistent think + 12B full precision) is better than Jev on all 8 numbers above (accuracy, Brier, ECE, NLL on both splits), on the exact questions Jev's own report uses.
"New sources" = transfer-v4 (MMLU, SciQ, QNLI, PAWS, Emotion, TweetEval, held-out policy rules) — never used to train
AUBIN. "Trained sources" = decision-v7. AUBIN's training used only decision-v7/train.
Table with columns: source, n, AUBIN Duo, Jev| source | n | AUBIN Duo | Jev |
|---|
| composition_held_and_or | 32 | 1.000 | 0.906 |
| composition_held_conditional | 32 | 1.000 | 0.781 |
| composition_held_or_not | 32 | 1.000 | 0.969 |
| contrastive_authorization | 40 | 1.000 | 1.000 |
AUBIN Duo ahead on 15 sources, tied on 15, behind on 5 (of 35).
Where AUBIN wins by multiples (same questions)
The costly failure of a decision model is being confidently wrong or misapplying a policy rule (refund windows, authorization, deadlines, nested conditions).
Table with columns: model, split, wrong while ≥90% confident ↓ (Jev → AUBIN), fewer, rule/policy errors ↓ (Jev → AUBIN), fewer| model | split | wrong while ≥90% confident ↓ (Jev → AUBIN) | fewer | rule/policy errors ↓ (Jev → AUBIN) | fewer |
|---|
| AUBIN-12B full precision (think) | new sources | 3.66% → 1.68% | 2.2× | 14 → 1 / 176 | 14.0× |
| AUBIN-12B full precision (think) | trained sources | 5.22% → 1.34% | 3.9× | 13 → 0 / 224 | ∞ (zero) |
|
"Wrong while ≥90% confident" = share of all questions answered wrongly with confidence ≥ 0.9 (Jev: confident_error_rate from its report). AUBIN is more cautious — it is ≥90% sure on fewer questions — but when it is, it is right more often, and its overall calibration (ECE) is better. Rule/policy = composition_* + contrastive_* sources.
Against the Kev family (Kev README, development / test, clean)
On the locked test split of sources AUBIN never trained on, AUBIN-31B (fast, one pass) is the most accurate model in the table (0.898 vs Kev-27B 0.896). On Kev's own trained sources the Kev models remain ahead.
Table with columns: model, new sources acc (dev / test), trained sources acc (dev / test), new sources Brier (dev / test)| model | new sources acc (dev / test) | trained sources acc (dev / test) | new sources Brier (dev / test) |
|---|
| AUBIN-12B full precision (think) | 0.857 / 0.878 | 0.851 / 0.835 | 0.233 / 0.199 |
| AUBIN Duo-SC (31B self-consistent think + 12B full precision) | 0.863 / 0.895 | 0.862 / 0.843 | 0.210 / 0.168 |
| AUBIN Trio (31B-SC + 12B full precision + Phi-4) | 0.872 / 0.889 | 0.851 / 0.851 | 0.209 / 0.169 |
| AUBIN Quad (31B-SC + 12B full precision + Phi-4 + Mistral-Small) |
Computer use: Mind2Web web-agent benchmark
Each step: task + previous actions + the official Mind2Web ranker's top candidates → AUBIN picks the element, the operation (click / type / select) and the value. Metrics as in the Mind2Web paper (Deng et al., 2023): element accuracy, operation F1, step success rate. Reference rows are the paper's Table 2; GPT-4 there chose from the top 10 candidates on a 50-task subset, AUBIN from the top 50.
Table with columns: split, model, element acc, op F1, step SR| split | model | element acc | op F1 | step SR |
|---|
| Cross-Task | AUBIN-31B, zero-shot (paper protocol: 5 + None) (200 steps) | 48.0 | 81.0 | 40.0 |
| Cross-Task | AUBIN-12B + Mind2Web fine-tune (paper protocol: 5 + None) (300 steps) | 40.7 | 82.1 | 36.3 |
| Cross-Task | AUBIN-12B, zero-shot (paper protocol: 5 + None) (300 steps) | 35.3 | 74.3 |
Game: AUBIN plays 2048 (live replay)
AUBIN-12B with a 2-step lookahead tool: the tool plays clear moves, AUBIN decides close calls (34% of moves). Same 10 seeds for every player.
Table with columns: player, mean score, min, max, reached 2048| player | mean score | min | max | reached 2048 |
|---|
| AUBIN + lookahead tool | 29,009 | 7,016 | 37,132 | 8/10 |
| lookahead tool alone | 30,076 | 12,268 | 47,436 | 8/10 |
| greedy merge rule | 2,982 | 1,464 | 3,676 | 0/10 |
Real-time control: command-following grid game
Each move: a 7x7 grid as text (walls, 5 deadly lava cells, 4 objects that block movement) and a command ("go to the red key"); the controller returns one move with a calibrated confidence. Same 100 episodes (seed 0) for every row, never seen in training (training episodes use seeds 1000+). Step accuracy = the move is on a shortest safe path (BFS ground truth). The safety shield only removes moves into visible lava/walls (AubinController.act(..., allowed=...)).
AUBIN-12B-Control rows are measured with code/control_bench.py on a Colab T4; its adapter upload follows in the next update.
Table with columns: controller, success, lava deaths ↓, step accuracy, median ms / move (T4)| controller | success | lava deaths ↓ | step accuracy | median ms / move (T4) |
|---|
| random | 6% | 40 | 35.3% | – |
| random + safety shield | 13% | 0 | 53.4% | – |
| greedy (ignores obstacles) | 58% | 26 | 56.8% | – |
Consistency and hallucination ("gray area" test types)
The test types follow an independent public review of Jev (Onur Tirpan, YouTube, Sep 2026: same question repeated, option order reversed, question rephrased, leading question, X vs not-X, false premise, invented quote, false alarm). AUBIN-12B was run on our own synthetic student records and invoices (Turkish + English, 40 documents); Jev's numbers are the ones shown in that review on its own Turkish items, so this is not a same-item comparison, and the first set of documents is short and simple. The hard version adds distractor fields, synonym labels, look-alike numbers (IBAN/fax), a blank signature line and a long note to every document.
Table with columns: test, AUBIN-12B, 40 docs, AUBIN-12B, 40 hard docs, Jev (review, TR items)| test | AUBIN-12B, 40 docs | AUBIN-12B, 40 hard docs | Jev (review, TR items) |
|---|
| decision changed when asked again | 0/120 | 0/120 | 1/60 |
| decision changed when options reversed | 0/120 | 1/120 | 11/60 |
| decision changed when question rephrased | 0/120 | 6/120 | 5/10 |
| followed a leading question | 0/40 | |
On the hard documents AUBIN-12B still contradicted itself on "X" vs "not X" in 4/65 pairs (Jev's review shows 0/40 on its own items); this is the open item for the next training round.
AUBIN Loop: generator + decider (aubin loop)
A generator writes candidate answers; AUBIN picks among them with calibrated probabilities; if confidence is below the target, the generator is asked again with the rejected candidates as feedback. On Gemma it is one set of weights: the generator is the same base with the AUBIN adapter switched off. Any OpenAI-compatible model (ChatGPT, vLLM, Ollama) can be the generator instead (--generator openai:<model>, key from the environment).
GSM8K test, 400 fixed problems, generator = Gemma-4-12B base, same candidates for every method:
Table with columns: method, accuracy, samples / problem| method | accuracy | samples / problem |
|---|
| generator alone (greedy) | 0.843 | 1 |
| majority vote of 5 samples | 0.858 | 5 |
| AUBIN picks among 5 samples | 0.900 | 5 |
| AUBIN × vote share | 0.905 | 5 |
| AUBIN Loop (2nd round only when unsure) | 0.925 | 6.46 |
| oracle: any of 5 samples correct (upper bound) | 0.917 |
Measured speed (free Kaggle GPUs, 4-bit, batch = 1 case)
Table with columns: setup, fast path, think-when-uncertain| setup | fast path | think-when-uncertain |
|---|
| AUBIN-12B, 1x T4 | 486 ms / question (runs/speed/speed12.json) | ~6-7 s extra per thought question (batched 8), 26-32% of questions |
| AUBIN-31B, 2x T4 | 1.3-2.0 s / question | ~13-15 s extra per thought question (batched 8), ~14% of questions |
| Jev (hosted API, Kev report) | ~220 ms median | — |
T4 is a 2018-era free GPU; AUBIN latency on modern datacenter GPUs has not been measured yet and is not claimed.
Use
pip install "git+https://huggingface.co/emrevrg/AUBIN-12B#subdirectory=code"from aubin import Aubin, AubinEnsemblem = Aubin("emrevrg/AUBIN-12B") # base + LoRA + fitted temperature + think-when-uncertainm.decide({"ticket": "Package arrived broken, customer wants money back"}, {"route": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "refunds", "shipping": "delivery", "tech": "software"}}, "urgent": {"type": "noul", "instructions": "Is this urgent?"}})# AUBIN Duo-SC (headline row; 12B in full precision, e.g. 2x T4):duo = AubinEnsemble([(Aubin("emrevrg/AUBIN-31B", think_margin=4, think_mix=0.5, think_samples=3), 0.6), (Aubin("emrevrg/AUBIN-12B", four_bit=False, device_map="auto", temperature=3.2, think_margin=4, think_mix=0.5), 0.4)], temperature=0.8)# AUBIN Loop: generator + decider on the same Gemma weights (or any OpenAI-compatible generator)from aubin import AubinLoop, GemmaGeneratorfrom aubin.loop import last_numberloop = AubinLoop(m, GemmaGenerator(m), extract=last_number, target=0.8, mode="aubin+vote")loop.solve("A shop sells 3 pens for $4. How much do 12 pens cost?")
CLI: aubin decide case.json --think 4 · aubin loop "task" --numeric · server: aubin serve --port 8009 (POST /decide).
Honest notes
- Every number above is produced by the scripts in
code/ from per-question outputs; nothing is hand-edited.
- Every ensemble row (Duo, Duo-SC, Trio, Quad) uses one fixed selection rule: member weights and final temperature minimize
Brier + ECE on Kev's calibration split (decision-v7 calibration, 300 questions;
code/select_cal.py). The development
and test splits are never used for selection; think mix is fixed at 0.5. The test columns are single confirmation runs.
The new-source Brier margin is small: if the rule were calibration NLL instead, Duo-SC would be behind Jev on new-source
Brier and ECE (6/8), while accuracy and NLL stay ahead on both splits.
- Which members to combine was our choice among about a dozen candidate combinations of the models listed here
(including 12B variants trained with distillation and data weighting); only weights and temperature come from the rule.
- AUBIN Duo-SC: the 31B member thinks with 3 samples (greedy + 2 sampled, answer probabilities averaged;
think_samples=3).
- AUBIN Trio adds microsoft/phi-4 (MIT, no adapter, temperature fitted on the calibration split) as a third, different-family member.
- AUBIN Quad adds mistralai/Mistral-Small-24B-Instruct-2501 (Apache-2.0, no adapter, calibration-fitted temperature 5.5) as a fourth
member; its trained-source test run hit the free-GPU session limit, so that test cell is not reported ("—").
- AUBIN-12B full precision = the same adapter on the unquantized base (fp16 on 2x T4, or bf16 on one T4 through IRIS);
its temperature (3.2) is fitted on the calibration split like every other model here.
AUBIN-Learn — instant self-learning (experimental, Norovox core)
AUBIN can learn from feedback without retraining. Verified cases go into an external decision memory (a write takes well under
a millisecond with the hashing embedder), and a self-calibrator (Hedge / multiplicative weights) shifts trust per source between
the model, the memory and their fusion — so where memory is not useful yet, AUBIN keeps trusting itself. Measured on Kev's public
suites; full tables, protocol and code: reports/AUBIN_LEARN.md, code/aubin/learn.py.
- Feedback stream, kev_test: all 6 AUBIN variants improve, +1.0 to +1.3 points (best 84.3 → 85.6). Separate protocol — the
label is revealed after each answer — so it is not comparable to static scores (Kev-9B 87.4 static).
- Never-seen sources (transfer, memory starts empty): −0.4 to +0.1 points — it does not hurt; a few hundred feedbacks per source
are not enough to help yet.
- Fast skills, static locked test: per-source classifiers learned from memory in seconds (switched on only where dev proves
them) lift kev_test for all 4 measured AUBIN variants, +0.3 to +0.6 points (ensemble 84.3 → 84.8; banking77 59.5 → 65.5).
- Raw memory (kNN) on the static test: no reliable gain (−1.0 to +0.7) — AUBIN already learned these sources.
Table with columns: benchmark, AUBIN, reference| benchmark | AUBIN | reference |
|---|
| Typed decisions (2,000 decisions) | 77.55 (AUBIN-Learn, weights fixed before test) | Laya 76.65 · meraGPT 76.8 · Jev 72.7 |
| Kev suites, Kev's training sources (kev_test) | 85.7 (AUBIN ensemble, selected on cal split) | Kev-0.8B 83.8 · Kev-4B 86.5 · Kev-9B 87.4 |
| Kev suites, transfer test | 86.5 ensemble · 89.0 AUBIN-31B | – |
| Mind2Web cross-domain step SR (200 steps) | 48.5 AUBIN-31B, no web training | MindAct-XL 39.6 · GPT-4 26.4 |
| Grid control, 100 unseen episodes | (12B-Control + shield) |
On Kev's own training sources AUBIN is still 1.7 points behind Kev-9B; this is stated, not hidden.
Evening update (3 Oct, all measured, details in the results report):
Table with columns: benchmark, AUBIN, reference| benchmark | AUBIN | reference |
|---|
| ScreenSpot click accuracy (visual computer use) | 69.3 AUBIN-E4B-Screen (zero-shot 48.5; 67.7 after round 1) | SeeClick 53.4 · CogAgent 47.4 · UGround-7B 73.3 · UI-TARS-7B 89.5 |
| Mind2Web step success, cross-task / website / domain | 47.0 / 37.0 / 43.5 (12B) · 47.0 / 34.5 / 42.0 (E4B, ≈2× faster) | MindAct-XL 52.0 / 38.9 / 39.6 |
| ViZDoom FPS, kills per episode (30 episodes) | 17.3 with in-game self-learning (AUBIN-Learn) · 15.6 without (AUBIN-12B, 0.6 s/move) | random 1.3 · scripted rule 18.8 |
| Kev training sources + learned skills (selected on kev_dev + cal only) | 87.08, NLL 0.43 (ensemble 85.69, NLL 0.79) | Kev-9B 87.4 (not yet beaten) · Kev-27B 87.0 · Kev-4B 86.5 |
New in code/: AubinLearning.acquire_skill (learns a skill, self-tests on held-out data, enables only on proven gain), learn_skill2/3.py, fuse_multi.py, screenspot_train/eval.py, fps_vizdoom.py.
The AUBIN family: every ability, one interface (AubinEngine routes between them)
Table with columns: ability, model, measured| ability | model | measured |
|---|
| typed decisions (choice / score / yes-no), calibrated | AUBIN-31B · AUBIN-12B · 12B-v3b · 12B-v3d · E4B-v3 | typed-decisions 77.55 (#1) · Jev's set 8/8 · Kev unseen sources 89.8 · Kev training sources 87.08 |
| web agent | AUBIN-12B-Web · AUBIN-E4B-Web (fast) · 31B web training running |
♥ Help AUBIN get seen
On Hugging Face, likes decide what people discover. Big labs have marketing teams; AUBIN has one 17-year-old student and measured results.
If AUBIN is useful, interesting or just impressive to you, press ♥ Like at the top of this page and on AUBIN-Omni, AUBIN-31B and AUBIN-12B, then share it with one person who builds with AI.
Every like helps an independent, open, honestly measured model get discovered.
🇹🇷 Beğenin, AUBIN'in görünür olmasını sağlar: sayfanın üstündeki ♥ Like'a basarak destek ol ve bir arkadaşına gönder.
Support Norovox
Built by a 17-year-old high school student: no sponsor, no budget, just free GPUs and AI subscriptions paid for with difficulty.
Support goes into GPU compute, training, and the AI development tools this work depends on (such as Claude); supporters are credited and get early access.
zgremre@gmail.com · emrevrgdev@gmail.com · Why and how →
🇹🇷 17 yaşında bir lise öğrencisinin eseri: destekçisiz, bütçesiz; ücretsiz GPU'lar ve zorlukla ödenen yapay zekâ abonelikleriyle. Desteğin GPU'ya, eğitime ve bu işin dayandığı yapay zekâ geliştirme araçlarına (Claude gibi) gider.
Built with Claude Opus 5.5 and GPT-5.6 Sol; because of OpenAI usage limits, the final stretch was completed with Claude Opus 5.5.
License: Apache-2.0 (adapters and code). Base models: Google Gemma 4 (Apache-2.0).