Results
Measured by Saanora on public benchmarks the model never trained on. Rows for other models are each benchmark's own
published results (October 2026). Two sets of numbers for Clara, because they differ slightly:
- These weights (bf16, as published here), run with the System One server in this folder or in process;
- The hosted API (FP8 on the platform), which is what api.saanora.com serves.
Tool routing, AgentDecisionBench (3,939 agent
states: pick the next tool, finish, or ask the user). Official metric: the mean over each task's occurrences, then over
tasks.
Table with columns: Model, Accuracy| Model | Accuracy |
|---|
| AgentDecision-4B | 99.1% |
| StartLux-27B | 93.9% |
| OpenJev-27B | 92.4% |
| Clara, these weights (one-way) | 92.1% |
| Clara, hosted API (fast mode) | 91.9% |
| Jev 1.13 | 91.5% |
| OpenAI Decisions API | 90.9% |
Coherence, JevBench-mini (1,200 metamorphic tests; 95% intervals):
Table with columns: Model, Overall| Model | Overall |
|---|
| Open-Jev-9B (leaderboard #1) | 83.9 [82.5–85.2] |
| Clara, these weights, two-way | 83.6 [81–86] |
| Clara, hosted API, consistent mode (two-way) | 83.0 [80.6–85.5] |
| Clara, these weights, one-way | 81.8 [79–84] |
| Clara, hosted API, fast mode (one-way) | 81.3 [78.7–83.8] |
| JevAny-Gemma-4B (leaderboard #2) | 81.2 [79.0–83.2] |
Against its base (same harness for both):
Table with columns: Test, h2o-lightning-4b, Clara| Test | h2o-lightning-4b | Clara |
|---|
| Customer-service yes/no, held out of training (n = 856) | 90.4% | 94.9% |
| JevBench-mini, two-way | 83.1 [81–85] | 83.6 [81–86] |
| AgentDecisionBench, one-way, scored per distinct question | 87.4% | 93.6% |
Speed: on the hosted API, a record with three questions took 144 ms of model time (fast mode).
Format errors: 0%, by construction.
How to run it
The System One server (both reading modes)
This folder carries H2O's serving files unchanged (serve.sh, h2o_lightning_shim.py, pooled_readout.py, box.py,
assets.py, serve_config.json) and label rows recorded for these weights (label_rows.lightning). On one GPU with
24 GB or more:
pip install vllm==0.30.0hf download Saanora/clara-1 --local-dir clara-1bash clara-1/serve.sh # POST http://127.0.0.1:8741/v1/systemone
serve.sh serves two-way reading by default (bidirectional attention over the whole prompt, the more consistent
mode, what the hosted API calls consistent). TWO_WAY=0 bash clara-1/serve.sh serves one-way reading (ordinary
left-to-right, what the API calls fast). The server checks that label_rows.lightning belongs to the
model.safetensors beside it by its SHA-256, which is why the weights ship as one file.
A request:
POST /v1/systemone{"state": "Ticket 48213: our nightly export has failed three nights in a row. Finance needs the file by 9am.", "questions": { "failing": {"type": "noul", "instructions": "Is a scheduled export currently failing?"}, "team": {"type": "choice", "instructions": "Which team should handle this ticket?", "criteria": {"engineering": "Broken product functionality", "billing": "Invoices and refunds"}}, "severity": {"type": "score", "instructions": "How severe is the business impact?", "criteria": ["none", "low", "high", "critical"]}}}
examples/clara_readout.py builds the same prompt as the server and reads the same label tokens, using the server's
own functions, so its answers match the server's one-way path:
pip install "transformers>=5.14" torch numpypython clara-1/examples/clara_readout.py --model clara-1
The core of it:
prompt = render_chat(cfg["prompt"]["system"], user_turn(record, question, names, descs, labels), cfg["prompt"]["prefill"])logits = model(**tok(prompt, return_tensors="pt", add_special_tokens=False)).logits[0, -1]logprobs = group_logprobs({i: float(logits[i]) for g in groups for i in g[1]}, groups) # softmax over the asked labelsanswer = format_answer(qtype, names, descs, logprobs, cfg["temperature"][qtype], cfg["noul_floor"])
Two-way reading needs the server's pooling setup and is not shown in plain transformers.
On a laptop
GGUF builds for llama.cpp are in the companion repository clara-1-GGUF: Q8_0 (4.6 GB) and Q4_K_M (2.8 GB), one-way
reading only. On 14 test questions, Q8_0 moves probabilities by 0.002 on average and Q4_K_M by 0.013, and neither
changes an answer: Saanora/clara-1-GGUF.
Intended use
Routing, triage, moderation, risk flags and agent control: any decision about a record that is yes/no, one of a list,
or a point on a scale. Use the probabilities with thresholds you choose on your own data.
Validate before you automate. Benchmarks describe other people's records. Before Clara acts on its own, run it on a
few hundred of your own labelled records and pick a threshold for each question. Do not use it as the only decision-maker
for decisions with legal or similarly significant effects on people (credit, employment, housing, insurance, healthcare,
legal status) without human review.
Limitations
- Asking the user. On AgentDecisionBench states where the right move is to ask the user, Clara is right 59% of the
time, below its base (63.5%). It tends to pick a lookup tool when a detail is missing. Add a separate yes/no question
("is a required detail missing?") with a threshold and check it first.
- Score questions (a point on a scale) are the least reliable type. Prefer yes/no or choice where you can; for a
scale, sum adjacent levels' probabilities instead of reading the expected level alone.
- Probability coherence (a question and its negation summing to 1) is the weakest JevBench dimension, as for every
model on that board.
- Workflow final actions. On our 99-case business-workflow evaluation, per-question accuracy matches the base, but a
final action computed from five answers by a code rule is 3 points below the base (the interval includes zero).
- English only. A record longer than
max_state_tokens in serve_config.json (32,000 tokens) keeps its head and
tail around a "..." line; details in the middle are lost.
- Text only was trained. The vision tower is the base's, unchanged and untested by us.
- One seed pair. The release averages two training seeds; differences under 1–2 points are within noise.
Training
LoRA (rank 16, alpha 32, on the language model's layers only) on h2o-lightning-4b, trained with a soft cross-entropy
over the option-label logits. Two seeds were averaged and merged into the weights. Training data, all with recorded
provenance:
- ZefanCai/Open-Jev,
release-v2-redistributable (CC0-1.0
generated content; the customer-control-v1 and customer-service sources were excluded);
- procedurally generated IT-operations tool-routing states (our code);
- tool-routing plans and multi-question workflow records labelled by a self-hosted Qwen3.5-27B-FP8 (Apache-2.0
weights) on our own GPUs.
No outputs of Claude, Gemini or OpenAI models were used for training. AgentDecisionBench (CC BY 4.0) and JevBench
(Apache-2.0) were used for evaluation only.
License
Apache License 2.0, the same license as the base model, for research and commercial use. NOTICE
records the attribution to h2oai/h2o-lightning-4b and Qwen/Qwen3.5-4B, the files carried over unchanged from H2O, and
Saanora's changes.
If you'd rather not run it yourself, the same model is hosted on the Saanora Platform.
Citation
@misc{saanora2026clara, title = {Clara: a decision model with a probability for every option}, author = {Saanora}, year = {2026}, url = {https://api.saanora.com}}