GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
plantwaterer
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
plantwaterer
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
The engines are lenavg_bayes_engine.py, lenavg_engine.py and oavg_engine.py on top of the single-pass engine dikit_engine.py, with garas_vbll_engine.py, vbll_core.py,
fullft_mem.py, garas_engine.py and garas_core.py. They need the Decision Index kit:
bash
pip install "git+https://github.com/apolinario/decision-index@9eb2dbe2a358004c8782c66e40a83ac07b953fec"REPO=$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('plantwaterer/oenogarum'))")PYTHONPATH="$REPO" python -m decision_index run --engine lenavg_bayes_engine:LenAvgBayesEngine --option model="$REPO" --compact --out runs/oenogarumFor the single pass, use --engine dikit_engine:DikitEngine instead.
In Python:
python
import sysfrom huggingface_hub import snapshot_download repo = snapshot_download("plantwaterer/oenogarum")sys.path.insert(0, repo)from lenavg_bayes_engine import LenAvgBayesEngine # needs the decision_index kit installed engine = LenAvgBayesEngine(model=repo) # bf16 on "cuda"; the head runs on the CPUresponse, _ = engine( "The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", {"plan": {"type": "choice", "instructions": "What should the organisers do?", "criteria": {"play": "Play as planned", "indoors": "Move it indoors", "postpone": "Postpone it"}}, "rain": {"type": "noul", "instructions": "Is heavy rain likely?"}})print(response["answers"])The inference procedure is as follows:
Yes
and No; when the request gives their definitions (its criteria), they are shown as Yes: <definition> / No: <definition>.lenavg_bayes_engine:LenAvgBayesEngine):
len(). A string state counts its characters; a dictionary state
counts its top-level keys, not its text; a missing state counts 0. The state is counted once per question. A request is averaged
if and only if L <= 939; otherwise every question takes the single pass described above.numpy.random.default_rng(20261021 + 1000 * j + K).permutation(K) for j = 1, 2, 3 (K options). For each order the head's output
(the softmax of the letters' logits divided by T = 1.080) is mapped back to the options; the 4 vectors are averaged with equal
weight, then raised to the power 1 / 0.95438 and renormalised. (0.95438 here and 0.93738 below are fixed constants of the engine,
fitted when the averaging was built for allec and used unchanged here.)No first), averaged, then raised to the
power 1 / 0.93738 and renormalised (p). Then the binary Bayesian correction: the head's yes / no readout is computed from the base
prompt's hidden state and the log of the averaged native probabilities, once with its fitted residual and posterior variance and
once with both switched off (beta = 0, infl = 0); the difference of the two log-odds is added to the log-odds of p. No extra
temperature.Requirements: one GPU with at least 75 GB (the bf16 weights are 62 GB). Tested on 1 x H100 80GB and 1 x RTX PRO 6000 (96 GB) with torch 2.14.1 (CUDA 13.0), transformers 5.12.1, huggingface_hub, safetensors and numpy. bf16 results differ slightly between GPU types.
oenogarum is the uniform average (equal weights, in fp32) of three fine-tunes of google/gemma-4-31B-it that differ only in their random seed. Each was trained for 600 updates on the same 38,400 decision questions: a sample drawn in proportion to the composition of our full pool of about 276,000 decision questions, restricted to questions of at most 8,192 tokens. On 24,718 of the questions the training target mixes the label with the predicted distribution of a parent model, with weight 0.15; the parent is the base model google/gemma-4-31B-it for 17,334 questions and our earlier fine-tune garas for 7,384. The attention, renderer and optimiser settings are those of allec. The averaged model's head was then fitted anew on its own hidden states, on the same head fit and calibration data as allec's head.
Decision Index 0.3, public panel (37 benchmarks), with order averaging (lenavg_bayes_engine:LenAvgBayesEngine). The run used
this repository's weights, head and engines, with the kit's runner and scorer at commit 9eb2dbe2, in bf16. The full public suite ran as
8 row shards on single H100 80GB GPUs, merged without changes. This is our own run of the public kit, not a leaderboard entry. The run
files are at plantwaterer/oenogarum-decision-index. 94,198 of the 140,620
requests (67%) were averaged.
| public index | knowledge | language | retrieval | tools | arts | |
|---|---|---|---|---|---|---|
| board rule (the test item listed below counted as wrong) | 62.80 | 52.32 | 65.98 | 64.59 | 77.34 | 51.51 |
| as scored | 62.80 | 52.32 | 65.98 | 64.59 | 77.34 | 51.51 |
140,620 requests were run and 140,178 scored after the official exclusions; all of them were answered (0 unsupported, 0 errors). Nothing was truncated. The listed trained-on test item was answered correctly; counting it as wrong lowers the index by 0.0004.
Comparisons (pre-registered; paired bootstraps over the suite's test items with the board's own metrics, 95% intervals; they do not cover variation between training runs):
dikit_engine:DikitEngine), these weights and head score 62.37, against allec's single pass (61.60):
+0.76 points (+0.38 to +1.16).The averaging of three seeds was decided after seeing the first seed's single-pass score (62.04); the comparisons above were registered after that and before the other two seeds were trained. The first launch of the registered comparison stopped before computing anything (a file missing on the machine that ran it) and was relaunched unchanged.
Latency (our measurement; the board measures on its own private sample): the kit runner's per-request wall time for
lenavg_bayes_engine:LenAvgBayesEngine with these weights on 1 x RTX PRO 6000, one request at a time, on 1,000 requests of the public
suite (100 drawn from each tenth of the length L, so the sample includes the longest requests); all 1,000 were answered:
| median | mean | 80th percentile | 95th percentile |
|---|---|---|---|
| 227 ms | 609 ms | 566 ms | 2,094 ms |
These are sample statistics, not a per-request bound: the slowest request in the sample took 26.1 s. A second pass over the same sample gave the same figures within 1% (median 226 ms, mean 607 ms).
No test split of any benchmark was used. The head's fit and calibration data include train-split rows of CLINC150, GSM8K, Amazon ESCI, BANKING77 and WinoGrande; the training data include Lichess puzzles (a different dataset from ChessBench's).
An audit of the training text, the head's data included, against the Decision Index public suite found 1 test item that is an
exact or near-exact duplicate of a CLINC150 train-split utterance in the head's fit data: 5:CLINC150+OOS:CLINC150+OOS:test:1591.
It is counted as wrong in the board-rule number above, as the board does for train-split duplicates of test items (see the
Contamination section of the Decision Index methodology).
The weights and the head are released under Apache-2.0, and the base model, google/gemma-4-31B-it, is Apache-2.0. The training data come from the sources below, under the licences and terms listed. Some of these sources carry use conditions that apply to the data itself; they are repeated here: US Census PUMS (never use the data to identify any person or household), NVD (not endorsed or certified), California DMHC (modified; not official government data), MITRE CVE / CWE (copyright notices below) and XNLI (CC BY-NC 4.0, non-commercial; its translations are part of AmericasNLI).
oenogarum was trained on decision questions built from the public data sources below, plus synthetic questions generated by the author (no third-party data). Thanks to everyone who builds and maintains them. Records were reformatted, and many were modified (fields selected, text clipped, identifiers masked), so none of the material should be read as an official copy of its source.
20231101.en, by Wikipedia contributors.
Licensed CC BY-SA 3.0/4.0 and GFDL. https://en.wikipedia.org/| dataset | reference | licence |
|---|---|---|
| BANKING77 | Casanueva et al., 2020 | CC BY 4.0 |
| CLINC150 | Larson et al., 2019 | CC BY 3.0 |
| GoEmotions | Demszky et al., 2020 | Apache-2.0 |
| GSM8K | Cobbe et al., 2021 | MIT |
| HelpSteer2 | Wang et al., 2024 | CC BY 4.0 |
| Measuring Hate Speech | Kennedy et al., 2020 | CC BY 4.0 |
| IBM Debater ArgQ-Rank-30kArgs | Gretz et al., 2019 | CC BY 3.0 |
| BBQ | Parrish et al., 2022 | CC BY 4.0 |
| CaSiNo | Chawla et al., 2021 | CC BY 4.0 |
| Circa | Louis et al., 2020 | CC BY 4.0 (dataset card; its README links CC BY-SA 4.0) |
| CodeContests | Li et al., 2022 | CC BY 4.0 |
| CosmosQA | Huang et al., 2019 | CC BY 4.0 |
| CommonsenseQA | Talmor et al., 2019 | MIT |
| e-CARE | Du et al., 2022 | MIT |
| Social IQa | Sap et al., 2019 | CC BY 4.0 |
| WinoGrande | Sakaguchi et al., 2019 | CC BY (code Apache-2.0) |
| MedMCQA | Pal et al., 2022 | Apache-2.0 |
| PubMedQA | Jin et al., 2019 | MIT |
| Stanford Politeness Corpus, via ConvoKit | Danescu-Niculescu-Mizil et al., 2013 | CC BY 4.0 |
| Scruples | Lourie et al., 2021 | Apache-2.0 |
| TabFact | Chen et al., 2020 | CC BY 4.0 |
| QASPER | Dasigi et al., 2021 | CC BY 4.0 |
| CUAD | Hendrycks et al., 2021 | CC BY 4.0 |
| TyDi QA | Clark et al., 2020 | Apache-2.0 (the Wikipedia text: CC BY-SA) |
Through tasksource (Sileo, 2024), each dataset under its own licence as recorded on its dataset page:
pip install "git+https://github.com/apolinario/decision-index@9eb2dbe2a358004c8782c66e40a83ac07b953fec"REPO=$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('plantwaterer/oenogarum'))")PYTHONPATH="$REPO" python -m decision_index run --engine lenavg_bayes_engine:LenAvgBayesEngine --option model="$REPO" --compact --out runs/oenogarumimport sysfrom huggingface_hub import snapshot_download repo = snapshot_download("plantwaterer/oenogarum")sys.path.insert(0, repo)from lenavg_bayes_engine import LenAvgBayesEngine # needs the decision_index kit installed engine = LenAvgBayesEngine(model=repo) # bf16 on "cuda"; the head runs on the CPUresponse, _ = engine( "The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", {"plan": {"type": "choice", "instructions": "What should the organisers do?", "criteria": {"play": "Play as planned", "indoors": "Move it indoors", "postpone": "Postpone it"}}, "rain": {"type": "noul", "instructions": "Is heavy rain likely?"}})print(response["answers"])