GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
plantwaterer
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
plantwaterer
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
The repository includes a Decision Index engine, dikit_engine.py, with
garas_vbll_engine.py, vbll_core.py, fullft_mem.py, garas_engine.py and garas_core.py. It needs the Decision Index kit:
bash
pip install "git+https://github.com/apolinario/decision-index@9eb2dbe2a358004c8782c66e40a83ac07b953fec"REPO=$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('plantwaterer/muria'))")PYTHONPATH="$REPO" python -m decision_index run --engine dikit_engine:DikitEngine --option model="$REPO" --compact --out runs/muriaIn Python:
python
import sysfrom huggingface_hub import snapshot_download repo = snapshot_download("plantwaterer/muria")sys.path.insert(0, repo)from dikit_engine import DikitEngine # needs the decision_index kit installed engine = DikitEngine(model=repo) # bf16 on "cuda"; the head runs on the CPUresponse, _ = engine( "The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", {"plan": {"type": "choice", "instructions": "What should the organisers do?", "criteria": {"play": "Play as planned", "indoors": "Move it indoors", "postpone": "Postpone it"}}, "rain": {"type": "noul", "instructions": "Is heavy rain likely?"}})print(response["answers"])The inference procedure is as follows:
Yes
and No; when the request gives their definitions (its criteria), they are shown as Yes: <definition> / No: <definition>.Requirements: one GPU (the bf16 weights are 24 GB, and the packed rows add activations for up to 32,768 tokens). Tested on 1 x H100 80GB and 1 x RTX PRO 6000 (96 GB) with torch 2.14.1 (CUDA 13.0), transformers 5.12.1, huggingface_hub, safetensors and numpy. bf16 results differ slightly between GPU types.
Decision Index 0.3, public panel (37 benchmarks). The run used this repository's weights, head and engine, with the kit's runner and scorer at commit 9eb2dbe2, in bf16. The full public suite ran as 8 row shards on single GPUs (shards 0, 1, 2 and 7 on H100 80GB, 3 to 6 on RTX PRO 6000), merged without changes. This is our own run of the public kit, not a leaderboard entry. The run files are at plantwaterer/muria-decision-index.
| public index | knowledge | language | retrieval | tools | arts | |
|---|---|---|---|---|---|---|
| board rule (the 3 test items listed below counted as wrong) | 51.40 | 31.41 | 56.62 | 58.90 | 70.82 | 39.06 |
| as scored | 51.40 | 31.41 | 56.62 | 58.90 | 70.82 | 39.06 |
140,620 requests were run and 140,178 scored after the official exclusions; all of them were answered (0 unsupported, 0 errors). Nothing was truncated. The two rows are equal at two decimals: one of the 3 listed items was answered correctly, and counting it as wrong moves ANLI's score from 0.7171 to 0.7168.
Latency (our approximation of the board's check, which uses a private sample): the kit runner's per-request wall time on the four RTX PRO 6000 shards of the run above, one request at a time, 70,309 answered requests:
| median | mean | 80th percentile |
|---|---|---|
| 48 ms | 209 ms | 169 ms |
No test split of any benchmark was used to build the training data. The head's fit and calibration data include train-split rows of CLINC150, GSM8K, Amazon ESCI, BANKING77 and WinoGrande, and units from DocNLI's training split, some of which contain premise-hypothesis pairs from ANLI (see the note at the end of this card); the training data include Lichess puzzles (a different dataset from ChessBench's).
An audit of the training text, the head's data included, against the Decision Index public suite found 3 ANLI test items that are
near-duplicates of ANLI pairs inside DocNLI units in the head's fit data: 12:ANLI:ANLI:test_r2:5460dddd-0280-47bd-9148-1422b6729a4d,
12:ANLI:ANLI:test_r2:f6a3b2f4-863e-41af-bde1-762da7822ebd and 12:ANLI:ANLI:test_r3:d469aabc-dd2a-4bd7-b840-ab39764b6625. They are
counted as wrong in the board-rule number above, as the board does for test items found in a model's training data (see the
Contamination section of the Decision Index methodology).
The weights and the head are released under Apache-2.0, and the base model, google/gemma-4-12B-it, is Apache-2.0. The training data come from the sources below, under the licences and terms listed. Some of these sources carry use conditions that apply to the data itself; they are repeated here: US Census PUMS (never use the data to identify any person or household), NVD (not endorsed or certified), California DMHC (modified; not official government data), MITRE CVE / CWE (copyright notices below), XNLI (CC BY-NC 4.0, non-commercial; its translations are part of AmericasNLI), and, in the head's fit and calibration data only, ANLI (CC BY-NC 4.0, non-commercial; its premise-hypothesis pairs are part of DocNLI) and wikiHow (CC BY-NC-SA 3.0, non-commercial; via goal-step-wikihow). The note at the end of this card gives the counts.
muria was trained on decision questions built from the public data sources below, plus synthetic questions generated by the author (no third-party data). Thanks to everyone who builds and maintains them. Records were reformatted, and many were modified (fields selected, text clipped, identifiers masked), so none of the material should be read as an official copy of its source.
20231101.en, by Wikipedia contributors.
Licensed CC BY-SA 3.0/4.0 and GFDL. https://en.wikipedia.org/| dataset | reference | licence |
|---|---|---|
| BANKING77 | Casanueva et al., 2020 | CC BY 4.0 |
| CLINC150 | Larson et al., 2019 | CC BY 3.0 |
| GoEmotions | Demszky et al., 2020 | Apache-2.0 |
| GSM8K | Cobbe et al., 2021 | MIT |
| HelpSteer2 | Wang et al., 2024 | CC BY 4.0 |
| Measuring Hate Speech | Kennedy et al., 2020 | CC BY 4.0 |
| IBM Debater ArgQ-Rank-30kArgs | Gretz et al., 2019 | CC BY 3.0 |
| BBQ | Parrish et al., 2022 | CC BY 4.0 |
| CaSiNo | Chawla et al., 2021 | CC BY 4.0 |
| Circa | Louis et al., 2020 | CC BY 4.0 (dataset card; its README links CC BY-SA 4.0) |
| CodeContests | Li et al., 2022 | CC BY 4.0 |
| CosmosQA | Huang et al., 2019 | CC BY 4.0 |
| CommonsenseQA | Talmor et al., 2019 | MIT |
| e-CARE | Du et al., 2022 | MIT |
| Social IQa | Sap et al., 2019 | CC BY 4.0 |
| WinoGrande | Sakaguchi et al., 2019 | CC BY (code Apache-2.0) |
| MedMCQA | Pal et al., 2022 | Apache-2.0 |
| PubMedQA | Jin et al., 2019 | MIT |
| Stanford Politeness Corpus, via ConvoKit | Danescu-Niculescu-Mizil et al., 2013 | CC BY 4.0 |
| Scruples | Lourie et al., 2021 | Apache-2.0 |
| TabFact | Chen et al., 2020 | CC BY 4.0 |
| QASPER | Dasigi et al., 2021 | CC BY 4.0 |
Through tasksource (Sileo, 2024), each dataset under its own licence as recorded on its dataset page:
The head's fit data (16,384 decision questions) and calibration data (4,086) include two sources whose text is non-commercial. The model's fine-tuning data contain neither.
No dataset text is redistributed.
pip install "git+https://github.com/apolinario/decision-index@9eb2dbe2a358004c8782c66e40a83ac07b953fec"REPO=$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('plantwaterer/muria'))")PYTHONPATH="$REPO" python -m decision_index run --engine dikit_engine:DikitEngine --option model="$REPO" --compact --out runs/muriaimport sysfrom huggingface_hub import snapshot_download repo = snapshot_download("plantwaterer/muria")sys.path.insert(0, repo)from dikit_engine import DikitEngine # needs the decision_index kit installed engine = DikitEngine(model=repo) # bf16 on "cuda"; the head runs on the CPUresponse, _ = engine( "The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", {"plan": {"type": "choice", "instructions": "What should the organisers do?", "criteria": {"play": "Play as planned", "indoors": "Move it indoors", "postpone": "Postpone it"}}, "rain": {"type": "noul", "instructions": "Is heavy rain likely?"}})print(response["answers"])