GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
plantwaterer
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
plantwaterer
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
The repository includes a Decision Index engine, dikit_engine.py, with
garas_vbll_engine.py, vbll_core.py, fullft_mem.py, garas_engine.py and garas_core.py. It needs the Decision Index kit:
bash
pip install "git+https://github.com/apolinario/decision-index@9eb2dbe2a358004c8782c66e40a83ac07b953fec"REPO=$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('plantwaterer/allec'))")PYTHONPATH="$REPO" python -m decision_index run --engine dikit_engine:DikitEngine --option model="$REPO" --compact --out runs/allecIn Python:
python
import sysfrom huggingface_hub import snapshot_download repo = snapshot_download("plantwaterer/allec")sys.path.insert(0, repo)from dikit_engine import DikitEngine # needs the decision_index kit installed engine = DikitEngine(model=repo) # bf16 on "cuda"; the head runs on the CPUresponse, _ = engine( "The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", {"plan": {"type": "choice", "instructions": "What should the organisers do?", "criteria": {"play": "Play as planned", "indoors": "Move it indoors", "postpone": "Postpone it"}}, "rain": {"type": "noul", "instructions": "Is heavy rain likely?"}})print(response["answers"])The inference procedure is as follows:
Yes
and No; when the request gives their definitions (its criteria), they are shown as Yes: <definition> / No: <definition>.Requirements: one GPU with at least 75 GB (the bf16 weights are 62 GB). Tested on 1 x H100 80GB and 1 x RTX PRO 6000 (96 GB) with torch 2.14.1 (CUDA 13.0), transformers 5.12.1, huggingface_hub, safetensors and numpy. bf16 results differ slightly between GPU types.
Decision Index 0.3, public panel (37 benchmarks). The run used this repository's weights, head and engine, with the kit's runner and scorer at commit 9eb2dbe2, in bf16. The full public suite ran as 8 row shards on single GPUs (shards 0, 1, 6 and 7 on H100 80GB, 2 to 5 on RTX PRO 6000), merged without changes. This is our own run of the public kit, not a leaderboard entry. The run files are at plantwaterer/allec-decision-index.
In that run, the engine's no-truncation check had a bug. Gemma 4's chat template trims the user message, so a state that starts with
whitespace is shown without it, and the check, which looked for the untrimmed state in the prompt, reported every such request as too
long. That hit 309 requests (MMLU-Pro 304, iSarcasmEval 5; the longest of their states is 172 tokens). The engine in this repository
has the fix (garas_engine.state_shown). The 309 requests were run again with it in one resume pass of the kit runner (1 x H100 80GB);
the other 140,311 result lines are unchanged, and their prompts are the same under both versions of the check.
| public index | knowledge | language | retrieval | tools | arts | |
|---|---|---|---|---|---|---|
| board rule (the test item listed below counted as wrong) | 61.60 | 49.82 | 64.77 | 64.85 | 77.44 | 48.41 |
| as scored | 61.60 | 49.82 | 64.77 | 64.85 | 77.44 | 48.41 |
140,620 requests were run and 140,178 scored after the official exclusions; all of them were answered (0 unsupported, 0 errors). Nothing was truncated. The listed trained-on test item was already answered wrong, so the two rows are equal.
Latency (our approximation of the board's check, which uses a private sample): the kit runner's per-request wall time on the four RTX PRO 6000 shards of the run above, one request at a time, 70,161 answered requests (before the fix; the fix does not change the work per request):
| median | mean | 80th percentile |
|---|---|---|
| 102 ms | 484 ms | 363 ms |
No test split of any benchmark was used. The head's fit and calibration data include train-split rows of CLINC150, GSM8K, Amazon ESCI, BANKING77 and WinoGrande; the training data include Lichess puzzles (a different dataset from ChessBench's).
An audit of the training text, the head's data included, against the Decision Index public suite found 1 test item that is an
exact or near-exact duplicate of a CLINC150 train-split utterance in the head's fit data: 5:CLINC150+OOS:CLINC150+OOS:test:1591.
It is counted as wrong in the board-rule number above, as the board does for train-split duplicates of test items (see the
Contamination section of the Decision Index methodology).
The weights and the head are released under Apache-2.0, and the base model, google/gemma-4-31B-it, is Apache-2.0. The training data come from the sources below, under the licences and terms listed. Some of these sources carry use conditions that apply to the data itself; they are repeated here: US Census PUMS (never use the data to identify any person or household), NVD (not endorsed or certified), California DMHC (modified; not official government data), MITRE CVE / CWE (copyright notices below) and XNLI (CC BY-NC 4.0, non-commercial; its translations are part of AmericasNLI).
allec was trained on decision questions built from the public data sources below, plus synthetic questions generated by the author (no third-party data). Thanks to everyone who builds and maintains them. Records were reformatted, and many were modified (fields selected, text clipped, identifiers masked), so none of the material should be read as an official copy of its source.
20231101.en, by Wikipedia contributors.
Licensed CC BY-SA 3.0/4.0 and GFDL. https://en.wikipedia.org/| dataset | reference | licence |
|---|---|---|
| BANKING77 | Casanueva et al., 2020 | CC BY 4.0 |
| CLINC150 | Larson et al., 2019 | CC BY 3.0 |
| GoEmotions | Demszky et al., 2020 | Apache-2.0 |
| GSM8K | Cobbe et al., 2021 | MIT |
| HelpSteer2 | Wang et al., 2024 | CC BY 4.0 |
| Measuring Hate Speech | Kennedy et al., 2020 | CC BY 4.0 |
| IBM Debater ArgQ-Rank-30kArgs | Gretz et al., 2019 | CC BY 3.0 |
| BBQ | Parrish et al., 2022 | CC BY 4.0 |
| CaSiNo | Chawla et al., 2021 | CC BY 4.0 |
| Circa | Louis et al., 2020 | CC BY 4.0 (dataset card; its README links CC BY-SA 4.0) |
| CodeContests | Li et al., 2022 | CC BY 4.0 |
| CosmosQA | Huang et al., 2019 | CC BY 4.0 |
| CommonsenseQA | Talmor et al., 2019 | MIT |
| e-CARE | Du et al., 2022 | MIT |
| Social IQa | Sap et al., 2019 | CC BY 4.0 |
| WinoGrande | Sakaguchi et al., 2019 | CC BY (code Apache-2.0) |
| MedMCQA | Pal et al., 2022 | Apache-2.0 |
| PubMedQA | Jin et al., 2019 | MIT |
| Stanford Politeness Corpus, via ConvoKit | Danescu-Niculescu-Mizil et al., 2013 | CC BY 4.0 |
| Scruples | Lourie et al., 2021 | Apache-2.0 |
| TabFact | Chen et al., 2020 | CC BY 4.0 |
| QASPER | Dasigi et al., 2021 | CC BY 4.0 |
Through tasksource (Sileo, 2024), each dataset under its own licence as recorded on its dataset page:
pip install "git+https://github.com/apolinario/decision-index@9eb2dbe2a358004c8782c66e40a83ac07b953fec"REPO=$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('plantwaterer/allec'))")PYTHONPATH="$REPO" python -m decision_index run --engine dikit_engine:DikitEngine --option model="$REPO" --compact --out runs/allecimport sysfrom huggingface_hub import snapshot_download repo = snapshot_download("plantwaterer/allec")sys.path.insert(0, repo)from dikit_engine import DikitEngine # needs the decision_index kit installed engine = DikitEngine(model=repo) # bf16 on "cuda"; the head runs on the CPUresponse, _ = engine( "The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", {"plan": {"type": "choice", "instructions": "What should the organisers do?", "criteria": {"play": "Play as planned", "indoors": "Move it indoors", "postpone": "Postpone it"}}, "rain": {"type": "noul", "instructions": "Is heavy rain likely?"}})print(response["answers"])