GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
plantwaterer
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
plantwaterer
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
python
import stringimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base_id = "google/gemma-4-31B-it"tok = AutoTokenizer.from_pretrained(base_id)base = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16, device_map="cuda")# autocast_adapter_dtype=False keeps the LoRA update in bf16, the arithmetic the adapter was trained and evaluated withmodel = PeftModel.from_pretrained(base, "plantwaterer/garas", autocast_adapter_dtype=False).eval() LETTERS = string.ascii_uppercase + string.ascii_lowercase # up to 52 options def option_probabilities(state, question, options): opts = "\n".join(f"{LETTERS[i]}) {o}" for i, o in enumerate(options)) msg = (f"{state}\n\nQuestion: {question}\n\nOptions:\n{opts}\n\n" "Answer with the letter of the single best option and nothing else.") text = tok.apply_chat_template([{"role": "user", "content": msg}], tokenize=False, add_generation_prompt=True, enable_thinking=False) ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device) letter_ids = [tok(c, add_special_tokens=False)["input_ids"][0] for c in LETTERS[: len(options)]] with torch.no_grad(): logits = model(**ids).logits[0, -1] return torch.softmax(logits[letter_ids].float(), dim=-1).tolist() print(option_probabilities("The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", "What should the organisers do?", ["Play as planned", "Move it indoors", "Postpone it"]))Yes / no questions use the options Yes and No in that order. When a request gives a yes / no question's definitions (its
criteria, keyed yes / no or true / false), the engine and the model class show them with the options, as
Yes: <definition> / No: <definition>.
modeling_garas.py (with garas_core.py) defines GarasForDecision, a torch.nn.Module that wraps the base model and the
adapter. It loads the adapter with autocast_adapter_dtype=False, so the LoRA update stays in bf16, the arithmetic the adapter
was trained and evaluated with. predict_proba gives the same answers as the Decision Index engine below.
python
import torchfrom modeling_garas import GarasForDecision model = GarasForDecision.from_pretrained("plantwaterer/garas", base="google/gemma-4-31B-it", device_map="cuda") # a request: a state plus named questions; each answer has a probability for every optionanswers = model.predict_proba( "The forecast gives a 70% chance of heavy rain this afternoon. The match is outdoors, on grass.", {"plan": {"type": "choice", "instructions": "What should the organisers do?", "criteria": {"play": "Play as planned", "indoors": "Move it indoors", "postpone": "Postpone it"}}, "rain": {"type": "noul", "instructions": "Is heavy rain likely?"}})print(answers["plan"]["probabilities"], answers["rain"]["noul"]) # the same prompts, batched and run through forward()enc = [model.encode("The match is outdoors.", "What should the organisers do?", {"play": "Play", "postpone": "Postpone"}, key="plan"), model.encode("The invoice total is $1,240; the purchase order says $1,420.", "Is the invoice consistent?", kind="noul", key="check")]batch = model.collate(enc) # right-padded input_ids / attention_mask, option_token_ids / option_maskwith torch.no_grad(): out = model(**batch) # option_logits, log_probs, probs: [batch, options] in letter orderfor e, p in zip(enc, out.probs): print({e.option_keys[o]: float(p[j]) for j, o in enumerate(e.order)}) # letter j shows option e.order[j]forward(..., labels=...) also returns the cross-entropy over the options, for a gold letter index per row (-100 to skip).
torch.compile works (use dynamic=True), but compiled kernels round bf16 differently, so use eager mode to reproduce the
reported numbers exactly.
garas_engine.py (with garas_core.py) runs the adapter through the
Decision Index kit's runner:
bash
python -m decision_index run --engine garas_engine:GarasEngine \ --option base=google/gemma-4-31B-it --option adapter=plantwaterer/garas \ --option max_len=32768 --option tokens_per_batch=32768 --compact --out runs/garasThe inference procedure is as follows:
Yes: <definition> / No: <definition>).Dependencies are torch, transformers, peft and numpy. The engine was tested with transformers 5.12.1 and peft 0.21.2.
Decision Index 0.3, public panel (37 benchmarks). The run used this repository's engine and adapter, with the kit's runner at
commit 9eb2dbe2, on 1 x GPU per shard. The full public suite ran as 8 row shards, merged without changes, followed by
one resume pass of the kit runner. This is our own run of the public kit, not a leaderboard entry. It used the earlier engine
(garas_engine.py md5 4af46b6e), which did not show yes / no definitions.
| public index | |
|---|---|
| board rule (the 19 test items listed below counted as wrong) | 56.57 |
| as scored | 56.59 |
Latency was measured on 1 x RTX PRO 6000, one request at a time, on our own 760-row sample of the public suite (750 timed rows). This is an approximation of the board's latency check, which uses a private sample:
| median | mean | 80th percentile |
|---|---|---|
| 114 ms | 509 ms | 364 ms |
Training used the train splits of BANKING77, CLINC150 and GSM8K. It also used WinoGrande train, Amazon ESCI train queries, and Lichess puzzles (a different dataset from ChessBench's). No test split of any benchmark was used.
An audit of the training text against the Decision Index public suite found 19 test items that are exact or near-exact
duplicates of utterances in the BANKING77 and CLINC150 train splits. They are counted as wrong in the board-rule number above, as
the board does for train-split duplicates of test items (see the Jet v6.2 and lev entries under Contamination in the
Decision Index methodology):
4:BANKING77:BANKING77:test: 554, 677, 727, 742, 747, 751, 754, 1432, 1735, 1754, 1993, 2081, 2211, 2440, 2644, 2651, 2987,
3070; 5:CLINC150+OOS:CLINC150+OOS:test:1591.
The same audit found no overlapping items for the other sources named above.
The adapter is released under Apache-2.0, and the base model, google/gemma-4-31B-it, is Apache-2.0. The training data come from the sources below, under the licences and terms listed. Some of these sources carry use conditions that apply to the data itself; they are repeated here: CourtListener (no Fair Credit Reporting Act uses), US Census PUMS (never use the data to identify any person or household), NVD / US Department of Labor (not endorsed or certified), California DMHC (modified; not official government data) and MITRE CVE / CWE (copyright notices below).
garas was trained on decision questions built from the public data sources below, plus synthetic questions generated by the author (no third-party data). Thanks to everyone who builds and maintains them. Records were reformatted, and many were modified (fields selected, text clipped, identifiers masked), so none of the material should be read as an official copy of its source.
20231101.en, by Wikipedia contributors.
Licensed CC BY-SA 3.0/4.0 and GFDL. https://en.wikipedia.org/| dataset | reference | licence |
|---|---|---|
| BANKING77 | Casanueva et al., 2020 | CC BY 4.0 |
| CLINC150 | Larson et al., 2019 | CC BY 3.0 |
| GoEmotions | Demszky et al., 2020 | Apache-2.0 |
| GSM8K | Cobbe et al., 2021 | MIT |
| HelpSteer2 | Wang et al., 2024 | CC BY 4.0 |
| Measuring Hate Speech | Kennedy et al., 2020 |
garas_engine.py, modeling_garas.py). The adapter and garas_core.py are unchanged.| CC BY 4.0 |
| IBM Debater ArgQ-Rank-30kArgs | Gretz et al., 2019 | CC BY 3.0 |
| BBQ | Parrish et al., 2022 | CC BY 4.0 |
| CaSiNo | Chawla et al., 2021 | CC BY 4.0 |
| Circa | Louis et al., 2020 | CC BY 4.0 (dataset card; its README links CC BY-SA 4.0) |
| CodeContests | Li et al., 2022 | CC BY 4.0 |
| CosmosQA | Huang et al., 2019 | CC BY 4.0 |
| CommonsenseQA | Talmor et al., 2019 | MIT |
| e-CARE | Du et al., 2022 | MIT |
| Social IQa | Sap et al., 2019 | CC BY 4.0 |
| WinoGrande | Sakaguchi et al., 2019 | CC BY (code Apache-2.0) |
| MedMCQA | Pal et al., 2022 | Apache-2.0 |
| PubMedQA | Jin et al., 2019 | MIT |
| Stanford Politeness Corpus, via ConvoKit | Danescu-Niculescu-Mizil et al., 2013 | CC BY 4.0 |
| Scruples | Lourie et al., 2021 | Apache-2.0 |
| TabFact | Chen et al., 2020 | CC BY 4.0 |