sthanika-ai

Sieve-31B-v3

README

License: apache-2.0

Ranked #1 (tied) on the Decision Index

Sieve-31B-v3 is ranked #1 (tied) on the Decision Index leaderboard, with a Full score of 71.10 (edition 0.4.1, board data of 2026-10-28, 243 models). The board ranks models by the Full score, which comes from the maintainers' private tests, and gives one rank to the models it counts as tied.

The Decision Index leaderboard on 2026-10-28.

Summary

Table

Backbone	google/gemma-4-31B-it (revision `842da37`)
Design	letter readout: the model's own output-layer logits of the option labels
Trained weights	full fine-tune of every linear projection of the 60 decoder layers (29.29B parameters); the token embeddings (tied with the output layer) and every norm are the backbone's
Attention	bidirectional in the 10 global (full-attention) layers; causal in the 50 sliding-window layers
Question types	`choice`, `noul` and `score` (trained on `choice` and `noul`)
Decision Index	ranked #1 (tied) on the leaderboard, with a Full score of 71.10 (edition 0.4.1); public score 71.68 on our own run of the 0.3 suite
Temperature	a global T = 1.27 and 22 per question category (0.955 to 1.380)
Token limits	none declared; trained on prompts of up to 16,384 tokens
GPU memory	58.3 GiB of bf16 weights; the serving benchmark's peak was 81.4 GiB allocated: one 96 GB GPU runs every public request

How it works

Prompt. Gemma's own chat template with thinking off: a system turn, then one user turn with the list of questions and the state, then one block per question with its labelled options, ending in an Answer: ( slot.
Score. At each slot, the model's output-layer logits of the question's option labels (A, B, …, then AA, AB, …, up to 255 options; final logit softcap included). A softmax at the question category's temperature gives the probabilities.
Two option orders. Every choice question whose options may be reordered is read twice, with its options as given and rotated by half; the two log-probability vectors are averaged. This cancels the preference for a label position. Questions marked "permute": false keep their order and are read once.
Bidirectional attention. See the next section.
Weights. A full Gemma-4 checkpoint in bf16 (13 safetensors shards), plus head.pt with the readout, the attention rule, the serving policy and the temperatures. The weights are a weighted average of the backbone (0.3) and the fine-tuned model (0.7).
Never. It never truncates, drops or reorders options, tunes prompts per benchmark, or sees gold labels.

Table with columns: type, options, returns
type	options	returns
`choice`	the caller's keys, optionally described (2 to 255)	the most likely key, a confidence, a probability per key
`noul`	yes / no	P(yes)
`score`	an ordered list of 2 to 10 levels	the expected level, a confidence, a probability per level

Bidirectional attention

Gemma-4-31B-it interleaves 10 global (full-attention) layers with 50 sliding-window layers (1,024-token window). A text-generation model has to be causal: each token only sees the tokens before it. A decision model generates nothing, so the global layers can read in both directions.

One question. The whole prompt is one block: in the global layers every token attends to every other token, before and after it, so the state is read in the light of the question and its options.
Several questions. The shared block (system turn, the list of questions, the state) attends only to itself. Each question's block attends to the shared block and to itself. Questions never see each other's options.
Sliding-window layers. Causal in every case, as in the base model.
Trained and served the same way. The fine-tune was trained with this attention, and sieve serves it the same way. head.pt records the rule, and sieve sets the attention when it loads the model.
Shared state. In a request with several questions and a shared block of 2,048 tokens or more, sieve computes the shared block once and reuses its keys and values for every question (and both option orders). The result is the same as computing it with each question.

Use

bash

pip install -r https://raw.githubusercontent.com/sthanika-ai/Sieve/main/requirements.txt   # the tested versions (torch 2.8.0)pip install "sieve-decisions[serve] @ git+https://github.com/sthanika-ai/Sieve"

python

from sieve import load_sieve, decide m = load_sieve("sthanika-ai/Sieve-31B-v3")decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},       {"department": {"type": "choice", "instructions": "Which team handles this?",                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

Loading. Load it with sieve 0.4.0 or later, which reads head.pt and applies the prompt, the attention rule, the two option orders and the temperatures. Loading the weights with transformers alone gives the language model without any of these.
GPU memory. The bf16 weights take 58.3 GiB. The serving benchmark observed a peak of 81.4 GiB allocated over 352 public Decision Index requests, including the largest by the kit's size estimate (17 questions; with this model's tokenizer, its shared block alone is 56,693 tokens and the prompt of its longest question 69,019). One 96 GB GPU runs every public request; shorter requests fit on an 80 GB GPU.
Token limits. None declared. Requests are never truncated; one that does not fit in GPU memory fails with an error instead of a wrong answer. The model was trained on prompts of up to 16,384 tokens, and every public Decision Index request, the longest about 54,000 tokens, was answered.
Server. sieve-serve --model sthanika-ai/Sieve-31B-v3.

Decision Index.

text

python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-31B-v3 --out runs/Sieve-31B-v3

Temperature per category. head.pt stores a global T = 1.27 and 22 temperatures (0.955 to 1.380), fitted by NLL on the calibration split (17,753 questions) under the two-order reading. A question's category is its type, its number of options (2, 3-4, 5-9, 10-25, 26+) and the kind of state (empty, text or JSON). A question takes the temperature of its most specific category in the table (choice:3-4:text, then choice:3-4, then choice), else the global T. Temperatures never change an answer, and load_sieve(..., temperature=1.0) gives raw probabilities.

Results

Decision Index

Sieve-31B-v3 is ranked #1 (tied) on the Decision Index leaderboard (edition 0.4.1, board data of 2026-10-28), with a Full score of 71.10 and a public score of 71.68. On our own run of the public 0.3 suite with the official kit at f5a6e4f (0.3) and its scorer, it scores 71.68 (raw index 78.39, breadth 71.18).

Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
How the run was made. One complete run of the 0.3 suite on eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, one shard per GPU.
Results. sthanika-ai/Sieve-31B-v3-decision-index-results.
Leaderboard. The board ranks by the Full score, which comes from the maintainers' private tests: 60% tests of the same skills as the public benchmarks and 40% decision tasks from new domains. The public score is not part of it.

Table with columns: Decision Index 0.3, public suite, Knowledge & Reasoning, Language Understanding, Retrieval & Classification, Tools & Automation, Arts & Human Taste
Decision Index 0.3, public suite	Knowledge & Reasoning	Language Understanding	Retrieval & Classification	Tools & Automation	Arts & Human Taste
71.68	61.2	71.6	76.1	86.9	62.3

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected.

Table with columns: benchmark, metric, score, skill (×100), requests
benchmark	metric	score	skill (×100)	requests
ACOS	per-review F1	0.501	48.5	1,565
Amazon ESCI	macro-F1	0.644	55.4	5,000
ANLI	macro-F1	0.809	71.4	3,200
API-Bank	accuracy	0.888	88.6	508
BANKING77	macro-F1	0.913	91.2	3,080
BBH fixed-option tasks	accuracy	0.896	84.9	5,507
BFCL	case exact accuracy	0.979	97.1	1,694
BPoMP	accuracy	0.992	98.4	5,000
BRIGHT	nDCG@10	0.506	44.1	220
cfcolor	accuracy	0.665	32.0	5,000
ChessBench	accuracy	0.255	18.9	5,000
CLadder	accuracy	0.923	84.6	5,000
CLINC150+OOS	macro-F1	0.968	96.8	5,500
ContractNLI	macro-F1	0.798	70.8	123
CRUXEval	accuracy	0.932	89.1	570
FinEntity	macro-F1	0.942	91.4	979
GPQA Diamond	accuracy	0.546	39.5	196
GSM8K	accuracy	0.910	89.3	2,638
Habermas Machine	accuracy	0.622	45.1	1,676
HellaSwag	accuracy	0.975	96.7	10,042
HLE	accuracy	0.130	0.0	501
Home appliance simulator	case exact accuracy	0.966	96.6	88
HoVer claim verification	accuracy	0.891	78.2	4,000
Humicroedit	accuracy	0.676	35.2	2,628
iSarcasmEval	Sarcasm F1 · track A, English	0.683	59.2	4,600
MMLU-Pro	accuracy	0.758	72.8	12,032
MuSR	accuracy	0.809	69.5	752
New Yorker caption matching	accuracy	0.873	84.1	528
NLI4CT	macro-F1	0.842	69.2	5,500
PhishNChips phishing decisions	accuracy	0.949	89.8	2,000
POP909-CL	accuracy	0.792	78.7	2,000
RAGTruth response-level hallucination	F1 on hallucinated class	0.821	63.0	2,700
SATA-Bench	case exact accuracy	0.394	38.6	1,650
ToolRet	nDCG@10	0.705	65.9	685
VAST	macro-F1	0.797	69.5	3,006
When2Call MCQ	accuracy	0.880	84.0	3,652
WinoGrande	accuracy	0.949	89.7	1,267
ARC-Challenge (not in the index)	accuracy	0.978	97.0	1,172
ARC-Easy (not in the index)	accuracy	0.993	99.1	2,376
MMLU (not in the index)	accuracy	0.897	86.2	14,033
RouterBench (not in the index)	selected quality (quality objective)	0.798	52.6	10,000
SGD/SGD-X (not in the index)	macro-F1	0.366	0.0	2,500
SimpleBench (not in the index)	accuracy	0.500	40.0	10

Calibration

The temperatures were fitted on the calibration split (17,753 questions), which was not used for training or for choosing the weights.

Table with columns: split, questions, NLL, ECE
split	questions	NLL	ECE
calibration, per-category temperatures (shipped)	17,753	0.2212	0.0083
calibration, one global T = 1.27	17,753	0.2216	0.0072

Latency

Per Decision Index request. In-process wall time of one request, CUDA-synchronized, eager, one request at a time, over the 140,620 requests of the 0.3 run, with one engine per NVIDIA RTX PRO 6000 Blackwell Server Edition: median 219 ms, mean 913 ms, 80th percentile 657 ms, 95th percentile 1,507 ms. This is not the maintainers' measurement. The leaderboard measures latency separately, also on one NVIDIA RTX PRO 6000, and lists a median of 232 ms, a mean of 663 ms and a 95th percentile of 1,520 ms.

Training

Recipe.
- Full fine-tune of every linear projection of the 60 decoder layers (29,286,727,680 parameters); the token embeddings (tied with the output layer) and every norm stay frozen. The 10 global attention layers were bidirectional during training, as in serving.
- Loss: cross-entropy on the option-label logits + 0.3 × the multi-class Brier score.
- AdamW, lr 2e-6, betas (0.9, 0.95), weight decay 0.01 on weight matrices, gradient clipping at 1.0, 5% linear warm-up then cosine decay to 0.1×, 1 epoch, 128 records per step.
- bf16 compute with fp32 master weights, gradient reduction and optimizer state (AdamW moments in CPU memory); FSDP2 with one unit per decoder layer; gradient checkpointing.
- A record's questions share one pass over its state under the attention rule above, in packs of up to 16,384 tokens. Longer records are dropped, never truncated (none were).
- Permuted choice options, "none of the above" augmentation (as the answer, as a wrong option and in minimal pairs) and irrelevant distractor options.
- 5,544 steps in two stages on two NVIDIA B300 GPUs: steps 1–500 (4.4 h), a check of the step-500 weights, then steps 501–5,544 (44.13 h) resumed with the optimizer state; about 97 GPU-hours.
Release weights. Weight snapshots were kept at steps 500, 1,386, 2,772, 4,158 and 5,544. The released weights are 0.3 × the backbone + 0.7 × the step-5,544 weights, chosen among the snapshots, their average and their blends with the backbone by a selection rule fixed before training. The two-order reading was chosen by the same rule. The Decision Index benchmarks were never used to choose.
Held-out monitor. On a fixed 6,000-question held-out sample, the fine-tuned weights reached accuracy 0.8925 and NLL 0.2797 at step 5,544.
Data. 709,604 training records (957,775 questions), with 12,395 records for calibration and 12,902 held out.
- Sources. Public datasets and generated decision records.
- Not published. The training files are not part of this release; the SHA-256 of each file is in training_config.json.

Limitations

score questions. The training data has only choice and noul questions. score questions are answered by the same readout with the levels as options, but the model was not trained on them.
More than 255 options. The readout has 255 option labels; a choice question with more options is refused.

Files

Table with columns: file, contents
file	contents
`model-*.safetensors`, `model.safetensors.index.json`, `config.json`, `generation_config.json`	the full model in bf16 (13 shards)
`head.pt`	the readout (labels and prompt format), the attention rule, the serving policy and the calibrated temperatures (global and per category)
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`	Gemma-4-31B-it's tokenizer and chat template, unchanged
`training_config.json`, `training_metrics.json`, `train.log`	the recipe, the run and its log
`result.json`	the calibration, Decision Index and latency numbers above
`provenance.json`	SHA-256 of every file and of the training data, and library versions
`assets/decision-index-leaderboard-2026-10-28.png`	the Decision Index leaderboard on 2026-10-28 (screenshot)

License

The weights are released under Apache-2.0 (see LICENSE), as is the code at github.com/sthanika-ai/Sieve. Use of the base model is governed by Gemma's terms, including Google's prohibited-use policy.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

sthanika-ai

Model Tree

Base

google/gemma-4-31B-it

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality