sthanika-ai

Sieve-31B

README

License: apache-2.0

How it reads

Prompt: the decider-ai chat layout, used unmodified. Each question is one user turn in Gemma's own chat template, with thinking off, followed by an Answer: ( slot. Options keep the order the request gives them.
Score: at the slot, the final hidden state is multiplied by the output-layer rows of the option label tokens (A, B, …, up to 255 options). A softmax over the question's own options at the fixed temperature T = 8.5495 gives the probabilities.
Adapter: applied unmerged with PEFT, exactly as in training and in the submitted run.

This is the readout of the Decider chat entry, so the gain over 57.79 comes from the adapter.

Run it on the Decision Index

pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128pip install transformers==5.17.0 peft==0.21.0 accelerate==1.15.0 huggingface_hub==1.32.0 "numpy<2" jinja2 httpxpip install --no-deps decider-ai==1.6.0pip install --no-deps "decision-index @ git+https://github.com/apolinario/decision-index@62d2f51"hf download sthanika-ai/Sieve-31B --local-dir Sieve-31BPYTHONPATH=Sieve-31B python -m decision_index pipeline --engine sieve31b.engine:Sieve31BEngine --out runs/Sieve-31B

Defaults: the engine needs no options. It loads the base model at its pinned revision and the adapter from the folder that holds sieve31b/. The temperature is fixed at 8.5495.
Memory: bf16. The GPU held about 83 GB during our run, including PyTorch's cache, so it fits one RTX PRO 6000 (96 GB).
Smaller GPUs: --option split='{"0": "40GiB", "1": "40GiB"}' spreads the model over two GPUs.

Engine behaviour

Never: it never truncates, drops or reorders options, tunes prompts per benchmark, or sees gold labels.
Declared limits: up to 255 options per question and 32,768 tokens per prompt row (state plus one question). Larger requests are refused as Unsupported. No public request reached either limit.
Out of memory: the forward chunk is halved and the request retried. A single row that still does not fit raises an ordinary error, so a resumed run retries it.

Latency

Measured with this engine on 1× RTX PRO 6000 Blackwell Server Edition: single process, eager, one request at a time, over 400 random public Decision Index 0.3 requests. Median 163 ms, mean 644 ms, 80th percentile 675 ms.

Training

Table

Method	LoRA r = 128, α = 128, dropout 0, on `q/k/v/o/gate/up/down_proj` of all 60 text layers (979M trainable parameters); the base stays frozen in bf16
Objective	Soft-label cross-entropy at T = 8.5495 + 0.5 × KL to the frozen base model's option distribution + 0.3 × multi-class Brier score
Optimisation	AdamW, peak learning rate 5e-5, warm-up / constant / linear decay over the last 20% to 0.1×, gradient clip 1.0, one pass of 2,742 steps (~69 questions per step)
Data	190,314 decision questions (76.1M tokens) over 155 tracks
Hardware	2× NVIDIA B200, about 12 h

The data covers:

classification, NLI, verification, retrieval relevance, tool selection, preference and knowledge multiple-choice tasks built from public training splits;
programmatic tasks (dates, policies, numeric reasoning, look-ups, long documents);
synthetic and model-labelled decision tasks.

Data handling:

Option order was shuffled per question, and prompts were capped at 8,192 tokens. Longer prompts were dropped, never truncated.
Decontamination: every training row was checked against every Decision Index suite row, using 13-gram, short-field and 8-gram near-duplicate checks. No suite row is in the training data.

The full settings are in training_config.json.

Results (Decision Index 0.3, public suite)

Table with columns: Area, Weight, Skill × 100
Area	Weight	Skill × 100
Knowledge & Reasoning	25.8%	49.9
Language Understanding	25.8%	65.6
Retrieval & Classification	20.0%	69.5
Tools & Automation	18.3%	78.7
Arts & Human Taste	10%	43.5
Public index

The complete results are in sthanika-ai/Sieve-31B-decision-index-results.

Licence

The adapter and code are released under Apache-2.0. Use of the base model is governed by Gemma's terms, including Google's prohibited-use policy.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

sthanika-ai

Model Tree

Base

google/gemma-4-31B-it

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality