sthanika-ai

Sieve-27B

README

License: apache-2.0

How it works

Backbone. Only the text part of Qwen3.8-27B is used; the model cannot produce text.
Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the probabilities.
Isolation. Each question runs as its own row from the state's cache, so questions never see each other.

Table with columns: type, options, returns
type	options	returns
`choice`	the caller's keys, optionally described (up to 255)	a probability per key
`noul`	yes / no	P(yes)
`score`	ordered levels	a probability per level

Use

bash

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"pip install --no-build-isolation causal-conv1d==1.7.0   # recommended: the fused kernel our runs used

python

from sieve import load_sieve, decide m = load_sieve("sthanika-ai/Sieve-27B")decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},       {"department": {"type": "choice", "instructions": "Which team handles this?",                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

Loading. Load it with sieve; the adapter alone is not a text-generation model. The backbone is downloaded at its pinned revision on first use.
GPU memory. About 55 GB in bf16 (52,683 MiB in use after loading), so one 80 GB GPU is enough.
Token limits. States and questions up to 16,384 tokens each, as in training. sieve 0.2.1 or later uses these limits by default; earlier versions stop at 8,192 tokens. Longer requests are refused, never truncated. The Decision Index engine declares larger limits (a 65,536-token state and a 32,768-token question). In the Decision Index run below, 241 requests had a question longer than 16,384 tokens (at most 29,944, from ToolRet and BRIGHT) and were answered in full; no state was longer than 16,384.
Server. sieve-serve --model sthanika-ai/Sieve-27B --graphs.
causal-conv1d. Without it, transformers falls back to a slower PyTorch convolution, whose probabilities can differ slightly. Install it to reproduce our numbers.
Temperature. head.pt stores T = 1.050, fitted by NLL on the calibration split (43,309 questions). T never changes an answer, and temperature=1.0 gives raw probabilities.

Results

Decision Index 0.3

Sieve-27B scores 60.04 (raw index 69.32, breadth 58.67) on the public 0.3 suite of the Decision Index, on our own run with the official kit at 62d2f51 (0.3) and its scorer.

Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
How the run was made. One complete run of the 0.3 suite on two A100 80GB PCIe GPUs, in two shards.
Results. sthanika-ai/Sieve-27B-decision-index-results.
Not on the leaderboard yet. The score is self-reported. On the 0.3 board the ranking uses the Full score, where this public suite counts 20% and the maintainers' private tests count 80%.

Table with columns: Decision Index 0.3, Knowledge & Reasoning, Language Understanding, Retrieval & Classification, Tools & Automation, Arts & Human Taste
Decision Index 0.3	Knowledge & Reasoning	Language Understanding	Retrieval & Classification	Tools & Automation	Arts & Human Taste
60.04	43.5	65.7	67.8	77.5	40.8

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected.

Table with columns: benchmark, metric, score, skill (×100), requests
benchmark	metric	score	skill (×100)	requests
ACOS	per-review F1	0.258	23.4	1,565
Amazon ESCI	macro-F1	0.599	49.7	5,000
ANLI	macro-F1	0.760	64.0	3,200
API-Bank

Calibration split

The calibration split chose the checkpoint and fitted the temperature.

Table with columns: split, questions, accuracy, NLL, Brier, ECE
split	questions	accuracy	NLL	Brier	ECE
calibration, T = 1.050 (shipped)	43,309	0.9013	0.2499	0.1369	0.0054
calibration, raw (T = 1)	43,309	0.9013	0.2502	0.1368	0.0061

Latency

Per Decision Index request. In-process wall time, one request at a time, eager, over the 140,620 requests of the 0.3 run (A100 80GB PCIe): median 151 ms, mean 416 ms, 80th percentile 425 ms, 95th percentile 966 ms. This is not the maintainers' RTX PRO 6000 measurement.
Cached state. Server-reported p50 on one A100 80GB PCIe (bf16, merged, CUDA graphs), for one choice question on a 1,433-token state:

Table with columns: options, 2, 10, 50
options	2	10	50
state cached (ms)	137.5	141.6	236.5
state new (ms)	580.8	583.8	678.6

By option count. For one choice question on a 41-token state, which is below the server's 64-token caching threshold, so every request ran the state again. Server p50 in ms, CUDA graphs / eager:

Table with columns: options, 2, 5, 10, 25, 50, 100, 255
options	2	5	10	25	50	100	255
graphs	177.3	182.7	186.7	252.6	369.4	535.8	1,182.2
eager	257.4	257.9	259.8	275.0

Training

Recipe.
- LoRA r=32, α=64, dropout 0.05 on the attention, MLP and linear-attention projections (12 modules), with the pointer head trained from scratch.
- AdamW, lr 5e-5 for the adapter and the head, OneCycle with 10% warm-up, 1 epoch, 32 records per step, bf16.
- States and questions up to 16,384 tokens each.
- 4,813 steps, about 27 hours on two A100 80GB PCIe GPUs.
- Permuted choice options and "none of the above" augmentation, with a cross-entropy loss.
Selection. The checkpoint with the highest accuracy on a fixed 2,001-question calibration sample, with NLL breaking ties. This rule was fixed before training, and it chose step 4,500. The benchmarks were never used to choose.
Data. 153,988 training records, with 30,776 for calibration and 123,177 for test (50/10/40).
- Sources: public datasets and generated decision records.
- Not published. The training files are not part of this release; the SHA-256 of each file is in .

Files

Table with columns: file, contents
file	contents
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (rank 32, PEFT format)
`head.pt`	the pointer head and the calibrated temperature
`tokenizer.json`, `tokenizer_config.json`	Qwen3.8-27B's tokenizer, unchanged
`training_config.json`, `training_metrics.json`, `train.log`	the recipe, the run and its log

License

Apache-2.0, see LICENSE.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

sthanika-ai

Model Tree

Base

Qwen/Qwen3.8-27B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality