How it works
- Backbone. Only the text part of Qwen3.5-9B is used; the model cannot produce text.
- Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the
probabilities.
- Isolation. Each question runs as its own row from the state's cache, so questions never see each other.
Table with columns: type, options, returns| type | options | returns |
|---|
choice | the caller's keys, optionally described (up to 255) | a probability per key |
noul | yes / no | P(yes) |
score | ordered levels | a probability per level |
Use
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"pip install --no-build-isolation causal-conv1d==1.7.0 # recommended: the fused kernel our runs used
from sieve import load_sieve, decide m = load_sieve("sthanika-ai/Sieve-9B-v2")decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."}, {"department": {"type": "choice", "instructions": "Which team handles this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}}, "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})
- Loading. Load it with
sieve 0.2.0 or later; the adapter alone is not a text-generation model. The backbone is
downloaded at its pinned revision on first use.
- GPU memory. About 20 GB in bf16. Longer states need more.
- Token limits. States and questions up to 16,384 tokens each, as in training.
sieve 0.2.1 or later uses these
limits by default; 0.2.0 stops at 8,192 tokens. Longer requests are refused, never truncated. The Decision Index
engine declares larger limits (a 65,536-token state and a 32,768-token question). In the Decision Index run below,
241 requests had a question longer than 16,384 tokens (at most 29,944, from ToolRet and BRIGHT) and were answered
in full; no state was longer than 16,384.
- Server.
sieve-serve --model sthanika-ai/Sieve-9B-v2 --graphs.
- causal-conv1d. Without it, transformers falls back to a slower PyTorch convolution, whose probabilities can
differ slightly. Install it to reproduce our numbers.
- Temperature per category.
head.pt stores a global T = 1.109 and 23 temperatures (1.000 to 1.338), fitted by
NLL on the calibration split (43,176 questions). A question's category is its type, its number of options (2, 3-4,
5-9, 10-25, 26+) and the kind of state (empty, text or JSON). A question takes the temperature of its most specific
category in the table (choice:3-4:text, then , then ), else the global T. Temperatures never
change an answer, and gives raw probabilities. versions before 0.2.0 ignore the table and
use the global T.
Results
Decision Index 0.3
Sieve-9B-v2 scores 53.35 (raw index 64.39, breadth 51.95) on the public 0.3 suite of the
Decision Index, on our own run with the official
kit at 62d2f51 (0.3) and its scorer.
- Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported
request and no error. The index averages 37 of the benchmarks.
- How the run was made. One complete run of the 0.3 suite on two H100 80GB GPUs, in four shards.
- Results. sthanika-ai/Sieve-9B-v2-decision-index-results.
- Not on the leaderboard yet. The score is self-reported. On the 0.3 board the ranking uses the Full score, where
this public suite counts 20% and the maintainers' private tests count 80%.
Table with columns: Decision Index 0.3, Knowledge & Reasoning, Language Understanding, Retrieval & Classification, Tools & Automation, Arts & Human Taste| Decision Index 0.3 | Knowledge & Reasoning | Language Understanding | Retrieval & Classification | Tools & Automation | Arts & Human Taste |
|---|
| 53.35 | 38.3 | 55.9 | 64.2 | 69.8 | 34.0 |
The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each
benchmark's own metric and the skill is chance-corrected.
Table with columns: benchmark, metric, score, skill (×100), requests| benchmark | metric | score | skill (×100) | requests |
|---|
| ACOS | per-review F1 | 0.175 | 14.8 | 1,565 |
| Amazon ESCI | macro-F1 | 0.499 | 37.2 | 5,000 |
| ANLI | macro-F1 | 0.651 | 47.7 | 3,200 |
| API-Bank |
Calibration split
The calibration split chose the checkpoint and fitted the temperatures.
Table with columns: split, questions, accuracy, NLL, Brier, ECE| split | questions | accuracy | NLL | Brier | ECE |
|---|
| calibration, per-category temperatures (shipped) | 43,176 | 0.8775 | 0.3071 | 0.1676 | 0.0068 |
| calibration, one global T = 1.109 | 43,176 | 0.8775 | 0.3077 | 0.1678 | 0.0074 |
| calibration, raw (T = 1) | 43,176 |
In a 2-fold check inside the calibration split (fit on one half, score the other), the per-category table and the
single T scored the same: NLL 0.3079 vs 0.3078, ECE 0.0087 vs 0.0084.
Latency
- Per Decision Index request. Measured on one H100 80GB GPU, single process, eager and in-process, over a random
3,000-request sample of the public 0.3 suite: median 43 ms, mean 138 ms, 80th percentile 58 ms, 95th percentile
144 ms. The answers on that sample matched the full run exactly.
Training
- Recipe.
- LoRA r=32, α=64, dropout 0.05 on the attention, MLP and linear-attention projections (12 modules), with the
pointer head trained from scratch.
- AdamW, lr 5e-5 for the adapter and the head, OneCycle with 10% warm-up, 1 epoch, 32 records per step, bf16.
- States and questions up to 16,384 tokens each; longer records are dropped, never truncated (none were).
- 5,183 steps, 5.6 hours on two H100 80GB GPUs.
- Permuted choice options and "none of the above" augmentation, with a cross-entropy loss.
- Selection. The checkpoint with the highest accuracy on a fixed 2,000-question calibration sample, with NLL
breaking ties. This rule was fixed before training, and it chose step 4,500. The benchmarks were never used to
choose.
- Temperatures. Fitted after selection on the whole calibration split: the global T, then one T per category with
at least 200 calibration questions.
- Data. 165,832 training records and 30,722 calibration records.
- Sources: public datasets and generated decision records.
- The training files are not part of this release; the SHA-256 of each file is in
.
Files
Table with columns: file, contents| file | contents |
|---|
adapter_model.safetensors, adapter_config.json | the LoRA adapter (rank 32, PEFT format) |
head.pt | the pointer head and the calibrated temperatures (global and per category) |
tokenizer.json, tokenizer_config.json | Qwen3.5-9B's tokenizer, unchanged |
training_config.json, training_metrics.json, train.log | the recipe, the run and its log |
License
Apache-2.0, see LICENSE.