How it works
Sieve follows the decision-model design of Kev:
- Backbone. Only the text part of Qwen3.5-9B is kept: 32 layers and the embeddings, 7.94B parameters.
- Removed: the language-model head, the multi-token-prediction layer and the vision encoder, 1.72B
parameters together.
- Readout. A pointer head scores the end of each option against the decision point.
- Isolation. Each question runs as its own row from the state's cache, so questions never see each other.
What Sieve-9B changes:
- Backbone. The post-trained Qwen3.5-9B instead of the base model.
- Averaged weights. The adapter and head are the exact average of two rank-16 runs from one initialisation.
- New data. 1,920 records from 240 rule structures not in the decision-v7 split, plus states padded with
unrelated text.
- Serving:
- CUDA graphs: 2 options take 18.6 ms instead of 70.2 ms.
- Long questions run as one row: 255 options take 330.5 ms instead of 371.0 ms.
Table with columns: type, options, returns| type | options | returns |
|---|
choice | the caller's keys, optionally described (up to 255) | a probability per key |
noul | yes / no | P(yes) |
score | ordered levels | a probability per level |
Use
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/sieve"
from sieve import load_sieve, decide m = load_sieve("sthanika-ai/Sieve-9B")decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."}, {"department": {"type": "choice", "instructions": "Which team handles this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}}, "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})
- Loading. Load it with
sieve; the adapter alone is not a text-generation model.
- Server.
sieve-serve --model sthanika-ai/Sieve-9B --graphs.
- GPU memory. About 20 GB; 17.6 GiB was measured with a 4,000-token state.
- Temperature.
head.pt stores T = 1.631, fitted by NLL on the decision-v7 calibration split (1,148 questions).
- That split is used only to fit T and to choose the checkpoint; no reported number comes from it.
- T never changes an answer, and
temperature=1.0 gives raw probabilities.
Results
All numbers are Sieve-9B's own. They are recomputed from its saved per-question predictions and stored in
result.json.
sieve-eval in the GitHub repository computes the same metrics for any
labelled JSONL file. With --no-merge it reproduces the public-test rows below exactly.
Decision Index 0.2.1
Sieve-9B scores 41.71 (raw index 56.47, breadth 40.06) on our own run of the full frozen
Decision Index suite. The run used the official
kit at 87d4650 and its scorer.
- Coverage. 150,317 requests over 44 benchmarks, all answered, with no truncation or errors. The index averages
38 of the benchmarks.
- Results. sthanika-ai/Sieve-9B-decision-index-results.
- Not on the leaderboard yet. The score is self-reported and expected, not confirmed, until the maintainers review
it. On the 2026-09-28 board it would place about 18th of 72.
Table with columns: Decision Index, Knowledge & Reasoning, Language Understanding, Retrieval & Classification, Tools & Automation, Arts & Human Taste| Decision Index | Knowledge & Reasoning | Language Understanding | Retrieval & Classification | Tools & Automation | Arts & Human Taste |
|---|
| 41.71 | 27.4 | 46.1 | 46.1 | 61.3 | 22.8 |
The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. The per-benchmark scores
below are in each benchmark's own metric.
Table with columns: benchmark, metric, score, requests| benchmark | metric | score | requests |
|---|
| ACOS | per-review F1 | 0.241 | 1,565 |
| ANLI | macro-F1 | 0.562 | 3,200 |
| API-Bank | accuracy | 0.726 | 508 |
| Amazon ESCI | macro-F1 | 0.493 | 5,000 |
| BANKING77 |
Decision suites, held-out benchmark and public test sets
Table with columns: benchmark, questions, accuracy [95% CI], NLL, Brier, ECE, wrong at p ≥ 0.9| benchmark | questions | accuracy [95% CI] | NLL | Brier | ECE | wrong at p ≥ 0.9 |
|---|
| decision-v7 development (in-domain) | 1,264 | 0.872 [0.851, 0.892] | 0.330 | 0.176 | 0.028 | 1.7% |
| transfer-v4 development (out-of-domain) | 656 | 0.829 [0.797, 0.860] | 0.447 | 0.242 | 0.045 |
- Calibration. The probability metrics use the stored temperature (T = 1.631), as served. Raw (T = 1) values are
in
result.json.
- Wrong at p ≥ 0.9 is the share of all questions answered wrongly with a probability of at least 0.9.
- Intervals. They come from a bootstrap with 4,000 resamples over groups of related records.
- Development partitions only. The decision suites are development partitions; no test partition of them has been
scored.
- Unmerged adapter. These rows were scored with the adapter unmerged. The Decision Index run and the latency use
the merged model, as served.
¹ In-distribution: the training data contains 1,000 Banking77 training questions.
² Our conversion. On the Decision Index's own MMLU, 14,033 questions outside the index, Sieve-9B scores 0.749.
³ The held-out benchmark:
- Size. 6,233 questions over 10 synthetic decision domains, with no family overlap with training. 6,224 were
scored; 9 were dropped because their options exceed the evaluation's 12,288-token question limit.
- Truncation. States over 384 tokens, the training limit, were truncated to 384.
- Option counts. 77 of the scored questions have 256 or 512 options, more than the served API accepts (255).
Table with columns: held-out benchmark by, questions, accuracy| held-out benchmark by | questions | accuracy |
|---|
| state ≤ 384 tokens | 5,186 | 0.889 |
| state > 384 tokens (truncated) | 1,038 | 0.697 |
| 2 options | 2,452 | 0.976 |
| 3–10 options | 2,969 | 0.823 |
| 11–50 options | 522 | 0.728 |
| 51–255 options | 204 |
The model picks the function to call from the offered ones, or "none of these". Argument values are not generated or
scored, so these are not BFCL leaderboard numbers.
Table with columns: BFCL v3, decision form, questions, accuracy| BFCL v3, decision form | questions | accuracy |
|---|
| function selection (simple, multiple, live simple, live multiple) | 1,909 | 0.963 |
| irrelevance detection (irrelevance, live irrelevance) | 1,118 | 0.720 |
Latency
Server-reported p50 on one A100 80GB (bf16, merged, CUDA graphs), for one choice question on a cached state of about
75 tokens, excluding tokenisation and HTTP:
Table with columns: options, 2, 5, 10, 25, 50, 100, 255| options | 2 | 5 | 10 | 25 | 50 | 100 | 255 |
|---|
| ms | 18.6 | 19.3 | 22.8 | 42.8 | 74.5 | 128.4 | 330.5 |
Training
- Recipe:
- LoRA r=16, α=32 on the attention, MLP and DeltaNet projections, with the pointer head trained from scratch;
- lr 5e-5, OneCycle, 2 epochs, 8 records per step, bf16;
- permuted choice options and "none of the above" augmentation, with a cross-entropy loss.
- Averaging. Two adapters were trained from one initialisation on different data mixes and averaged 50/50
("model soups", Wortsman et al., 2022).
- Selection. The checkpoint was chosen by calibration-split accuracy, then NLL. That rule was fixed after both
runs were evaluated and before the average was scored.
- The benchmarks were never used to choose. The second data mix did target weaknesses seen on the development
partitions.
- No outputs of other models were used.
- Data. The four training files are in
data/sieve-9b/,
unchanged.
Files
Table with columns: file, contents| file | contents |
|---|
adapter_model.safetensors, adapter_config.json | the LoRA adapter (rank 32, PEFT format) |
head.pt | the pointer head and the calibrated temperature |
tokenizer.json, tokenizer_config.json | Qwen3.5-9B's tokenizer, unchanged |
training_config.json, training_metrics.json, train.log | the recipe, the two runs and how they were averaged |
Credits and license
- Design. The typed-decision format, the pointer-head design and the stored temperature follow
Kev (Apache-2.0).
- Training data. It comes from Kev's decision-v7 suite, its date-policy files and its rule generator. It also
contains text from ten public datasets under their own licences, listed in the GitHub NOTICE.
- Backbone. Qwen3.5-9B, Apache-2.0.
- License. The weights are Apache-2.0 (see
LICENSE and NOTICE).