sthanika-ai

Sieve-4B

README

License: apache-2.0

Summary

Table

Backbone	Qwen/Qwen3.5-4B (revision `851bf6e8`)
Design	option codes
Trained weights	LoRA r=32, α=64 on every linear layer, and a 255-way answer head
Question types	`choice`, `noul` and `score` (up to 10 levels)
Decision Index	Full score 40.96 and public score 50.06 on the leaderboard (edition 0.4.1)
Temperature	one per question type: choice 1.161, yes/no 1.440, score 1.000
Token limits	32,768 tokens for the state plus the longest question, 65,536 per request
GPU memory	about 10 GB in bf16

How it works

Backbone. The post-trained Qwen3.5-4B, a hybrid of Gated DeltaNet and full-attention layers. Only its text part is used; the model cannot produce text.
Adapter. LoRA r=32, alpha 64, on every linear layer: attention, MLP and the Gated DeltaNet projections.
Answer head. Each option is listed under a code (A, B, ..., AA, ...). A 255-way linear head, initialised from the base model's own output rows for those codes, scores all codes at the question's decision point. The head reads the whole option list.
Isolation. The state is encoded once; each question runs as its own branch from that cache, so questions never see each other.
Yes/no in both orders. A yes/no question is asked with the options in both orders and the answers are averaged.

Table with columns: type, options, returns
type	options	returns
`choice`	the caller's keys, optionally described (2 to 255)	the most likely key, a confidence, a probability per key
`noul`	yes / no	P(yes)
`score`	an ordered list of 2 to 10 levels	the expected level, a confidence, a probability per level

Use

bash

pip install -r https://raw.githubusercontent.com/sthanika-ai/Sieve/main/requirements.txt   # the tested versions (torch 2.8.0)pip install "sieve-decisions[serve] @ git+https://github.com/sthanika-ai/Sieve"

python

from sieve import load_sieve, decide m = load_sieve("sthanika-ai/Sieve-4B")     # fetches Qwen/Qwen3.5-4B at its pinned revision on first usedecide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},       {"department": {"type": "choice", "instructions": "Which team handles this?",                        "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

Server. python -m sieve.codes.server --ckpt sthanika-ai/Sieve-4B --port 8123 --name sieve-4b serves the same thing as a /v1/systemone HTTP endpoint. The Decision Index run below used it.

Decision Index.

text

python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-4B --out runs/Sieve-4B

answers each request as that server does, in process.

GPU memory. About 10 GB in bf16. Longer states need more.
Token limits. Up to 32,768 tokens for the state plus the longest question, 65,536 per request, 255 choice options and 10 score levels. Longer requests are refused, never truncated. Training records reached 16,806 tokens.
Temperatures. sieve_config.json stores one temperature per question type: choice 1.161, yes/no 1.440 and score 1.000. A temperature never changes an answer.

Results

Decision Index

On the Decision Index leaderboard (edition 0.4.1, board data of 2026-10-28), Sieve-4B has a Full score of 40.96 and a public score of 50.06. Our own run of the public 0.3 suite, with the official kit at 9eb2dbe (0.3) and its scorer, scores 50.06 (raw index 62.10, breadth 48.32).

Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
How the run was made. A complete run of the 0.2.1 suite, extended to 0.3 with the 2,638 rebuilt GSM8K requests, on two A100 80GB PCIe GPUs with three server processes per GPU.
Results. sthanika-ai/Sieve-4B-decision-index-results.
Full score. The board ranks by the Full score, which comes from the maintainers' private tests: 60% tests of the same skills as the public benchmarks and 40% decision tasks from new domains.

Table with columns: Decision Index 0.3, public suite, Knowledge & Reasoning, Language Understanding, Retrieval & Classification, Tools & Automation, Arts & Human Taste
Decision Index 0.3, public suite	Knowledge & Reasoning	Language Understanding	Retrieval & Classification	Tools & Automation	Arts & Human Taste
50.06	33.9	53.9	61.6	66.8	28.3

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected. On edition 0.2.1 the same run scores 49.21.

Table with columns: benchmark, metric, score, skill (×100), requests
benchmark	metric	score	skill (×100)	requests
ACOS	per-review F1	0.206	18.1	1,565
Amazon ESCI	macro-F1	0.491	36.2	5,000
ANLI	macro-F1	0.635	45.3	3,200
API-Bank	accuracy	0.738	73.3	508
BANKING77	macro-F1	0.828	82.6	3,080
BBH fixed-option tasks	accuracy	0.682	53.9	5,507
BFCL	case exact accuracy	0.944	92.4	1,694
BPoMP	accuracy	0.804	61.0	5,000
BRIGHT	nDCG@10	0.444	37.1	220
cfcolor	accuracy	0.613	17.8	5,000
ChessBench	accuracy	0.138	6.1	5,000
CLadder	accuracy	0.639	27.8	5,000
CLINC150+OOS	macro-F1	0.902	90.1	5,500
ContractNLI	macro-F1	0.798	70.8	123
CRUXEval	accuracy	0.558	29.9	570
FinEntity	macro-F1	0.886	83.2	979
GPQA Diamond	accuracy	0.423	23.1	196
GSM8K	accuracy	0.645	57.4	2,638
Habermas Machine	accuracy	0.400	12.9	1,676
HellaSwag	accuracy	0.950	93.4	10,042
HLE	accuracy	0.094	0.0	501
Home appliance simulator	case exact accuracy	0.261	26.1	88
HoVer claim verification	accuracy	0.750	50.0	4,000
Humicroedit	accuracy	0.595	18.9	2,628
iSarcasmEval	Sarcasm F1 · track A, English	0.461	30.6	4,600
MMLU-Pro	accuracy	0.550	49.4	12,032
MuSR	accuracy	0.551	28.5	752
New Yorker caption matching	accuracy	0.616	51.9	528
NLI4CT	macro-F1	0.768	54.9	5,500
PhishNChips phishing decisions	accuracy	0.843	68.5	2,000
POP909-CL	accuracy	0.079	7.2	2,000
RAGTruth response-level hallucination	F1 on hallucinated class	0.779	54.1	2,700
SATA-Bench	case exact accuracy	0.206	19.6	1,650
ToolRet	nDCG@10	0.673	62.3	685
VAST	macro-F1	0.524	28.6	3,006
When2Call MCQ	accuracy	0.799	73.2	3,652
WinoGrande	accuracy	0.858	71.6	1,267
ARC-Challenge (not in the index)	accuracy	0.944	92.5	1,172
ARC-Easy (not in the index)	accuracy	0.979	97.1	2,376
MMLU (not in the index)	accuracy	0.758	67.7	14,033
RouterBench (not in the index)	selected quality (quality objective)	0.790	50.8	10,000
SGD/SGD-X (not in the index)	macro-F1	0.382	0.0	2,500
SimpleBench (not in the index)	accuracy	0.100	0.0	10

Calibration

Table with columns: choice, yes/no
	choice	yes/no
served temperature	1.161	1.440
ECE on held-out sources, out of fold	0.039	0.030

Fit. The served temperatures were fitted by NLL on 1,456 held-out records from 12 public sources that are not in the training data, and reported 2-fold out of fold. A fit on the in-domain calibration split gives lower temperatures (choice 0.914, yes/no 0.862); both reports are in calibration/.
On the Decision Index. The leaderboard's calibration panel (edition 0.4.1) gives Sieve-4B a calibration error (ECE) of 0.031, and 0.5% of the answers it gives with a top probability of at least 95% are wrong.

Latency

HTTP wall time as recorded in the Decision Index run, with three server processes sharing each A100 80GB PCIe: median 302 ms, mean 433 ms, 80th percentile 400 ms, 95th percentile 817 ms.

Training

Recipe.
- LoRA r=32, α=64, dropout 0.05 on the attention, MLP and Gated DeltaNet projections, and the 255-way answer head.
- AdamW, lr 5e-5 for the adapter and 1e-4 for the head, 3% warm-up then cosine, 1 epoch, 128 records per step, bf16.
- Loss: cross-entropy on the target distribution, an option-order consistency term and a small replay term that keeps the model near its base.
- 1,295 steps, 11.6 hours on two A100 80GB PCIe GPUs. No record was skipped or truncated.
Selection. The last checkpoint. The benchmarks were never used to choose or tune anything.
Data. 165,832 training records.
- Sources. Public datasets and generated decision records.
- Not published. The training files are not part of this release; the SHA-256 of each file is in training_config.json.

Files

Table with columns: file, contents
file	contents
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (rank 32, PEFT format)
`head.safetensors`	the 255-way answer head
`sieve_config.json`	option codes, prompt preamble and the served temperatures
`calibration/`	the two temperature fits, with out-of-fold metrics
`tokenizer.json`, `tokenizer_config.json`	Qwen3.5-4B's tokenizer, unchanged
`training_config.json`, `training_metrics.json`, `train.log`	the recipe, the run and its log
`result.json`	the Decision Index, calibration and latency numbers above
`provenance.json`	SHA-256 of every file and of the training data, and library versions

License

Apache-2.0, see LICENSE.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

sthanika-ai

Model Tree

Base

Qwen/Qwen3.5-4B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality