How it works
- Backbone. The post-trained Qwen3.5-4B, a hybrid of Gated DeltaNet and full-attention layers. Only its text part
is used; the model cannot produce text.
- Adapter. LoRA r=32, alpha 64, on every linear layer: attention, MLP and the Gated DeltaNet projections.
- Answer head. Each option is listed under a code (A, B, ..., AA, ...). A 255-way linear head, initialised from the
base model's own output rows for those codes, scores all codes at the question's decision point. The head reads the
whole option list.
- Isolation. The state is encoded once; each question runs as its own branch from that cache, so questions never
see each other.
- Yes/no in both orders. A yes/no question is asked with the options in both orders and the answers are averaged.
Table with columns: type, options, returns| type | options | returns |
|---|
choice | the caller's keys, optionally described (up to 255) | a probability per key |
noul | yes / no | P(yes) |
score | ordered levels (up to 10) | a probability per level |
Use
hf download sthanika-ai/Sieve-4B --local-dir Sieve-4B && cd Sieve-4Bpip install -r requirements.txt # torch, transformers>=5.17, peft>=0.21, safetensors, pydanticpip install flash-linear-attention==0.5.2 # recommended on GPU: the Gated DeltaNet kernels our runs used
from sieve4b import load, decide engine = load(".", device="cuda:0") # this folder; fetches Qwen/Qwen3.5-4B at its pinned revision on first usedecide(engine, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."}, {"department": {"type": "choice", "instructions": "Which team handles this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}}, "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})
- Server.
python -m sieve4b.server --ckpt . --port 8123 --name sieve-4b serves the same thing as a
/v1/systemone HTTP endpoint (needs fastapi and uvicorn). The Decision Index run below used it.
- GPU memory. About 10 GB in bf16. Longer states need more.
- Token limits. Up to 32,768 tokens for the state plus the longest question, 65,536 per request, 255 choice
options and 10 score levels. Longer requests are refused, never truncated. Training records reached 16,806 tokens.
- Temperatures.
sieve_config.json stores one temperature per question type: choice 1.161, yes/no 1.440 and
score 1.000. A temperature never changes an answer.
Results
Decision Index 0.3
Sieve-4B scores 50.06 (raw index 62.10, breadth 48.32) on the public 0.3 suite of the
Decision Index, on our own run with the official
kit at 9eb2dbe (0.3) and its scorer.
- Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported
request and no error. The index averages 37 of the benchmarks.
- How the run was made. A complete run of the 0.2.1 suite, extended to 0.3 with the 2,638 rebuilt GSM8K requests,
on two A100 80GB PCIe GPUs with three server processes per GPU.
- Results. sthanika-ai/Sieve-4B-decision-index-results.
- Not on the leaderboard yet. The score is self-reported. On the 0.3 board the ranking uses the Full score, where
this public suite counts 20% and the maintainers' private tests count 80%.
Table with columns: Decision Index 0.3, Knowledge & Reasoning, Language Understanding, Retrieval & Classification, Tools & Automation, Arts & Human Taste| Decision Index 0.3 | Knowledge & Reasoning | Language Understanding | Retrieval & Classification | Tools & Automation | Arts & Human Taste |
|---|
| 50.06 | 33.9 | 53.9 | 61.6 | 66.8 | 28.3 |
The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each
benchmark's own metric and the skill is chance-corrected. On edition 0.2.1 the same run scores 49.21.
Table with columns: benchmark, metric, score, skill (×100), requests| benchmark | metric | score | skill (×100) | requests |
|---|
| ACOS | per-review F1 | 0.206 | 18.1 | 1,565 |
| Amazon ESCI | macro-F1 | 0.491 | 36.2 | 5,000 |
| ANLI | macro-F1 | 0.635 | 45.3 | 3,200 |
| API-Bank |
Calibration
Table with columns: choice, yes/no | choice | yes/no |
|---|
| served temperature | 1.161 | 1.440 |
| ECE on held-out sources, out of fold | 0.039 | 0.030 |
- Fit. The served temperatures were fitted by NLL on 1,456 held-out records from 12 public sources that are not in
the training data, and reported 2-fold out of fold. A fit on the in-domain calibration split gives lower temperatures
(choice 0.914, yes/no 0.862); both reports are in
calibration/.
- On the Decision Index. Over the 0.2.1 rows, with our reimplementation of the board's calibration panel
(benchmarks with a per-field gold answer, equally weighted; confidence = the probability of the chosen option): ECE
0.019, and 0.5% of answers wrong with confidence of 0.95 or more.
Latency
HTTP wall time as recorded in the Decision Index run, with three server processes sharing each A100 80GB PCIe: median
302 ms, mean 433 ms, 80th percentile 400 ms, 95th percentile 817 ms.
Training
- Recipe.
- LoRA r=32, α=64, dropout 0.05 on the attention, MLP and Gated DeltaNet projections, and the 255-way answer head.
- AdamW, lr 5e-5 for the adapter and 1e-4 for the head, 3% warm-up then cosine, 1 epoch, 128 records per step,
bf16.
- Loss: cross-entropy on the target distribution, an option-order consistency term and a small replay term that
keeps the model near its base.
- 1,295 steps, 11.6 hours on two A100 80GB PCIe GPUs. No record was skipped or truncated.
- Selection. The last checkpoint. The benchmarks were never used to choose or tune anything.
- Data. 165,832 training records.
- Sources: public datasets and generated decision records.
- Not published. The training files are not part of this release; the SHA-256 of each file is in
training_config.json.
Files
Table with columns: file, contents| file | contents |
|---|
adapter_model.safetensors, adapter_config.json | the LoRA adapter (rank 32, PEFT format) |
head.safetensors | the 255-way answer head |
sieve_config.json | option codes, prompt preamble and the served temperatures |
calibration/ | the two temperature fits, with out-of-fold metrics |
sieve4b/ | inference code: model, request contract, engine and HTTP server |
License
Apache-2.0, see LICENSE and NOTICE.