Quickstart
from arc_decide import Decider # arc_decide.py ships in this repo (torch + transformers) d = Decider("arc53/arc-decide-0.4b-v0.2")chunk = """## Rotating API keys To rotate a key, open **Settings → API keys**, select the key and click **Rotate**. A new key is issuedimmediately; the old key keeps working for 24 hours so you can update your clients, then it is revoked."""d.decide( {"query": "How do I rotate an API key?", "passage": chunk}, {"relevant": {"type": "noul", "instructions": "Is the passage relevant to the query: does it contain " "information that answers or directly addresses it?"}},)# {'relevant': {'type': 'noul', 'noul': 0.99}}
On CPU, the same call through the ONNX client:
import json from huggingface_hub import hf_hub_downloadfrom arc_decide_onnx import Decider # arc_decide_onnx.py ships here too (onnxruntime + tokenizers + numpy) questions = json.load(open(hf_hub_download("arc53/arc-decide-0.4b-v0.2", "questions.json")))d = Decider("arc53/arc-decide-0.4b-v0.2") # fetches onnx/ onlyd.decide({"query": "How do I rotate an API key?", "passage": chunk}, {"relevant": questions["relevant"]})# {'relevant': {'type': 'noul', 'noul': 0.99}}
Put every question about a state in one call: the state is encoded once and each question reuses its cache.
Questions follow TypeSafe's System One shapes:
noul: yes/no, answered with P(yes);choice: one of 2–16 options you pass in;score: a 3- or 4-level rubric.
Use the canonical questions. questions.json has every question the model was trained on, verbatim, under
the name that selects its fitted temperature. They are:
Table with columns: Family, Question names| Family | Question names |
|---|
| retrieval | relevant, grade |
| guardrails | injection, severity, supported |
| routing | needs_retrieval, needs_clarification, complexity |
| conversation analytics | answer_status, , , , , , , |
States use the key names shown there:
{"query", "passage"} for relevance;{"text"} for injection;{"source", "claim"} for grounding, one claim per call;{"message"} for routing.
A state is cut at 2,048 tokens, keeping its start. For a check of your own, the practised form is
{"type": "noul", "instructions": "Based on the passage, answer this yes/no question:\n<question>?", "criteria": {"true": "Yes", "false": "No"}}.
On CPU
Two ONNX files ship with this model. Both have int8 weights (blocks of 32, with the four matrix multiplies most
sensitive to quantisation kept in fp32); they differ in the embedding table:
onnx/ (the default): an int8 embedding table. It fits in 1 GB on every call shape.onnx-bf16emb/: the embedding table kept exactly as trained (bf16), for about 0.13 GB more. On our samples
its answers are no closer to PyTorch's than the default's (table below): the int8 weights account for most of the
difference.
from huggingface_hub import snapshot_download local = snapshot_download("arc53/arc-decide-0.4b-v0.2", allow_patterns=["onnx-bf16emb/*"], local_dir="arc-decide-0.4b-v0.2")d = Decider(f"{local}/onnx-bf16emb")d.decide({"query": "How do I rotate an API key?", "passage": chunk}, {"relevant": questions["relevant"]})# {'relevant': {'type': 'noul', 'noul': 0.99}}
arc_decide_onnx.py needs only onnxruntime (1.23 or later), tokenizers and numpy. By default it runs in a
low-memory mode that gives the same answers bit for bit: one key-value buffer shared by every pass, a single pass
for single-question calls, prefill in 256-token steps, and an onnxruntime arena that grows by exactly what it needs
and gives memory back after every call. Decider(path, low_memory=False) runs plain session.run calls instead.
A low-memory Decider serialises its calls; use one per thread to run calls in parallel.
Peak memory (private plus file-backed, with the whole embedding table resident, as in a long-running server) and
the time per call, each call shape in a fresh process:
Table with columns: file, guard: one injection check (171 tokens), rerank: 20 documentation chunks (22,002 tokens), analytics: a 2,815-token turn, 8 questions, all three, one process, same, low_memory=False, download| file | guard: one injection check (171 tokens) | rerank: 20 documentation chunks (22,002 tokens) | analytics: a 2,815-token turn, 8 questions | all three, one process | same, low_memory=False | download |
|---|
onnx/ | 683 MiB, 0.21 s | 864 MiB, 27 s | 969 MiB, 4.8 s | 978 MiB | 1,246 MiB | 499 MiB |
onnx-bf16emb/ |
Measured on 4 vCPU of an AMD EPYC 9354P 32-Core Processor, onnxruntime 1.30.0.
The same measurement with the files and client in this repo, on an AMD EPYC 9J45, came within 25 MiB of these
peaks, at about 1.8 times the speed.
A 40-chunk relevance prescreen takes about a minute on 4 cores, so a busy prescreen still belongs on a GPU.
The int8 files were checked against the fp32 PyTorch model on the same inputs, drawn from the evaluation sets of
every family. A flipped decision is a yes/no answer that crossed 0.5, or a different top option:
Table with columns: file, sample, questions, max |Δp|, mean |Δp|, decisions flipped| file | sample | questions | max |Δp| | mean |Δp| | decisions flipped |
|---|
onnx/ | a fresh sample (drawn after the recipe was chosen) | 439 | 0.0142 | 0.0010 | 0 |
onnx/ | the large sample | 433 | 0.0317 | 0.0012 | 0 |
The fresh-sample rows ran this repo's client on its files (AMD EPYC 9J45); the large-sample rows come from the
packaging run (EPYC 9354P). int8 kernels differ slightly between processors, so the same file's numbers do too.
And on the task metrics, the default file against PyTorch on the same items:
Table with columns: PyTorch, onnx/ | PyTorch | onnx/ |
|---|
| DocsGPT-native relevance, nDCG@10 | 0.914 | 0.914 |
| deepset injection, AUROC | 0.982 | 0.983 |
| NotInject false positives at 0.5 | 5.3% | 5.3% |
| Analytics, an-native, 3,000 items (seeded subsample): labels within 1 point / largest change | – | 9 of 9 / goal met −0.7 [−1.9, 0.0] |
| Analytics, an-realistic, 6,446 items: labels within 1 point / largest change | – | 9 of 9 / reaction −0.2 [−0.9, +0.3] |
On Apple Silicon
On a Mac, use the PyTorch client on the GPU: onnxruntime's int8 kernels are slow on Apple's CPUs.
d = Decider("arc53/arc-decide-0.4b-v0.2", device="mps", dtype="float16")
On an M1 with 16 GB, while other work was running: one injection check 0.14 s, one ~1.2k-token relevance chunk
0.8 s, 10 chunks 5.7 s, an analytics turn with 8 questions 2.2 s. Metal reserves about 1.2 GB. fp16 on the GPU
matched fp32 on the CPU to within 0.0007 on every probability we checked. The int8 ONNX file on the same M1's CPU
took 2.2 s, 15 s, 117 s and 18 s for the same calls.
On a GPU
Stock vLLM serves the safetensors weights, because the heads are folded into the LM head. Start it with
--logprobs-mode processed_logprobs --max-logprobs 16. For each question:
- Send the prompt as token ids, with
max_tokens: 1 and allowed_token_ids set to the head's labels from
arcd_serving.json.
- Apply the bias and temperature from that file, as
arc_decide.py does.
On a GPU, v0.1 is the stronger model at little extra cost.
Good to know
- Calibration. Temperatures are fitted per question on held-out data, so 0.5 means even odds. Every analytics
label is within 5.8 points of calibrated on held-out conversations, except topic (10).
- This version has not been measured on production traffic.
- v0.1's predecessor was overconfident there on why-unanswered, reaction and request type. Report those three as
rates, not thresholded per conversation, until you have checked them on your traffic.
- Not a sole security control. Injection detection is strong on known attack styles (0.98 AUROC on deepset) and
weaker on novel ones (0.79–0.85). It flags 5% of benign-but-alarming text at 0.5. Use a threshold of at least
0.9 and other defences too.
- Passage length. Relevance was trained on documentation chunks, from a paragraph up to about 1,400 tokens.
One-line snippets score markedly lower, so rank them against each other rather than thresholding them.
- Languages. Most labels are within 1.5 points of English in every language. The exceptions:
- topic is 11 points behind in Russian and 5 in Spanish;
- answer status and why-unanswered are 2–5 behind;
- Spanish and Chinese MIRACL relevance (0.57) are the weakest multilingual scores.
- takes up to 16 options and 3 or 4 levels. Questions far from the canonical ones are
much less reliable, so validate them on your own data.
Baselines were run on the same harness, following their model cards. The DocsGPT-native and analytics sets come
from ten documentation sites held out of all training data. Analytics gold labels are Kimi K2.5 and DeepSeek V4 Pro
where the two agree decisively, so they measure agreement with strong models, not human judgement.
Retrieval relevance (nDCG@10)
Table with columns: set, arc-decide-0.4b-v0.2 (0.41B), arc-decide-0.6b-v0.1 (0.6B), bge-reranker-v2-m3 (568M), Qwen3-Reranker-0.6B (0.6B), Qwen3-Reranker-4B (4B)| set | arc-decide-0.4b-v0.2 (0.41B) | arc-decide-0.6b-v0.1 (0.6B) | bge-reranker-v2-m3 (568M) | Qwen3-Reranker-0.6B (0.6B) | Qwen3-Reranker-4B (4B) |
|---|
| docsgpt-native | 0.916 | 0.916 | 0.881 | 0.874 | 0.912 |
| beir-scifact | 0.750 | 0.756 | 0.731 | 0.762 | 0.783 |
| beir-fiqa |
Prompt injection (AUROC)
Table with columns: set, arc-decide-0.4b-v0.2 (0.41B), arc-decide-0.6b-v0.1 (0.6B), Llama Prompt Guard 2 86M (86M), ProtectAI DeBERTa-v3 injection v2 (184M), Laya (System One encoder) (421M)| set | arc-decide-0.4b-v0.2 (0.41B) | arc-decide-0.6b-v0.1 (0.6B) | Llama Prompt Guard 2 86M (86M) | ProtectAI DeBERTa-v3 injection v2 (184M) | Laya (System One encoder) (421M) |
|---|
| inj-deepset | 0.982 | 0.992 | 0.910 | 0.905 | 0.901 |
| inj-jailbreakhub | 0.854 | 0.851 | 0.811 | 0.890 | 0.664 |
| inj-safeguard |
Over-refusal: NotInject false-positive rate at 0.5 (lower is better)
Table with columns: set, arc-decide-0.4b-v0.2 (0.41B), arc-decide-0.6b-v0.1 (0.6B), Llama Prompt Guard 2 86M (86M), ProtectAI DeBERTa-v3 injection v2 (184M), Laya (System One encoder) (421M)| set | arc-decide-0.4b-v0.2 (0.41B) | arc-decide-0.6b-v0.1 (0.6B) | Llama Prompt Guard 2 86M (86M) | ProtectAI DeBERTa-v3 injection v2 (184M) | Laya (System One encoder) (421M) |
|---|
| inj-notinject | 5.3 | 12.1 | 5.0 | 43.1 | 19.8 |
Grounding: is the claim supported by the source (balanced accuracy)
Table with columns: set, arc-decide-0.4b-v0.2 (0.41B), arc-decide-0.6b-v0.1 (0.6B), HHEM-2.1-Open (~0.25B), MiniCheck-RoBERTa-L (355M), Bespoke-MiniCheck-7B (7B), Laya (System One encoder) (421M)| set | arc-decide-0.4b-v0.2 (0.41B) | arc-decide-0.6b-v0.1 (0.6B) | HHEM-2.1-Open (~0.25B) | MiniCheck-RoBERTa-L (355M) | Bespoke-MiniCheck-7B (7B) | Laya (System One encoder) (421M) |
|---|
| gr-aggrefact | 0.701 | 0.712 | 0.715 | 0.748 | 0.768 | 0.595 |
| gr-ragtruth | 0.687 | 0.706 | 0.660 | 0.593 |
Intent over options given at request time (accuracy)
Table with columns: set, arc-decide-0.4b-v0.2 (0.41B), arc-decide-0.6b-v0.1 (0.6B), GLiClass modern-base v3 (~150M), Qwen3-4B-Instruct (label logits) (4B), Laya (System One encoder) (421M)| set | arc-decide-0.4b-v0.2 (0.41B) | arc-decide-0.6b-v0.1 (0.6B) | GLiClass modern-base v3 (~150M) | Qwen3-4B-Instruct (label logits) (4B) | Laya (System One encoder) (421M) |
|---|
| ch-banking77 | 85.7 | 87.5 | 76.9 | 88.9 | 86.5 |
| ch-clinc-unseen | 97.9 | 97.2 | 85.6 | 97.8 | 98.3 |
| ch-massive-unseen |
Conversation analytics (held-out doc sites, four languages; macro-F1 for choices, balanced accuracy for yes/no; ECE in points)
Table with columns: label, arc-decide-0.4b-v0.2 (0.41B), arc-decide-0.6b-v0.1 (0.6B), embedding + LR (TnT-LLM), same labels (311M), Qwen3-4B-Instruct (label logits) (4B), Laya (System One encoder) (421M)| label | arc-decide-0.4b-v0.2 (0.41B) | arc-decide-0.6b-v0.1 (0.6B) | embedding + LR (TnT-LLM), same labels (311M) | Qwen3-4B-Instruct (label logits) (4B) | Laya (System One encoder) (421M) |
|---|
| answer_status | 81.9 / 3.5 | 81.8 / 3.3 | 52.6 / 1.9 | 50.8 / 38.5 | 16.9 / 5.1 |
| unanswered_reason | 70.0 / 1.4 | 69.0 / 1.4 | 41.2 / 2.8 | 51.4 / 28.9 | 8.6 / 25.7 |
Training
- Model: the first 16 of arc-decide-0.6b-v0.1's 28 layers (407,278,592 parameters, counting the embedding once;
the base model is Qwen3-0.6B-Base), with v0.1's typed linear heads (yes/no, 3- and 4-level scores, a 16-way
choice) read at the last token. The questions about one state share its encoding through a prefix fork.
- Objective: v0.1's training mix, with every target mixed half and half with v0.1's own probability for that row
and question (2.1M rows, 2.8M questions), then temperature-scaled per question on held-out rows against the
original labels.
- Relevance: 640k public retrieval pairs plus questions over 23 documentation sites, labelled by Qwen3.6-27B, and
every judged MIRACL passage for Spanish, Chinese and Russian (v0.1 kept five per query).
- Guardrails, grounding, routing and QA: public sets, translated to Spanish, Chinese and Russian.
- Analytics:
- about 50k simulated conversations and 10k agent traces over documentation sites, generated with open models
and labelled by Gemma 4 and gpt-oss-120b;
- plus WildChat reactions.
- No real user conversations were used.
No data was added beyond v0.1's sources. DATA.md lists every training source, its licence, and every model whose
outputs are in the data.
Licence
MIT. The base model, Qwen3-0.6B-Base, is Apache-2.0; its licence ships as LICENSE-Qwen3.
@misc{arc53-arc-decide, title = {arc-decide: a small calibrated decision model for RAG and agents}, author = {Arc53}, year = {2026}, url = {https://huggingface.co/arc53/arc-decide-0.4b-v0.2}}
