Results: Typed Decisions (general, zero-shot)
LocalLLaMA/typed-decisions, test split: 400 cases
and 2,000 decisions, one request per case, all answered with no errors. Served with vLLM nightly with the benchmark's
flags (--max-model-len 8192 --max-num-seqs 8 --logprobs-mode raw_logprobs) and scored with the benchmark's scorer.
Table with columns: Accuracy ↑, KL from gold ↓, Brier ↓, ECE ↓ | Accuracy ↑ | KL from gold ↓ | Brier ↓ | ECE ↓ |
|---|
| Signpost-D 4B | 0.712 [0.691, 0.734] | 0.162 | 0.085 | 0.091 |
| Prior (ignores the input) | 0.470 | 0.347 | 0.189 | 0.088 |
| Uniform | 0.308 | 0.444 | 0.238 | 0.169 |
The 95% interval is a bootstrap over cases.
Table with columns: Type, n, Accuracy| Type | n | Accuracy |
|---|
noul | 600 | 0.788 |
choice | 600 | 0.692 |
score | 800 | 0.670 |
- Latency: median 62 ms per case (5 questions), on one Colab G4 GPU (RTX PRO 6000) with vLLM at fp16.
- Predictions: all 2,000 are in
typed_decisions_test_predictions.json
(case index in the test split, question id, type, probabilities).
Gold is the mean of three samples from a teacher of roughly 4B-class capability, so these scores measure agreement with
that teacher; the benchmark gives the teacher's self-agreement as 0.735.
How the model was chosen
Candidates were compared on the benchmark's train split, which was never trained on. Accuracy and KL on all 1,200
cases:
Table with columns: Candidate, train split| Candidate | train split |
|---|
| seed 42 | 0.712 / 0.174 |
| seed 1042 | 0.714 / 0.171 |
| seed 2042 | 0.706 / 0.177 |
| soup of 3 seeds, α 1.0 | 0.715 / 0.168 |
| soup of 3 seeds, α 0.75 (this model) | 0.720 / 0.166 |
There were four test runs for this model and its candidates: two candidate data mixes (0.7055 and 0.7105), one data
batch that was then dropped (0.702), and this model, scored once (0.712). Every choice was made on the train split
first; test confirmed it and never reversed one. The published weights were rebuilt from the three adapters and
scored again on test before upload, with identical results.
Training
- Data: only our own generated data: 2,575 records (12,274 questions) in 163 invented domains. Each record is a short JSON state with five questions.
- LLM generators design the families of records and write the records.
- The gold is the mean of three simulated judges: a careful operator, a by-the-book auditor, and a generalist.
- Every record passes accept-or-drop checks.
- No row, label or question of Typed Decisions is in the data. No record uses the benchmark's four workflows (agent
traces, customer service, invoices, security incidents) or their question schemas. The generators never had
access to the benchmark.
- Use of the
train split: the model was never trained, fitted or calibrated on it: no weights, temperatures or
priors. We analysed a sample of it to understand the decisions it asks for and the shape of their gold
distributions. That analysis was used to prompt the LLM generators for the data-generation jobs, together with three
simulated judges.
- Format: one question per sequence, in exactly the prompt vLLM's
/v1/systemone builds for this architecture:
the state as JSON, the question, the labelled options and a reply line; no system message, thinking off. It is
trained on the logit of the first answer token over the option labels.
- Recipe:
- LoRA rank 16 (α 16) on 200 modules, including the linear-attention projections;
- learning rate 1e-4, 15% warm-up, dropout 0.05, 2 epochs, fp16;
- loss: soft-label cross-entropy, plus 0.5 × the expected-level error on
score questions;
- about 37 minutes per seed on one Colab G4 GPU (RTX PRO 6000).
- Soup: the three seeds' LoRA weight changes (seeds 42, 1042 and 2042, epoch 2) are averaged, and 0.75 of that
average is added to the base weights. The adapters are in
adapters/, and merged.json records the merge.
Use
vllm serve skamalj/signpost-d-4b --max-model-len 8192 --max-num-seqs 8 --logprobs-mode raw_logprobs curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "skamalj/signpost-d-4b", "state": {"order": {"id": "A-17", "paid": true, "shipped": false, "days_since_payment": 9}}, "questions": { "late": {"type": "noul", "instructions": "The order is late to ship.", "criteria": null}, "next": {"type": "choice", "instructions": "What should happen next?", "criteria": {"ship": "Ship now", "refund": "Refund the payment", "wait": "Wait for stock"}}, "risk": {"type": "score", "instructions": "How likely is a complaint?", "criteria": ["Low", "Medium", "High"]} }}'
- vLLM version: measured with vLLM nightly
0.33.0rc1.dev451; /v1/systemone needs a current nightly.
- Temperature: the probabilities are used as served, with no temperature.
Limits
- Weakest type:
score questions (0.670).
- Language: trained and measured in English only.
- Teacher, not truth: the benchmark measures agreement with its teacher. Where the teacher is wrong, a better
answer can score lower.
- Not a chat model: it doesn't write text.