Usage
Install the packages in requirements.txt, then run:
import sysimport torchfrom huggingface_hub import snapshot_downloadfrom transformers import AutoModelForImageTextToText, AutoTokenizerfrom peft import PeftModel folder = snapshot_download("alphaXiv/OR-decision-4B")sys.path.insert(0, folder)from decision import decision_probabilities tokenizer = AutoTokenizer.from_pretrained(folder, trust_remote_code=False)base = AutoModelForImageTextToText.from_pretrained( "Qwen/Qwen3.5-4B", dtype=torch.bfloat16, device_map="auto", trust_remote_code=False,)model = PeftModel.from_pretrained(base, folder).eval() probabilities = decision_probabilities( model, tokenizer, context="The customer needs a copy of an invoice.", instructions="Which department should handle this request?", options=[ ("billing", "Invoices and payments"), ("technical_support", "Software and device problems"), ], kind="choice",)print(probabilities)print("Decision:", max(probabilities, key=probabilities.get))
For yes/no questions use kind="noul" and options labeled false and true.
For ordinal ratings use kind="score" and ordered, numeric option labels. The
helper returns the probability distribution; applications can calculate its
expected ordinal value or apply a decision threshold. Keep the supplied option
descriptions and full context. The helper rejects inputs exceeding the evaluated
4,096-token window and supports up to 62 distinct option labels.
Training and checkpoint selection
- Backbone: Qwen/Qwen3.5-4B.
- LoRA rank: 16; learning rate: 0.00001; global batch size: 128.
- Corpus: 117,186 training examples after a content-hash split of 123,339 source
examples. The mixture includes classification, reasoning, adequacy judgments,
and synthetic examples, with frozen Qwen teacher distributions where available.
- Objective: gold-label loss, teacher KL weight 1.0, and frozen-base KL weight 0.3.
Ordinal gold targets include adjacent-label smoothing.
- Trained across two epochs. The released checkpoint is update 925 of the second
epoch, selected by minimum gold-label negative log likelihood on a fixed
1,024-example validation set. It is not the checkpoint with the highest
validation accuracy.
- Checkpoint export uses Spindle/Miles' PEFT export. Optimizer state and the base
model weights are not included in this adapter repository.
Evaluation
The released checkpoint was evaluated using Spindle's native vocabulary readout.
The exported Transformers/PEFT runtime has not been separately rebenchmarked;
runtime, precision, and batching can produce numerical differences.
Table with columns: Evaluation, Result| Evaluation | Result |
|---|
| Fixed validation set | 754 / 1,024, 73.63% |
| Validation gold-label NLL | 0.6549 |
| JevBench, primary 4,096-token window | 193 / 231, 83.55% |
| JevBench, secondary 3,072-token window | 194 / 231, 83.98% |
| Base Qwen3.5-4B, primary JevBench window | 176 / 231, 76.19% |
JevBench used the public tasks and original scoring implementation pinned at
1bcc55eb6c8cffde2306b3db03ede39b61c6152a.
Raw and temperature-calibrated correctness counts were identical. Temperature
fitting used a separate 4,000-example calibration subset, with a separate
4,000-example calibration test. No training updates occurred during evaluation.
The helper above returns raw probabilities, without this fitted temperature.
Limitations
The validation set was reused to select among checkpoints, so its score is not
an independent test estimate. The public benchmark has also been reused across
experiments. Results are from one training seed and 231 benchmark tasks; small
accuracy differences should not be treated as robust improvements. Exact-input
overlap checks do not establish absence of semantic contamination. Confidence
values are not guaranteed to be calibrated in a new application.
This adapter was trained and evaluated on text decisions. The base model's
multimodal capabilities were not evaluated as part of this release.
License
Apache-2.0. The base model is distributed separately by Qwen under its own
Apache-2.0 license. See LICENSE and the
Qwen3.5-4B model card.