AstroHan

decision-head-qwen3.5-4b-rlcd-32k

README

License: apache-2.0

What it is for

Bounded decisions where the caller supplies the material, the question and the exact option set, and wants a calibrated-ish distribution over those options: request routing, answer-adequacy judging, policy yes/no checks, intent classification, ordinal severity scoring, evidence support.

It does not generate text. It cannot be prompted into free-form answers. Input is capped at 4,096 tokens and is not truncated — a longer input raises rather than silently losing material.

Results

Against the untrained base on the frozen held-out sets:

Table with columns: Set, n, base, seed 1, seed 2, seed 3
Set	n	base	seed 1	seed 2	seed 3
in-distribution	3,433	0.686	0.955	0.939	0.942
out-of-distribution	4,637	0.670	0.927	0.919	0.903
wide option sets (up to 77)	1,200	0.563	0.872	0.864	0.790
public human-labelled panel	3,880	0.717	0.749	0.728	0.721
BFCL V4 (cleaned)	3,206	0.897	0.934	0.938	0.929
RewardBench 2	1,802	0.675	0.721	0.708	0.718

JevBench v1.2 (external, 231 public items)

JevBench is run by a third party; we did not write its items. Scored with its own harness through an adapter that mirrors semif_direct, the Qwen3.5-4B entrant already on its board, so the two are comparable item for item.

Table with columns: easy 48, standard 72, hard 111, all 231
	easy 48	standard 72	hard 111	all 231
base (untrained)	0.979	0.861	0.577	0.749
seed 1 (this model)	1.000	0.986	0.550	0.779
seed 3	1.000	0.986	0.586	0.797

On the easy and standard tiers this ties Jev item for item. The whole remaining gap is the hard tier.

The part worth reading: transfer has a direction

The headline numbers hide a split that three independent evaluations agree on. Grouping held-out tasks by their relationship to the training stream:

Table with columns: relationship to training data, site, base, seed 1 (this model), seeds 2 / 3, Jev
relationship to training data	site	base	seed 1 (this model)	seeds 2 / 3	Jev
same dataset, held-out split	massive ×2, paws (mean)	80.8	87.9	88.0 / 86.6	88.0
same structure, similar material	JevBench probability	0.30	0.50	0.50 / 0.60	0.70
same structure, similar material

Six rows, three seeds, same sign in every cell. RL on a narrow slice of a structure does not merely fail to generalise to other material of that structure — it lands below the untrained starting point. The algorithm never varied, so the variable is the data, and narrow data is a liability rather than a no-op.

Two cells that look more dramatic in one seed are deliberately left out of the table: JevBench multi_hop rises 0.50 → 0.83 and tradeoff falls 0.83 → 0.33 in seed 3 only (seed 1: 0.56 and 0.83). An earlier version of this card showed those seed-3 cells next to seed-2 public-panel cells under a single "after RL" column; that was a cross-seed selection and is corrected here.

Calibration also degrades: Brier 0.342 → 0.378 on JevBench, and expected calibration error rises on the public panel. The model becomes both more accurate and more overconfident.

If you plan to use this on material that does not look like ordinary web/wiki prose, or on decisions that require weighing competing objectives, measure against the untrained base before adopting it.

Known limits

No ablation arm. This run changed data, reward, base model and learning rate at once. Gains are not attributable to any one of them.
Adapter capacity is a live confound. rank 8, attention-only, inherited unexamined from a 0.6B prototype. Whether the regressions above come from narrow data or from an adapter too small to hold two strategies is not separable from this run alone.
One-pass stream, one frozen order. All three seeds share the same data order; seeds vary only LoRA init and RLCD noise, so run-to-run variance is understated.
Arithmetic and dates are out of reach. JevBench temporal_numeric: base 0.27, after RL 0.13, Jev 0.27. One forward pass with no intermediate computation cannot do this, for anyone.
Small cells. JevBench long_policy is 19 items, tradeoff 6. The regression below base rests on three sites agreeing across three seeds (pubmedqa, vitaminc, long_policy), not on any one cell. The tradeoff drop, which would speak to "no rule decides" items directly, is seed-dependent and is not part of the claim.

Usage

python

from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE, REV = "Qwen/Qwen3.5-4B", "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"tok = AutoTokenizer.from_pretrained(BASE, revision=REV)model = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype="bfloat16", device_map="cuda")model = PeftModel.from_pretrained(model, "AstroHan/decision-head-qwen3.5-4b-rlcd-32k").eval()

The prompt format matters: the adapter was trained on one exact template (system instruction, then Material: / Question: / Options: with single-token letter labels, chat template, no thinking). decision.py in the code repository is the reference implementation — reproduce it exactly or the letter logits will not mean what they meant in training.

Training data

32,000 items in a single pass: 29,000 choice, 1,800 ordinal (3/4/5/6 levels), 1,200 boolean, drawn from eight synthetic families plus public datasets.

The mixture is not redistributed. It derives from public datasets under differing licences and from synthetic items generated through commercial APIs. The frozen manifest (SHA-256, per-family proportions, per-domain concentration, prefix drift) and the build scripts are in the code repository, and the snapshot rebuilds byte-identically from them with your own API keys.

Two ways the mixture is not diverse, measured and recorded before the run: routing has only 3 domains (largest 33.8%) and evidence only 4 (25.1%); the interleaver balances stream buckets but not sources, so a 512-item prefix has source TV distance 0.059 from the final stream. Both turned out to matter — see the transfer table above.

Compute

One H200 SXM, 9.0 GPU-hours total for three seeds plus all evaluation.

Citation

Full report with the five pre-registered predictions and their resolutions: see REPORT.md in the code repository. The criteria were written before the run; two resolved against expectation and one revealed a flaw in its own criterion, all of which the report states.

License

Apache 2.0, matching the base model. JevBench results are reproduced from its MIT-licensed repository; JevBench is not affiliated with TypeSafe AI, whose Jev model it measures.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

AstroHan

Model Tree

Base

Qwen/Qwen3.5-4B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality