AEECollier

s1-jev-cvltist

README

License: apache-2.0

1. What this is

system one (s1) is a small decision model trained to answer Jev-style questions (choice, score, noul) over a free-form "state", returning calibrated probabilities rather than a bare answer. It is deliberately built on a general instruction base (Qwen3.5-4B) with a thin LoRA adapter, so it carries knowledge a purpose-built encoder can't, while the readout (single-token logit scoring over option keys) keeps inference cheap and the probabilities well-formed.

The design goal was not just to match TypeSafe Jev's accuracy, but to be open and auditable, and to be structurally strong where Jev and its open clones are weak: calibration, negation/invariance consistency, and coverage (knowing when the evidence is insufficient).

2. Task

Given a state (free-form text / structured context) and one or more questions, return a well-formed answer per the Jev wire spec:

Table with columns: type, answer
type	answer
`choice`	the chosen option key + full probability vector over options
`score`	a score on a 2–10 level scale + probability distribution
`noul`	a probability in [0,1] (the "yes/no/unless" branch)

Up to 255 options per choice (vendor spec). Answers carry probabilities, not just the top pick — that is what makes calibration and coverage meaningful downstream.

3. Model

Base: Qwen/Qwen3.5-4B (Apache-2.0, 262k native context).
Adapter: LoRA (r=16, alpha=32, dropout 0.05) on every attention projection (q/k/v/o) and MLP projection (gate/up/down) plus the gated in_proj modules. ~124 MB. Inference-mode, bf16.
Readout: LogitReadout (eval/readout.py) — teacher-forces each option key and reads the single-token logit; option keys are scored in chunks (key_batch) to bound peak memory. This is the same NLL proper scoring rule used at training time.

4. Training data

mixture_v6 — 264,162 choice/score/noul rows, 1 epoch, 8,077 optimizer steps (effective batch 32, 4096 context), loss → ~0.06–0.28.

Table with columns: family, rows, %, source
family	rows	%	source
tsi	128,000	48.5%	TaskSource-instruct (129 tasks, NLI/counterfactual/knowledge, `tsi-perm` commercial-safe subset)
go_emotions	86,820	32.9%	GoEmotions (multi-label, order-augmented)
banking77	19,986	7.6%	Banking77 (77-way intent, order-augmented, key-only render)
boolq	18,854

License posture (release line): permissive / commercial-safe. Qwen3.5-4B is Apache-2.0; TSI is the commercial-safe subset; BoolQ is CC-BY-SA and enters only via the --include-sa gate. Non-commercial rows (anli, multi_nli, snli, ai2_arc, hellaswag) are quarantined out of the release mixture.

5. Evaluation — release numbers (full 35,594-row frozen holdout)

Frozen holdout, 9 families, s1-jev-cvltist at 1024 context, T=1.0. This is the citable number (the 300/family sample used during iteration is superseded by the full run).

Table with columns: metric, s1-jev-cvltist (release)
metric	s1-jev-cvltist (release)
macro accuracy (unweighted, 9 families)	0.7747
mmlu_pro (n=12,032)	0.438
negation violation (lower = more consistent)	0.022
ECE @ raw T=1 (pooled)	0.105
ECE @ best temperature (T=1.5)	0.0169
Brier @ best temperature	0.1441

Per-family accuracy (full holdout)

Table with columns: family, n, accuracy
family	n	accuracy
injection	126	0.9921
banking77	3,076	0.9324
boolq	3,270	0.9128
negation	3,270	0.9110
pubhealth	7,929	0.7874
baserate	27	0.7407

s1-jev-cvltist per-family accuracy — strong on most families, one clear knowledge gap

Strong on six families (≥0.65): injection, banking77, boolq, negation, pubhealth, baserate. The two gaps — go_emotions (0.601, the weakest large family) and mmlu_pro (0.438, the knowledge gap) — are the honest shortfalls.

Calibration

The raw pooled ECE (0.105) is high only because mmlu_pro (12k of 35.5k rows, where the model is genuinely uncertain) dominates the pool — every individual family's raw ECE is fine. Offline temperature scaling (the standard post-hoc phase) at T=1.5 brings pooled ECE to 0.0169 — the best calibration held across any version, well under the 0.03 gate — with Brier 0.1441.

JevBench (public-231, harness's own `score_task`)

Table with columns: model, all-public, hard tier
model	all-public	hard tier
s1-jev-cvltist @4096	0.7749	62/111
s1-v5	0.7489	59/111
AlexWortega/openjev v5	0.814	69/111
Jev 1.13 (hosted)	0.866	81/111

JevBench public-231 — s1 versions vs the external baselines we track

s1-jev-cvltist (coral) is the release checkpoint; s1-v3→v6 is our local progression. openjev (0.814) and Jev 1.13 (0.866) are external, documented numbers, not local runs. Gap to openjev closed from 0.065 to 0.039; Jev 1.13 remains 0.091 ahead.

Evidence-sensitivity (VitaminC flip probe, 200 conflict families)

Table with columns: s1-jev-cvltist
	s1-jev-cvltist
overall	0.799
SUPPORTS	0.919
REFUTES	0.713
NOT ENOUGH INFO	0.582
lazy_rate (lower = better)	0.090

The broad NLI/counterfactual coverage in the TSI data taught the model to return "not enough info" when the evidence genuinely runs out, instead of forcing an answer — the "don't over-claim" property that is the whole point of a coverage-aware decision model.

6. How it got here (version trajectory, frozen 300/family suite)

Table with columns: version, macro, negation, Brier, mmlu_pro
version	macro	negation	Brier	mmlu_pro
Laya (encoder baseline)	0.410	0.710	0.336	0.137
s1-v1	0.592	0.032	0.140	0.303
s1-v2	0.710	0.268*	0.161	0.320
s1-v3

* v2's negation spike was a training-data bug (built without BoolQ); fixed in v3. Each line is a deliberate data decision, documented in research/05–research/11. The v6 jump is TSI breadth: it closed most of the knowledge gap and improved the differentiator axes (negation, coverage) simultaneously.

Frozen 9-family macro accuracy — from encoder baseline (Laya) to s1-jev-cvltist

Laya (the purpose-built encoder we set out to beat) is 0.410; s1-jev-cvltist is 0.775 — +0.365 over the encoder baseline, and the largest single jump is the v5→v6 TSI-breadth step.

7. How to run it

Minimal (logit readout, no server):

python

from eval.readout import LogitReadoutro = LogitReadout(ckpt="checkpoints/s1-v6", base="Qwen/Qwen3.5-4B",                  device="cuda:0", temperature=1.5, max_len=1024, key_batch=8)answers = ro.call(state_text, {"q1": {"type": "choice", "options": {...}}})# answers["q1"] = {"choice": key, "probabilities": {key: p, ...}}

HTTP (Jev wire spec, src/s1/schema.py speaks the request/response format):

bash

python scripts/serve_s1.py --ckpt checkpoints/s1-v6 --base Qwen/Qwen3.5-4B \  --device cuda:0 --temperature 1.5 --port 8000# POST /v1/answer  { "state": "...", "questions": { ... } }

Serving note: LogitReadout + schema.py are the working pieces; serve_s1.py wraps them in FastAPI. There is no batching/continuous-batching layer yet — that is the next serving step if throughput matters.

8. Limitations (honest)

Knowledge is the gap. mmlu_pro 0.438 vs Jev's documented 0.83. This is the one axis we are clearly behind, and it is bounded by the TSI per-task extraction cap (500/task) which throttled the knowledge-heavy tasks.
go_emotions 0.601 is the weakest large family — multi-label emotion is hard and it is only in the mix via order-augmentation.
baserate (n=27) and injection (n=126) are small families; their per-family numbers carry wide confidence intervals.
Raw pooled ECE is dominated by the uncertain knowledge family; always report the temperature-scaled ECE (0.0169) or per-family ECE, not the raw pool.
Numbers are from a frozen 9-family holdout + JevBench public-231. They are not measured on Jev's private/internal suites.

9. Reproducibility

Release checkpoint: checkpoints/s1-v6 (commit 554debd..f9f229e).
Release eval: results/s1v6_release_full.json (35,594 rows) + results/calibration_s1v6_release.json.
Build: scripts/train_v6.sh (training), scripts/eval_v6.sh (eval chain), scripts/tsi_extract.py (TSI data).
Research log: research/01–research/11 (one doc per milestone).
Frozen holdout: data/holdout/*.parquet + (sha256-pinned).

10. License

Base model Apache-2.0; release mixture permissive/commercial-safe (see §4). Adapter weights released under the project license (TBD — see LICENSE).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

AEECollier

Model Tree

Base

Qwen/Qwen3.5-4B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

1. What this is

2. Task

Given a state (free-form text / structured context) and one or more questions, return a well-formed answer per the Jev wire spec:

Table with columns: type, answer
type	answer
`choice`	the chosen option key + full probability vector over options
`score`	a score on a 2–10 level scale + probability distribution
`noul`	a probability in [0,1] (the "yes/no/unless" branch)

Up to 255 options per choice (vendor spec). Answers carry probabilities, not just the top pick — that is what makes calibration and coverage meaningful downstream.

3. Model

Base: Qwen/Qwen3.5-4B (Apache-2.0, 262k native context).
Adapter: LoRA (r=16, alpha=32, dropout 0.05) on every attention projection (q/k/v/o) and MLP projection (gate/up/down) plus the gated in_proj modules. ~124 MB. Inference-mode, bf16.
Readout: LogitReadout (eval/readout.py) — teacher-forces each option key and reads the single-token logit; option keys are scored in chunks (key_batch) to bound peak memory. This is the same NLL proper scoring rule used at training time.

4. Training data

mixture_v6 — 264,162 choice/score/noul rows, 1 epoch, 8,077 optimizer steps (effective batch 32, 4096 context), loss → ~0.06–0.28.

Table with columns: family, rows, %, source
family	rows	%	source
tsi	128,000	48.5%	TaskSource-instruct (129 tasks, NLI/counterfactual/knowledge, `tsi-perm` commercial-safe subset)
go_emotions	86,820	32.9%	GoEmotions (multi-label, order-augmented)
banking77	19,986	7.6%	Banking77 (77-way intent, order-augmented, key-only render)
boolq	18,854

5. Evaluation — release numbers (full 35,594-row frozen holdout)

Frozen holdout, 9 families, s1-jev-cvltist at 1024 context, T=1.0. This is the citable number (the 300/family sample used during iteration is superseded by the full run).

Table with columns: metric, s1-jev-cvltist (release)
metric	s1-jev-cvltist (release)
macro accuracy (unweighted, 9 families)	0.7747
mmlu_pro (n=12,032)	0.438
negation violation (lower = more consistent)	0.022
ECE @ raw T=1 (pooled)	0.105
ECE @ best temperature (T=1.5)	0.0169
Brier @ best temperature	0.1441

Per-family accuracy (full holdout)

Table with columns: family, n, accuracy
family	n	accuracy
injection	126	0.9921
banking77	3,076	0.9324
boolq	3,270	0.9128
negation	3,270	0.9110
pubhealth	7,929	0.7874
baserate	27	0.7407

s1-jev-cvltist per-family accuracy — strong on most families, one clear knowledge gap

Calibration

JevBench (public-231, harness's own `score_task`)

Table with columns: model, all-public, hard tier
model	all-public	hard tier
s1-jev-cvltist @4096	0.7749	62/111
s1-v5	0.7489	59/111
AlexWortega/openjev v5	0.814	69/111
Jev 1.13 (hosted)	0.866	81/111

JevBench public-231 — s1 versions vs the external baselines we track

Evidence-sensitivity (VitaminC flip probe, 200 conflict families)

Table with columns: s1-jev-cvltist
	s1-jev-cvltist
overall	0.799
SUPPORTS	0.919
REFUTES	0.713
NOT ENOUGH INFO	0.582
lazy_rate (lower = better)	0.090

6. How it got here (version trajectory, frozen 300/family suite)

Table with columns: version, macro, negation, Brier, mmlu_pro
version	macro	negation	Brier	mmlu_pro
Laya (encoder baseline)	0.410	0.710	0.336	0.137
s1-v1	0.592	0.032	0.140	0.303
s1-v2	0.710	0.268*	0.161	0.320
s1-v3

Frozen 9-family macro accuracy — from encoder baseline (Laya) to s1-jev-cvltist

Laya (the purpose-built encoder we set out to beat) is 0.410; s1-jev-cvltist is 0.775 — +0.365 over the encoder baseline, and the largest single jump is the v5→v6 TSI-breadth step.

7. How to run it

Minimal (logit readout, no server):

python

from eval.readout import LogitReadoutro = LogitReadout(ckpt="checkpoints/s1-v6", base="Qwen/Qwen3.5-4B",                  device="cuda:0", temperature=1.5, max_len=1024, key_batch=8)answers = ro.call(state_text, {"q1": {"type": "choice", "options": {...}}})# answers["q1"] = {"choice": key, "probabilities": {key: p, ...}}

HTTP (Jev wire spec, src/s1/schema.py speaks the request/response format):

bash

python scripts/serve_s1.py --ckpt checkpoints/s1-v6 --base Qwen/Qwen3.5-4B \  --device cuda:0 --temperature 1.5 --port 8000# POST /v1/answer  { "state": "...", "questions": { ... } }

Serving note: LogitReadout + schema.py are the working pieces; serve_s1.py wraps them in FastAPI. There is no batching/continuous-batching layer yet — that is the next serving step if throughput matters.

8. Limitations (honest)

Knowledge is the gap. mmlu_pro 0.438 vs Jev's documented 0.83. This is the one axis we are clearly behind, and it is bounded by the TSI per-task extraction cap (500/task) which throttled the knowledge-heavy tasks.
go_emotions 0.601 is the weakest large family — multi-label emotion is hard and it is only in the mix via order-augmentation.
baserate (n=27) and injection (n=126) are small families; their per-family numbers carry wide confidence intervals.
Raw pooled ECE is dominated by the uncertain knowledge family; always report the temperature-scaled ECE (0.0169) or per-family ECE, not the raw pool.
Numbers are from a frozen 9-family holdout + JevBench public-231. They are not measured on Jev's private/internal suites.

9. Reproducibility

Release checkpoint: checkpoints/s1-v6 (commit 554debd..f9f229e).
Release eval: results/s1v6_release_full.json (35,594 rows) + results/calibration_s1v6_release.json.
Build: scripts/train_v6.sh (training), scripts/eval_v6.sh (eval chain), scripts/tsi_extract.py (TSI data).
Research log: research/01–research/11 (one doc per milestone).
Frozen holdout: data/holdout/*.parquet + (sha256-pinned).

10. License

Base model Apache-2.0; release mixture permissive/commercial-safe (see §4). Adapter weights released under the project license (TBD — see LICENSE).

s1-jev-cvltist

README

1. What this is

2. Task

3. Model

4. Training data

5. Evaluation — release numbers (full 35,594-row frozen holdout)

Per-family accuracy (full holdout)

Calibration

JevBench (public-231, harness's own score_task)

Evidence-sensitivity (VitaminC flip probe, 200 conflict families)

6. How it got here (version trajectory, frozen 300/family suite)

7. How to run it

8. Limitations (honest)

9. Reproducibility

10. License

Explore FriendliAI today

README

1. What this is

2. Task

3. Model

4. Training data

5. Evaluation — release numbers (full 35,594-row frozen holdout)

Per-family accuracy (full holdout)

Calibration

JevBench (public-231, harness's own score_task)

Evidence-sensitivity (VitaminC flip probe, 200 conflict families)

6. How it got here (version trajectory, frozen 300/family suite)

7. How to run it

8. Limitations (honest)

9. Reproducibility

10. License

JevBench (public-231, harness's own `score_task`)

JevBench (public-231, harness's own `score_task`)