Raymond1122

metask-jev-4b-policy-mix

README

License: apache-2.0

On the JevBench board

Self-measured axes inserted into the published v1.2.7 ranking (16 official entrants + this model). Official run pending — axes here use our 231-decision protocol for Intelligence, val-fit temperature for Calibration, self-hosted 4090 for Speed/Cost.

Would rank #5 — ahead of GPT-5.6 Luna and DeepSeek V4.1 Flash, behind djev — with the top-right quadrant of the Intelligence×Speed plane to itself among open weights:

JevBench v1.2 — public 231 decisions, tier split (422-as-wrong protocol, @4096 ctx):

Table with columns: tier, items, metask-jev-4b
tier	items	metask-jev-4b
judge (original)	72	98.6%
easy	48	100.0%
hard	111	59.5%
total	231	80.1%

The hard tier contains long policy documents: at the 9B pipeline's 2048-token limit 36 of 111 items are rejected; this model natively handles 4096 and answers 88% of them correctly. Context length, not capability, was the bottleneck.

Head-to-head summary

Table with columns: metask-jev-4b, Bespoke Nimble-9B, Jev 1.13.0
	metask-jev-4b	Bespoke Nimble-9B	Jev 1.13.0
13 human-labeled subsets (3,880 items), macro	79.6%	74.8%	76.0%
JevBench v1.2 public 231 @4096 ctx	80.1%	63.5%	75.3
ECE after per-kind temperature	0.028	—	—
p50 latency (single question)	~24 ms	~190 ms

12 of 13 subsets exceed Bespoke Nimble-9B — a model 2.2× its size — same prompt format, same scoring protocol.

13 human-labeled subsets (3,880 items)

The primary suite: BoolQ, MultiNLI, PAWS, PubMedQA, SQuAD-2, VitaminC, Civil Comments, Aegis 2.0, MASSIVE (en/de), HelpSteer-2, SummEval (consistency / relevance). Every item human-labeled; byte-reproducible (manifest-locked ids + sha256); same protocol as the Bespoke Nimble evaluation.

Table with columns: subset, type, n, metask-jev-4b, 95% CI, Nimble-9B, Δ
subset	type	n	metask-jev-4b	95% CI	Nimble-9B	Δ
civil_comments	noul	300	91.3%	87.6–94.0	70.3%	+21.0
paws	noul	250	94.0%	90.3–96.3	82.8%

Wins: verification-style noul (civil +21.0, paws +11.2) and consistency scoring (+9.7). Loss: summeval-relevance — a 5-level rubric with a systematic 3↔4 boundary shift; see Honest limits.

vs Laya (421M, the strongest open small-model baseline)

Laya trains a 25M marker head on ModernBERT-large with RLCD (pure RL, no cross-entropy) over ~30k human-labeled decisions; its typed-decisions checkpoint reports 0.766 acc / 0.062 Brier on its own 400-case suite. Different architectures, different suites — the comparison below is indicative, not apples-to-apples.

Table with columns: metask-jev-4b, laya
	metask-jev-4b	laya
backbone	Qwen3.5-4B (decoder, LoRA merged)	ModernBERT-large (encoder + 25M head)
params	4.54B	421M
context	4096 (native 32k)	512 (root) / 1024 (typed-decisions ckpt)
training	SFT, candidate CE, 44.8k decisions	RLCD (proper-scoring reward), ~30k
raw ECE	0.100	0.466
ECE after temp

Where we win: calibration out of the box (raw ECE 0.100 is below laya's post-temperature 0.081; after our own temperature fit it is 0.028, ~3× lower), long-context hard items (59.5% on JevBench hard — laya's 512–1024 budget cannot run that tier), and 12/13 over Nimble-9B on human-labeled data.

Where laya wins: parameter efficiency (421M vs 4.5B), 100+ languages via its multilingual checkpoint, a mature packaging story (PyPI, Router, demo Space), and the RLCD training methodology is fully documented (arXiv:2510.01237).

Calibration

Ships over-confident, like every model in this family. One temperature per question kind, fit by NLL minimization on a held-out validation split (never on eval). ECE (10 bins): 0.100 → 0.028.

Table with columns: kind, T
kind	T
choice	1.7875
noul	2.25
score	2.05

Score evolution

Speed

Single forward pass over the prompt, one softmax over ≤26 candidate logits.

Training

Backbone — Qwen3.5-4B @ 851bf6e, LoRA r16 α32 on all language-model linear layers, merged at release.
Supervision — 44.8k view-augmented decisions from 11 public datasets (3 criteria orderings per item; gold follows its option, killing position-collapse priors).
Policy-mix — 390 synthetic policy-family decisions (long_policy, multi_hop, temporal_numeric, judge_hard, trap, probability, ambiguous, adversarial, tradeoff) with teacher soft labels, 2× upsampled — mirroring the JevBench hard-tier families at ≤2048-token states.
Objective — candidate cross-entropy at the last prompt position. 1 epoch, lr 2e-5, batch 4×2, BF16 + gradient checkpointing. Single RTX 4090, 2h34m, peak 19 GB.

Objective and prompt format are unchanged from the official Nimble protocol; the recipe card with reproduction commands lives in the GitHub repo.

Honest limits

summeval-relevance (26.7%) is the one clear regression vs 9B (49.2%): a 5-level rubric with a systematic 3↔4 boundary shift. NLL and expected-score error are actually better than 9B — the argmax metric amplifies the boundary shift. If your use case is fine-grained relevance scoring, evaluate this subset yourself first.
helpsteer2 (43.0%): rubric scoring is the weakest primitive family-wide (9B 39.0%, Jev ~50%).
1 item over 4096 tokens is still rejected (422-scored-wrong under JevBench protocol).
Distillation share: 390 of 45.6k training decisions (~0.9%) carry teacher soft labels; the rest are human-labeled public data.
Temperatures are fit on our validation split. Refit on your own data before trusting probabilities in a new domain (one NLL sweep, minutes).

Intended use

Routing, triage, moderation, guardrails, evidence-grounded verification, rubric scoring — anywhere calibrated probabilities matter more than generated explanations. Not a generative model.

Licence

Apache-2.0. Qwen3.5-4B base keeps its own terms.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Raymond1122

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

On the JevBench board

Would rank #5 — ahead of GPT-5.6 Luna and DeepSeek V4.1 Flash, behind djev — with the top-right quadrant of the Intelligence×Speed plane to itself among open weights:

JevBench v1.2 — public 231 decisions, tier split (422-as-wrong protocol, @4096 ctx):

Table with columns: tier, items, metask-jev-4b
tier	items	metask-jev-4b
judge (original)	72	98.6%
easy	48	100.0%
hard	111	59.5%
total	231	80.1%

Head-to-head summary

Table with columns: metask-jev-4b, Bespoke Nimble-9B, Jev 1.13.0
	metask-jev-4b	Bespoke Nimble-9B	Jev 1.13.0
13 human-labeled subsets (3,880 items), macro	79.6%	74.8%	76.0%
JevBench v1.2 public 231 @4096 ctx	80.1%	63.5%	75.3
ECE after per-kind temperature	0.028	—	—
p50 latency (single question)	~24 ms	~190 ms

12 of 13 subsets exceed Bespoke Nimble-9B — a model 2.2× its size — same prompt format, same scoring protocol.

13 human-labeled subsets (3,880 items)

Table with columns: subset, type, n, metask-jev-4b, 95% CI, Nimble-9B, Δ
subset	type	n	metask-jev-4b	95% CI	Nimble-9B	Δ
civil_comments	noul	300	91.3%	87.6–94.0	70.3%	+21.0
paws	noul	250	94.0%	90.3–96.3	82.8%

Wins: verification-style noul (civil +21.0, paws +11.2) and consistency scoring (+9.7). Loss: summeval-relevance — a 5-level rubric with a systematic 3↔4 boundary shift; see Honest limits.

vs Laya (421M, the strongest open small-model baseline)

Table with columns: metask-jev-4b, laya
	metask-jev-4b	laya
backbone	Qwen3.5-4B (decoder, LoRA merged)	ModernBERT-large (encoder + 25M head)
params	4.54B	421M
context	4096 (native 32k)	512 (root) / 1024 (typed-decisions ckpt)
training	SFT, candidate CE, 44.8k decisions	RLCD (proper-scoring reward), ~30k
raw ECE	0.100	0.466
ECE after temp

Calibration

Ships over-confident, like every model in this family. One temperature per question kind, fit by NLL minimization on a held-out validation split (never on eval). ECE (10 bins): 0.100 → 0.028.

Table with columns: kind, T
kind	T
choice	1.7875
noul	2.25
score	2.05

Score evolution

Speed

Single forward pass over the prompt, one softmax over ≤26 candidate logits.

Training

Backbone — Qwen3.5-4B @ 851bf6e, LoRA r16 α32 on all language-model linear layers, merged at release.
Supervision — 44.8k view-augmented decisions from 11 public datasets (3 criteria orderings per item; gold follows its option, killing position-collapse priors).
Policy-mix — 390 synthetic policy-family decisions (long_policy, multi_hop, temporal_numeric, judge_hard, trap, probability, ambiguous, adversarial, tradeoff) with teacher soft labels, 2× upsampled — mirroring the JevBench hard-tier families at ≤2048-token states.
Objective — candidate cross-entropy at the last prompt position. 1 epoch, lr 2e-5, batch 4×2, BF16 + gradient checkpointing. Single RTX 4090, 2h34m, peak 19 GB.

Objective and prompt format are unchanged from the official Nimble protocol; the recipe card with reproduction commands lives in the GitHub repo.

Honest limits

summeval-relevance (26.7%) is the one clear regression vs 9B (49.2%): a 5-level rubric with a systematic 3↔4 boundary shift. NLL and expected-score error are actually better than 9B — the argmax metric amplifies the boundary shift. If your use case is fine-grained relevance scoring, evaluate this subset yourself first.
helpsteer2 (43.0%): rubric scoring is the weakest primitive family-wide (9B 39.0%, Jev ~50%).
1 item over 4096 tokens is still rejected (422-scored-wrong under JevBench protocol).
Distillation share: 390 of 45.6k training decisions (~0.9%) carry teacher soft labels; the rest are human-labeled public data.
Temperatures are fit on our validation split. Refit on your own data before trusting probabilities in a new domain (one NLL sweep, minutes).

Intended use

Routing, triage, moderation, guardrails, evidence-grounded verification, rubric scoring — anywhere calibrated probabilities matter more than generated explanations. Not a generative model.

Licence

Apache-2.0. Qwen3.5-4B base keeps its own terms.

metask-jev-4b-policy-mix

README

On the JevBench board

Head-to-head summary

13 human-labeled subsets (3,880 items)

vs Laya (421M, the strongest open small-model baseline)

Calibration

Score evolution

Speed

Training

Honest limits

Intended use

Links

Licence

Explore FriendliAI today

README

On the JevBench board

Head-to-head summary

13 human-labeled subsets (3,880 items)

vs Laya (421M, the strongest open small-model baseline)

Calibration

Score evolution

Speed

Training

Honest limits

Intended use

Links

Licence