Berk

assay-27b

README

License: apache-2.0

In one line

It abstains. On tasks it has never seen, this model acts on 88% of bool questions at an 11.2% error rate, 84% of choice at 7.4%, and hands the rest over instead of guessing. That is a conformal prediction set against a fitted error rate, not a threshold on a softmax someone eyeballed.

Accuracy on those unseen tasks is 0.842. Every answer carries a calibrated probability, a confidence, and the set of options that could not be ruled out.

The other five models, and the dataset, are in the Assay collection.

How it is built

Backbone Qwen/Qwen3.8-27B with a LoRA adapter (r=16, alpha=32, lr=5e-05, 1.0 epoch, batch 4 x 2 accumulation, 4-bit base (QLoRA)); this repository holds the adapter (adapter/) and the evidence head; the base is loaded from Qwen/Qwen3.8-27B in 4bit (bitsandbytes) at load time. Loading downloads the base model separately; the 4-bit base needs about 15 GB of GPU memory.
The answer is read from the model's own next-token logits over option label tokens at a single decision position, so the base model's zero-shot competence is the starting point.
Questions are isolated branches over a shared state (block attention mask, restarted positions): packed and separate requests agree exactly.
Trained with cross-entropy against soft targets: human label distributions where the source has them, SORD-smoothed levels for ordinal questions, one-hot otherwise. Choice options are shuffled per example.
An evidence head (linear on the decision token, assay_head.safetensors) predicts whether the state supports the question, trained on passage-swapped negatives.
Global temperature 1.234 fitted on the calibration split of the training tasks and applied unchanged everywhere else.

Evaluation

Table with columns: split, n, accuracy, Brier, NLL, ECE, confident errors
split	n	accuracy	Brier	NLL	ECE	confident errors
seen tasks (dev), raw	5513	0.834	0.245	0.466	0.048	0.036
seen tasks (dev), scaled	5513	0.834	0.243	0.451	0.040	0.021

"Unseen tasks" are eleven datasets never used in training (bbc_news, app_reviews, scitail, medical_questions_pairs, tweet_irony, ethos, stance_climate, dream, copa, truthful_qa, hh_rlhf). "kev transfer-v4 dev" is the public suite from jaredpalmer/kev-suites (mmlu, emotion, sciq, tweet_offensive, qnli, paws and synthetic rule holdouts); none of its sources are in the training data. Brier is the multi-class sum of squared errors (0..2), ECE uses 15 bins, confident errors are answers with p >= 0.9 that are wrong.

Table with columns: transfer-v4 source, n, accuracy, Brier, ECE
transfer-v4 source	n	accuracy	Brier	ECE
composition_held_and_or	32	1.000	0.002	0.012
composition_held_conditional	32	0.812	0.347	0.177
composition_held_or_not	32	0.906	0.168	0.087

Latency on one RTX 5090 (bf16, transformers, packed questions over one state versus separate requests):

text

questions  packed_ms  separate_ms        1      109.3        109.3        3      192.5        332.3        6      250.6        663.5       12      275.4       1331.3       24      431.3       2656.1

Abstention

conformal.json holds per-question-type thresholds fitted on the seen-task calibration split (alpha 0.1, delta 0.05): a prediction-set threshold with coverage at least 1 - alpha and an act threshold on the top probability whose acted-on error rate is at most alpha at confidence 1 - delta, both on inputs distributed like the calibration split. assay.server returns them as act and set on every answer. How they carry over to other tasks:

Table with columns: split, type, coverage, set size, act rate / error among acted
split	type	coverage	set size	act rate / error among acted
unseen tasks	bool	0.84	0.97	88% / 11.2%
unseen tasks	choice	0.94	1.22	84% / 7.4%
unseen tasks	score	0.86	1.95	no threshold

Score questions get prediction sets but usually no act threshold, because exact-level accuracy is the wrong error notion for ordinal answers. Refit on your own labelled data with python -m assay.conformal for a guarantee about your distribution.

The family

Table with columns: model, size, unseen tasks, transfer-v4
model	size	unseen tasks	transfer-v4
assay-0.6b	0.6B	0.704 / 0.397	0.636 / 0.499
assay-1.7b	1.7B	0.752 / 0.334	0.670 / 0.436
assay-4b	4B	0.803 / 0.271

Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones; per-tier abstention and latency are in docs/models.md.

Serving

bash

python -m assay.server --model Berk/assay-27b --port 8000

POST /v1/decide is the native shape; POST /v1/systemone and /v1/systemone/batch accept the shape other open decision models use (criteria options, noul booleans);

text

POST /v1/decide_graph

walks a decision tree in one forward pass; POST /v1/agents registers an agent -- a graph plus the actions its outcomes stand for -- and /v1/agents/<name>/run decides a case. Requests arriving together share a pass, and /health and /metrics are for operations. assay.backends.sglang runs the same model on an SGLang deployment.

Guides: deployment and the full API, agents, runnable examples.

Train one on your own data

python -m assay.pipeline --config <your>.json runs training, temperature calibration, evaluation and the conformal thresholds over your own records, and writes a directory this same server and publisher accept. The repository's examples/ has a configuration per tier and a dataset in the record format; records written for other decision models (criteria options, noul booleans) load unchanged.

Usage

python

from assay import load_modelfrom assay.schema import Question model = load_model("Berk/assay-27b")answers = model.answer(    state="My card was charged twice for order A-104.",    questions={        "refund": Question(type="bool", instructions="Does the customer ask for money back?"),        "team": Question(type="choice", instructions="Which team should handle this?",                         options={"billing": "Charges and refunds", "technical": "Bugs"}),    },)print(answers["team"].probabilities, answers["refund"].p_true, answers["refund"].evidence)

Limitations

Text only, English training data. No arithmetic, counting, date comparison or multi-hop reasoning in one pass; keep those in code. Accuracy drops with unrelated state. The evidence head is trained on coarse swapped-passage negatives. Probabilities are calibrated in aggregate on the evaluated distributions, which is not a guarantee about any single answer or about your data; check calibration on your own labels before acting on thresholds.

Training data

Fifty-five public classification, inference, reading-comprehension and preference datasets rendered as typed questions with described options, plus a synthetic policy-application generator (see assay/data/tasks.py in the repository for the rubrics). Each dataset keeps its own licence; the per-dataset list is in docs/datasets.md. Several sources carry non-commercial or research-only terms; check them before commercial use.

Relationship to other work

Assay is an independent project. Jev and System One are names of TypeSafe AI's products and are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. Assay is not affiliated with or endorsed by either.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Berk

Model Tree

Base

Qwen/Qwen3.8-27B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

In one line

Accuracy on those unseen tasks is 0.842. Every answer carries a calibrated probability, a confidence, and the set of options that could not be ruled out.

The other five models, and the dataset, are in the Assay collection.

How it is built

Backbone Qwen/Qwen3.8-27B with a LoRA adapter (r=16, alpha=32, lr=5e-05, 1.0 epoch, batch 4 x 2 accumulation, 4-bit base (QLoRA)); this repository holds the adapter (adapter/) and the evidence head; the base is loaded from Qwen/Qwen3.8-27B in 4bit (bitsandbytes) at load time. Loading downloads the base model separately; the 4-bit base needs about 15 GB of GPU memory.
The answer is read from the model's own next-token logits over option label tokens at a single decision position, so the base model's zero-shot competence is the starting point.
Questions are isolated branches over a shared state (block attention mask, restarted positions): packed and separate requests agree exactly.
Trained with cross-entropy against soft targets: human label distributions where the source has them, SORD-smoothed levels for ordinal questions, one-hot otherwise. Choice options are shuffled per example.
An evidence head (linear on the decision token, assay_head.safetensors) predicts whether the state supports the question, trained on passage-swapped negatives.
Global temperature 1.234 fitted on the calibration split of the training tasks and applied unchanged everywhere else.

Evaluation

Table with columns: split, n, accuracy, Brier, NLL, ECE, confident errors
split	n	accuracy	Brier	NLL	ECE	confident errors
seen tasks (dev), raw	5513	0.834	0.245	0.466	0.048	0.036
seen tasks (dev), scaled	5513	0.834	0.243	0.451	0.040	0.021

Table with columns: transfer-v4 source, n, accuracy, Brier, ECE
transfer-v4 source	n	accuracy	Brier	ECE
composition_held_and_or	32	1.000	0.002	0.012
composition_held_conditional	32	0.812	0.347	0.177
composition_held_or_not	32	0.906	0.168	0.087

Latency on one RTX 5090 (bf16, transformers, packed questions over one state versus separate requests):

text

questions  packed_ms  separate_ms        1      109.3        109.3        3      192.5        332.3        6      250.6        663.5       12      275.4       1331.3       24      431.3       2656.1

Abstention

Table with columns: split, type, coverage, set size, act rate / error among acted
split	type	coverage	set size	act rate / error among acted
unseen tasks	bool	0.84	0.97	88% / 11.2%
unseen tasks	choice	0.94	1.22	84% / 7.4%
unseen tasks	score	0.86	1.95	no threshold

The family

Table with columns: model, size, unseen tasks, transfer-v4
model	size	unseen tasks	transfer-v4
assay-0.6b	0.6B	0.704 / 0.397	0.636 / 0.499
assay-1.7b	1.7B	0.752 / 0.334	0.670 / 0.436
assay-4b	4B	0.803 / 0.271

Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones; per-tier abstention and latency are in docs/models.md.

Serving

bash

python -m assay.server --model Berk/assay-27b --port 8000

POST /v1/decide is the native shape; POST /v1/systemone and /v1/systemone/batch accept the shape other open decision models use (criteria options, noul booleans);

text

POST /v1/decide_graph

Guides: deployment and the full API, agents, runnable examples.

Train one on your own data

Usage

python

from assay import load_modelfrom assay.schema import Question model = load_model("Berk/assay-27b")answers = model.answer(    state="My card was charged twice for order A-104.",    questions={        "refund": Question(type="bool", instructions="Does the customer ask for money back?"),        "team": Question(type="choice", instructions="Which team should handle this?",                         options={"billing": "Charges and refunds", "technical": "Bugs"}),    },)print(answers["team"].probabilities, answers["refund"].p_true, answers["refund"].evidence)