What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of earlier versions only need to change the model
name, because the interface and serving settings are unchanged.
Table | |
|---|
| Base model | Qwen/Qwen3.5-4B (Apache-2.0) |
| Training | LoRA SFT (rank 32, alpha 64, language model only), merged into bf16 weights; the release averages two such models (see What changed in Gen5) |
| Release format | bf16 weights, with a readout temperature T = 1.50 folded into the final norm weight (see Calibration). The vision tower and MTP head are unchanged from the base model; the config, tokenizer and chat template are identical to those of Gen1-Gen4. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-4B-Gen5 0 8080 # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Front end:
knowline_server.py is a single file that depends only on transformers and requests. It is the same
file as in Gen4.
- Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.
What changed in Gen5
- What we noticed. The Decision Index public suite is only part of the picture: the board's full score comes from
private tests of the same skills on new data and from tasks in new domains. To see how well our models generalize,
we built evaluation sets of the same kind: decision tasks from domains absent from training (a 34-task set, and an
18-task set of real-text decisions such as refunds, utility bills, appeals, eligibility checks, multilingual requests
and legal questions), and sibling sets that test the skill of a public benchmark on different data. On these, our
earlier models did clearly worse than their public scores suggested. For example, removing the training rows written
in the format of Decision Index benchmarks cost about 20 public points but changed our new-domain score very little.
The board shows the same pattern for our KnowLine-35B-A3B-Gen1, which ranks 7th on the public suite but 27th on the
full score of Decision Index 0.4.
- New data mix. We rebalanced the training data toward judgements over real, ambiguous text: fewer verbatim copies
of items in benchmark format and more reworded layouts; new real-text data on business policies, several languages,
legal and contract questions, tables, human judgement and syllogisms; and a smaller share of procedural items (rule
execution, arithmetic, logic puzzles). All of the data went through our decontamination check; see
Disclosures.
- Training. A model was retrained from the base Qwen3.5-4B on the new mix, for one epoch. Gen5 is the 50/50
weight average of this model and a second model, which continues our earlier 4B line on a further round of data
(rule-following and routing tasks, agent decisions, reasoning items, and a replay of earlier data). A single temperature was then fitted and folded into the weights. It changes the size of the
probabilities, but not which option ranks first.
- Selection. We compared the candidates on our estimate of the board's full score (built from our own evaluation
sets for the private same-skill and new-domain parts) and on our new-domain sets, not on the public score. Gen5 had
the highest estimate and the highest new-domain scores among them.
- What it means. The public score moved little (64.90 → 65.67): it was not the target. The gains are on the tests
that measure transfer: our new-domain sets (+5.2 on the 18-task set, +4.0 on the 34-task set), our sibling sets
(+3.0), our Chinese in-house sets (+1.6 and +1.8, both significant) and unseen injection templates (6.2% → 2.5%).
Comparison
Scores on the Decision Index. Since Decision Index 0.4 (2026-10-19), the board's full score is 60% private tests of the
same skills and 40% private tasks from new domains; the public suite is shown but no longer counts. Our full score is not
available yet.
Table with columns: model, size, DI 0.3 public, DI 0.4 full, source| model | size | DI 0.3 public | DI 0.4 full | source |
|---|
| KnowLine-4B-Gen5 | 4B | 65.67 | not yet scored | self-run, official kit |
| KnowLine-4B-Gen4 | 4B | 64.90 | not yet scored | self-run, official kit |
| StartLux-Decision v1.1 4B | 4.7B | 67.81 | 56.35 | board |
| Perplexity Decider Flash v1.2 | 4.5B | 60.06 | 55.39 | board |
| vLLM-SR d3-mini | 4.5B | 54.90 | 51.25 | board |
| OneJev 1.6 | 4.5B | 61.95 | 49.54 | board |
| Jev 1.14 | (API) | 60.70 | 62.22 | board |
| KnowLine-35B-A3B-Gen1 | 35B-A3B | 69.49 | 62.28 | board |
Board rows as of the board's latest update (Decision Index 0.4).
Public-suite scores by area (DI 0.3):
Table with columns: area, KnowLine-4B-Gen5, KnowLine-4B-Gen4| area | KnowLine-4B-Gen5 | KnowLine-4B-Gen4 |
|---|
| Knowledge & reasoning | 45.0 | 44.6 |
| Language | 66.8 | 65.5 |
| Retrieval & routing | 73.3 | 72.5 |
| Tools & agents | 87.4 | 87.6 |
| Arts & taste | 61.2 | 59.2 |
Releases
The model is trained in a self-evolving loop, and new versions will follow.
Table with columns: model, date, DI 0.3 public, golden held-out (en / zh-Hans / zh-Hant), notes| model | date | DI 0.3 public | golden held-out (en / zh-Hans / zh-Hant) | notes |
|---|
| KnowLine-4B-Gen5 (this model) | 2026-10-21 | 65.67 | 69.6 / 71.7 / 67.1 | fifth release; rebalanced data, aimed at generalization |
| KnowLine-4B-Gen4 | 2026-10-10 | 64.90 | 69.7 / 70.1 / 65.1 | fourth release |
| KnowLine-4B-Gen3 | 2026-10-09 | 63.11 | 69.4 / 70.8 / 65.0 | third release |
| KnowLine-4B-Gen2 | 2026-10-08 | 62.54 | 69.7 / 71.2 / 64.8 | second release |
| KnowLine-4B-Gen1 | 2026-10-07 | 60.47 | 68.7 / 69.9 / 64.1 | first release |
Evaluation
All results are self-run and have not been verified by a third party.
Held-out, Chinese and generalization evaluations
Table with columns: suite, Gen5, Gen4| suite | Gen5 | Gen4 |
|---|
| In-house evaluation set, English | 69.6 | 69.7 |
| In-house evaluation set, Simplified Chinese | 71.7 | 70.1 |
| In-house evaluation set, Traditional Chinese | 67.1 | 65.1 |
| C-Eval (4 categories, macro) | 79.2 | 78.7 |
| Open-Jev 1.1 test / OOD | 90.0 / 88.0 | 88.4 / 87.1 |
| JevBench / JevBench-hard | 83.1 / 66.7 | 86.1 / 72.1 |
| Decision tasks from new domains (our own, 18 real-text tasks) | 51.3 | 46.1 |
| Decision tasks from new domains (our own, 34 tasks) | 43.0 | 39.0 |
| Decision tasks from new domains (our own, 10 tasks) | 49.4 | 49.3 |
| Sibling sets (same skill, other data; mean skill) | 63.7 | 60.7 |
| Prompt injection: answers changed (lower is better) | 0.6% | 1.6% |
| Prompt injection with templates not seen in training (lower is better) | 2.5% | 6.2% |
In a paired sign test against Gen4, the gains on the Simplified and Traditional Chinese in-house sets (p = 0.004 and
p = 0.028) and on unseen injection templates (p < 0.001) are significant. The differences on the English in-house set
(p = 0.90), Open-Jev 1.1 OOD (p = 0.23) and JevBench-hard (72.1 → 66.7, p = 0.15) are not.
Web operation (Mind2Web official test splits, evaluation only)
Table with columns: split, Gen5 element selection, Gen5 operation (balanced), Gen4 element selection, Gen4 operation (balanced)| split | Gen5 element selection | Gen5 operation (balanced) | Gen4 element selection | Gen4 operation (balanced) |
|---|
| test_task (websites seen in training, new tasks) | 92.6 | 97.1 | 92.9 | 97.0 |
| test_website (new websites) | 90.9 | 96.9 | 91.9 | 95.7 |
| test_domain (new domains) | 91.8 | 98.4 | 91.7 | 97.2 |
- About the same as Gen4; the differences are within noise.
- The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
leaderboard.
Images (zero-shot)
- No image training: Gen5 keeps the base model's vision tower unchanged and saw no images in training.
knowline_server.py accepts OpenAI-style image / image_url content parts in a chat state and passes them to
SGLang; each question is still one prefill.
- How we measured it: the same reconstruction of the Decision Index 0.3.1 vision public suite as for Gen4 (11
benchmarks, chance-corrected skill and the board's weights), run through the release server.
Table with columns: benchmark, Gen5, Gen4, stock Qwen3.5-4B (same front end)| benchmark | Gen5 | Gen4 | stock Qwen3.5-4B (same front end) |
|---|
| CV-Bench | 75.5 | 75.7 | 72.1 |
| BLINK | 58.1 | 58.1 | 59.9 |
| RealWorldQA | 65.0 | 64.7 | 59.7 |
| CharXiv | 65.1 | 66.3 | 52.4 |
| InfographicVQA | 86.3 | 86.3 | 73.2 |
| Mind2Web | 78.9 | 77.4 | 65.3 |
| Winoground | 79.8 | 80.5 | 73.5 |
| KIE (CORD+FUNSD) | 94.8 | 95.5 | 95.7 |
| Moderation (Hateful Memes) | 24.2 | 27.1 | 18.4 |
| R-Bench-M | 15.5 | 14.6 | 6.3 |
| MMMU-Pro vision | 32.5 | 31.9 | 21.4 |
| weighted public index | 66.0 | 66.4 | 59.4 |
- About the same as Gen4, and 6.6 points above the stock model.
- Indicative only: these are self-run numbers on our reconstruction, not on the board's rows.
Game harness (KOF '98)
- Setup: single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for
Gen1-Gen4.
- Result: 14-3-1 against Jev, 14-3-1 against StartLux-Decision-4B and 9-9-0 against Gen4; 37-15-2 overall.
- How the games ended: against Jev, 2 of the 14 wins were by K.O. and 12 at time-out; against StartLux-Decision-4B,
6 of 14 by K.O. All 6 losses to these two came at time-out. Against Gen4, 4 of the 9 wins and 4 of the 9 losses were
by K.O.
- Play style: special_2 47%, special_1 26%, heavy kick 11%, heavy punch 7%, special_3 4%.
- Caveats: 18 games per pair give wide intervals; this is a measured result in this harness, not general
fighting-game skill.
Calibration
Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.
Table with columns: metric, Gen5, Gen4| metric | Gen5 | Gen4 |
|---|
| ECE (lower is better) | 0.04-0.05 | 0.04-0.05 |
| Brier score (lower is better) | 0.28-0.31 | 0.29-0.31 |
| Confidence ≥95% but wrong | 1.5-1.6% | 1.6-1.8% |
| Mean confidence / accuracy | 0.82-0.83 / 0.78-0.80 | 0.82 / 0.77 |
- Temperature in the weights: the final norm weight includes a readout temperature T = 1.50, so every logit is
divided by 1.50 before the softmax over the option labels. The top-ranked option does not change; the probabilities
are just less extreme. T was fitted on a set of our own evaluation items whose distribution is covered by the
training data, not on the held-out sets or on Decision Index rows. On our held-out sets, ECE with the temperature in
the weights is 0.027 (Gen4: 0.043).
- Why ranges: the board does not publish which 32 benchmarks it uses, so we report ranges over the plausible
benchmark sets, as for the earlier releases.
- Recommendation: the temperature was fitted on our data. If your use depends on probability thresholds, check
calibration on your own data.
Disclosures
- Decision Index format training data: Gen5 combines two training lines. In the data of this round, about 24% of
the rows target Decision Index benchmarks in their request format or a reworded version of it, and another 1.4% are
copies of the same items rendered in other formats. These include train splits of public datasets rewritten in that
format (for example GSM8K, WinoGrande XL, ACOS and Amazon ESCI) and synthetic items written in the same format.
- Decontamination: all training data of this round, new and replayed, was checked with the same tool as Gen4
against every Decision Index 0.3 row, all of our evaluation sets (including the new-domain and sibling sets above) and
the other public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM). About 22,000 of 3.1 million rows were
removed: rows that nearly duplicate an evaluation item, repeat its question or claim, or contain its whole state, and,
in a stricter second pass, most rows that share a source passage with an evaluation item.
- Selection on evaluations: we picked Gen5 by our estimate of the board's full score and by our new-domain sets,
not by the public score.
- Game data: labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation
states.
- No model outputs as labels: no output of Jev or any other decision model was used as a training label.
- Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
not all of them were checked by a human.
Limitations
- Knowledge-heavy reasoning: weaker than larger models. The knowledge area score is 45.0 (DI 0.3); a 4B model is
limited here, and the new-domain part of the board favours larger models.
- JevBench-hard: 66.7 against 72.1 for Gen4; the drop is not significant, but check it if your use is close to
JevBench-hard.
- New domains: better than Gen4 on our own sets, but still well below larger models; our sets are our own
construction and not the board's private tasks.
- Prompt injection: an instruction planted in the state changes the answer about 0.6% of the time on our injection
set, and about 2.5% of the time with templates not seen in training. Keep untrusted text clearly delimited.
- Images: there was no image training. The image scores above are zero-shot and self-run on our own reconstruction of
the suite, so check them on your own data before relying on them.
Citation
@misc{knowline4bgen5, title = {KnowLine-4B-Gen5: a 4B decision model}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-4B-Gen5}}
License
- Weights: Apache-2.0, the same as the base model.
- Code:
knowline_server.py is MIT (see the file header).