What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of earlier versions only need to change the model
name, because the interface and serving settings are unchanged.
Table | |
|---|
| Base model | Qwen/Qwen3.5-4B (Apache-2.0) |
| Training | LoRA SFT (rank 32, alpha 64, language model only), merged into bf16 weights |
| Release format | bf16 weights, with a readout temperature T = 1.32 folded into the final norm weight (see Calibration). The vision tower and MTP head are unchanged from the base model; the config, tokenizer and chat template are identical to those of Gen1-Gen3. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-4B-Gen4 0 8080 # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Front end:
knowline_server.py is a single file that depends only on transformers and requests. It is the same
file as in Gen3.
- Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.
Comparison
Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers
run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.
Table with columns: model, size, DI 0.3 public, DI 0.3 full, source| model | size | DI 0.3 public | DI 0.3 full | source |
|---|
| KnowLine-4B-Gen4 | 4B | 64.90 | not yet scored | self-run, official kit |
| KnowLine-4B-Gen3 | 4B | 63.11 | not yet scored | self-run, official kit |
| KnowLine-4B-Gen2 | 4B | 62.54 | not yet scored | self-run, official kit |
| KnowLine-4B-Gen1 | 4B | 60.47 | not yet scored | self-run, official kit |
| Perplexity Decider v1.1 | 27B | 62.25 | 62.75 | board |
| Clef | 27B | 61.71 | 53.08 | board |
| Jev 1.13 | (API) | 57.96 | 60.11 | board |
| RSI-Jev v6.1-VL | 4B | 50.98 | not listed | self-reported |
| ezjev 4B s2 | 4B | 50.82 | 46.95 | board |
| jiwo 4B | 4B | 45.76 | 42.86 | board |
| Nox 4B | 4B | 44.21 | 44.95 | board |
Public-suite scores by area (DI 0.3):
Table with columns: area, KnowLine-4B-Gen4, KnowLine-4B-Gen3, Perplexity Decider v1.1 (27B), Jev 1.13| area | KnowLine-4B-Gen4 | KnowLine-4B-Gen3 | Perplexity Decider v1.1 (27B) | Jev 1.13 |
|---|
| Knowledge & reasoning | 44.6 | 43.9 | 52.0 | 53.9 |
| Language | 65.5 | 64.8 | 67.5 | 59.2 |
| Retrieval & routing | 72.5 | 70.9 | 61.3 | 55.4 |
| Tools & agents | 87.6 | 86.6 | 78.9 | 75.1 |
| Arts & taste | 59.2 | 49.8 | 47.0 | 39.1 |
New versions will follow.
Table with columns: model, date, DI 0.3 public, DI 0.2.1, notes| model | date | DI 0.3 public | DI 0.2.1 | notes |
|---|
| KnowLine-4B-Gen4 (this model) | 2026-10-10 | 64.90 | — | fourth release |
| KnowLine-4B-Gen3 | 2026-10-09 | 63.11 | — | third release |
| KnowLine-4B-Gen2 | 2026-10-08 | 62.54 | — | second release |
| KnowLine-4B-Gen1 | 2026-10-07 | 60.47 | 60.92 | first release |
Since Gen2, we run only Decision Index 0.3, not 0.2.1.
Evaluation
All results are self-run and have not been verified by a third party.
Other public benchmarks
Table with columns: suite, Gen4, Gen3| suite | Gen4 | Gen3 |
|---|
| C-Eval (4 categories, macro) | 78.7 | 79.2 |
| JevBench / JevBench-hard | 86.1 / 72.1 | 86.6 / 72.1 |
Web operation (Mind2Web official test splits)
Table with columns: split, Gen4 element selection, Gen4 operation (balanced), Gen3 element selection, Gen3 operation (balanced)| split | Gen4 element selection | Gen4 operation (balanced) | Gen3 element selection | Gen3 operation (balanced) |
|---|
| test_task (Cross-Task) | 92.9 | 97.0 | 92.7 | 97.0 |
| test_website (Cross-Website) | 91.9 | 95.7 | 91.3 | 97.3 |
| test_domain (Cross-Domain) | 91.7 | 97.2 | 91.8 | 97.2 |
- About the same as Gen3. The lower operation score on test_website comes from 3 more misses among its 72 SELECT
steps; the differences are within noise.
- So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some
degree.
- The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
leaderboard.
Calibration
Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.
Table with columns: metric, Gen4, Gen3, board median (114 models), Jev 1.13| metric | Gen4 | Gen3 | board median (114 models) | Jev 1.13 |
|---|
| ECE (lower is better) | 0.04-0.05 | 0.07-0.08 | 0.084 | 0.074 |
| Brier score (lower is better) | 0.29-0.31 | 0.31-0.33 | 0.49 | 0.36 |
| Confidence ≥95% but wrong | 1.7% | 2.7% | 1.3% | 2.1% |
| Mean confidence / accuracy | 0.82 / 0.77 | 0.84 / 0.76 | | 0.81 / 0.74 |
- Temperature in the weights: the final norm weight includes a readout temperature T = 1.32, so every logit is
divided by 1.32 before the softmax over the option labels. The top-ranked option does not change, so neither do the
Decision Index scores; the probabilities are just less extreme. T was fitted on held-out data.
- Why ranges: the board does not publish which 32 benchmarks it uses. We checked our computation against three
board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the
plausible benchmark sets.
- Still slightly overconfident: mean confidence is about 3-5 points above accuracy. Most of the gap comes from hard
reasoning (HLE, CRUXEval, MuSR) and New Yorker caption matching.
- Recommendation: if your use depends on probability thresholds, check calibration on your own data.
Disclosures
- Decontamination:
- All training data of this round, new and replayed, was checked with a revised tool against every Decision Index
0.3 row and the public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM).
- Rows that nearly duplicate an evaluation item, repeat its question or claim, or contain its whole state were
removed (1,909 rows). Rows that only share a source passage with an evaluation item, such as the same Wikipedia
paragraph or tool description, were kept.
- No model outputs as labels: no output of Jev or any other decision model was used as a training label.
- Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
not all of them were checked by a human.
- Public suite: the Decision Index public suite is public; the private tests may show a lower score.
Limitations
- Knowledge-heavy reasoning: weaker than larger models. The knowledge area score is 44.6 (DI 0.3); MMLU-Pro barely
moved (51.6), and HLE remains at chance level.
- Math: answers are given without step-by-step reasoning. The rebuilt 0.3 GSM8K scores 61.6, about the same as
Gen3.
- Prompt injection: an instruction planted in the state can change the answer. Keep untrusted text clearly
delimited.
- Images: there was no image training; the serving path accepts images, but image use is zero-shot, so check it on
your own data before relying on it.
- Private tests: the 0.3 private tests may score lower than the public suite.
Citation
@misc{knowline4bgen4, title = {KnowLine-4B-Gen4: a 4B decision model}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-4B-Gen4}}
License
- Weights: Apache-2.0, the same as the base model.
- Code:
knowline_server.py is MIT (see the file header).