What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no generated text;
- the output is the probability of each option's label token;
- existing Jev clients only need a new base URL; Gen1 users only change the model name, since the interface and
serving settings are the same.
Table | |
|---|
| Base model | Qwen/Qwen3.5-4B (Apache-2.0) |
| Training | Two LoRA SFT runs (rank 32, alpha 64, language model only), merged, then averaged tensor by tensor: Gen1 (run "mix E", step 5,650) at 0.5, and steps 4,500 and 5,000 of run "mix F" at 0.25 each. |
| Release format | bf16 weights. The vision tower and MTP head are the base model's; config, tokenizer and chat template are identical to Gen1. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-4B-Gen2 0 8080 # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Without SGLang:
python knowline_server.py --model PelaAI/KnowLine-4B-Gen2 --backend hf --port 8080 uses
transformers only. It is slower and runs bf16.
- Front end:
knowline_server.py is a single file and needs only transformers and requests. It is the same file
as in Gen1.
- Full settings: the exact settings of our Decision Index run are in INFERENCE.md.
What changed in Gen2
- Finding the gaps. Gen1 was weakest on knowledge and reasoning (37.7), and behind earlier internal models on
Jev-style evaluations such as JevBench-hard. The agent traced the latter to the data mix: to balance channels,
Gen1's data had cut the Jev-style data (OpenJevData and synthetic tasks) by about 70%.
- New data (mix F). On top of Gen1's data:
- all Jev-style data restored (about 180k OpenJevData rows and 76k synthetic rows);
- fighting-game data regenerated with balanced behaviour;
- a new group aimed at Decision Index gaps: ACOS (5% "yes", with hard negatives), WinoGrande XL train, and GSM8K
train rewritten with Decision Index 0.3 style distractors;
- the new rows went through decontamination again. Mix F has about 1.05M rows (Gen1's mix E: about 713k).
- Training mix F. Planned for 3 epochs. After the first epoch the model started to memorise the data and its
Decision Index score declined, so the run was stopped at step 8,455. On their own, mix F checkpoints were below Gen1
on most evaluations, but clearly more robust to prompt injection.
- Weight averaging. We tried 7 ways of averaging Gen1 with mix F checkpoints:
- 6 used a single mix F checkpoint (steps 4,500 / 4,800 / 5,000 / 5,500; Gen1 weight 0.4-0.6) and all scored
61.89-62.14 on the 0.3 public suite;
- averaging mix F steps 4,500 and 5,000 first, then taking half of that and half of Gen1, scored 62.54 and was at or
above Gen1 on all of our own evaluations. That is Gen2.
Comparison
Decision Index 0.3, public suite. The full 0.3 score adds private tests that only the maintainers run (0.5 same-skill,
0.3 new-domain); ours is not available yet.
Table with columns: model, size, DI 0.3 public, DI 0.3 full, source| model | size | DI 0.3 public | DI 0.3 full | source |
|---|
| KnowLine-4B-Gen2 | 4B | 62.54 | not yet scored | self-run, official kit |
| KnowLine-4B-Gen1 | 4B | 60.47 | not yet scored | self-run, official kit |
| Clef | 27B | 61.71 | 53.08 | board |
Public-suite scores by area:
Table with columns: area, KnowLine-4B-Gen2 (0.3 public), KnowLine-4B-Gen1 (0.3 public), Jev 1.13 (0.2.1)| area | KnowLine-4B-Gen2 (0.3 public) | KnowLine-4B-Gen1 (0.3 public) | Jev 1.13 (0.2.1) |
|---|
| Knowledge & reasoning | 42.7 | 37.7 | 51.4 |
| Language | 63.3 | 60.8 | 62.0 |
| Retrieval & routing | 71.3 | 70.9 | 55.4 |
| Tools & agents | 86.6 | 86.8 | 75.1 |
Releases
The model is trained in a self-evolving loop, and new versions will follow.
Table with columns: model, date, DI 0.3 public, DI 0.2.1, golden held-out (en / zh-Hans / zh-Hant), notes| model | date | DI 0.3 public | DI 0.2.1 | golden held-out (en / zh-Hans / zh-Hant) | notes |
|---|
| KnowLine-4B-Gen4 | 2026-10-10 | 64.90 | — | 69.7 / 70.1 / 65.1 | fourth release |
| KnowLine-4B-Gen3 | 2026-10-09 | 63.11 | — |
From Gen2 on we run Decision Index 0.3 only, not 0.2.1.
Evaluation
All results are self-run and not verified by a third party.
Held-out and Chinese evaluations
Table with columns: suite, Gen2, Gen1| suite | Gen2 | Gen1 |
|---|
| In-house evaluation set, English | 69.7 | 68.7 |
| In-house evaluation set, Simplified Chinese | 71.2 | 69.9 |
| In-house evaluation set, Traditional Chinese | 64.8 | 64.1 |
| C-Eval (4 categories, macro) | 78.6 | 77.9 |
| Open-Jev 1.1 test / OOD | 87.4 / 86.6 | 87.4 / 86.6 |
| Prompt injection: answers changed (lower is better) |
Web operation (Mind2Web official test splits, evaluation only)
Table with columns: split, Gen2 element selection, Gen2 operation (balanced), Gen1 element selection, Gen1 operation (balanced)| split | Gen2 element selection | Gen2 operation (balanced) | Gen1 element selection | Gen1 operation (balanced) |
|---|
| test_task (websites seen in training, new tasks) | 92.9 | 97.0 | 93.1 | 96.6 |
| test_website (new websites) | 90.8 | 97.3 | 90.7 | 96.5 |
| test_domain (new domains) | 91.7 | 97.4 | 91.5 |
- About the same as Gen1; the differences are within noise.
- So far this is only used to explore the model in RPA-style automation and to check that it generalises to some degree.
- The task is to pick the target element among it and up to 5 other candidates sampled from the page. This is easier
than the original Mind2Web protocol, so do not compare it with the Mind2Web leaderboard.
Game harness (KOF '98)
- Setup: single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as Gen1's round
robin.
- Result: 14-3-1 against Jev 1.13, 13-5 against Gen1 and 11-7 against StartLux-Decision-4B; 38-15-1 overall, score
0.713 [0.58, 0.82].
- Play style: much less reliance on the 623C anti-air uppercut, which Gen2 uses 24% of the time (Gen1 about 55% in
its round robin), with a more varied move mix. It picks moves by distance: uppercut and heavy punch up close,
special_2 and heavy kick at mid range, and almost only special_2 from far away.
- Caveats:
- 18 games per pair give wide intervals. Gen1's 17-0-1 against Jev in its round robin and Gen2's 14-3-1 here are not
significantly different.
- 36 of the 54 games ended at time-out, so most wins are on remaining health rather than by K.O.
- This is a measured result in this harness, not general fighting-game skill.
Calibration
Computed on the 0.3 public suite with the method the Decision Index board describes: each field is right or wrong,
confidence is the probability on the chosen option, benchmarks are weighted equally, and there are 10 equal-width bins.
Table with columns: metric, Gen2, Gen1, board median (112 models), Jev 1.13| metric | Gen2 | Gen1 | board median (112 models) | Jev 1.13 |
|---|
| ECE (lower is better) | 0.06-0.07 | 0.08-0.09 | 0.084 | 0.074 |
| Brier score (lower is better) | 0.31-0.33 | 0.34-0.36 | 0.49 | 0.36 |
| Confidence ≥95% but wrong | 2.2% | 3.3% | 1.3% | 2.1% |
- Why ranges: the board does not say which 32 benchmarks it uses. We checked our computation against three board
models whose results are public. Our ECE was within about 0.015 of the board's, so we give ranges over the plausible
benchmark sets.
- Still overconfident overall: mean confidence is about 7 points above accuracy. Most of the gap is on hard
reasoning (HLE, CRUXEval), humour (Humicroedit) and colour judgements (cfcolor).
- Recommendation: if you act on probability thresholds, fit a temperature on your own data.
Disclosures
- Decision Index format training data: Gen2 averages models from two training runs. About 25% of mix E and about
31% of mix F is in Decision Index request format. This includes train splits of public datasets rewritten in that
format (for example GSM8K, WinoGrande XL and ACOS), and synthetic items written in the same format. No Decision Index
test item is included.
- Decontamination:
- Every component of a training row of at least 60 characters (a line or paragraph) was checked against the
components of every Decision Index row and all of our evaluation sets.
- Rows with a component identical to an evaluation component, or with the same first 50 characters, were removed.
Text that appears in more than 4 evaluation items counts as boilerplate and is not matched.
- None of the ~230k rows new in mix F was flagged; the restored Jev-style data comes from an earlier mix that had
already been decontaminated.
- Selection on evaluations: of the 7 averaging variants, we picked the one with the highest Decision Index 0.3
public score that was also at or above Gen1 on all of our own evaluations.
- Game data: labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation
states.
- No model outputs as labels: no output of Jev or any other decision model was used as a training label.
- Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
not all were checked by a human.
Limitations
- Knowledge-heavy reasoning: weaker than larger models. The knowledge area is 42.7 (0.3); MMLU-Pro and HLE are
still close to the base model.
- Math: answered without reasoning. The rebuilt 0.3 GSM8K scores 58.0; the gain comes from training on GSM8K train
rewritten in the same format.
- Prompt injection: an instruction planted in the state changes the answer about 4% of the time on our injection
set. Keep untrusted text clearly delimited.
- Calibration: overconfident overall; see Calibration.
- Private tests: part of our public-suite advantage comes from adapting to the question formats. About two thirds of
Gen2's gain over Gen1 comes from three benchmarks whose train splits were trained on, so the 0.3 private tests may
score lower.
Citation
@misc{knowline4bgen2, title = {KnowLine-4B-Gen2: a 4B decision model}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-4B-Gen2}}
License
- Weights: Apache-2.0, the same as the base model.
- Code:
knowline_server.py is MIT (see the file header).