What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no generated text;
- the output is the probability of each option's label token;
- existing Jev clients only need a new base URL.
Table | |
|---|
| Base model | Qwen/Qwen3.5-4B (Apache-2.0) |
| Training | LoRA SFT (rank 32, alpha 64) on the language model, one epoch. |
| Release format | LoRA merged into the base; bf16 weights. The vision tower and MTP head are unchanged. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-4B-Gen1 0 8080 # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Without SGLang:
python knowline_server.py --model PelaAI/KnowLine-4B-Gen1 --backend hf --port 8080 uses
transformers only. It is slower and runs bf16.
- Front end:
knowline_server.py is a single file and needs only transformers and requests.
- Full settings: the exact settings of our Decision Index runs are in INFERENCE.md.
Comparison
Decision Index 0.3, public suite. The full 0.3 score adds private tests that only the maintainers run (0.5 same-skill,
0.3 new-domain); ours is not available yet.
Table with columns: model, size, DI 0.3 public, DI 0.3 full, source| model | size | DI 0.3 public | DI 0.3 full | source |
|---|
| KnowLine-4B-Gen1 | 4B | 60.47 | not yet scored | self-run, official kit |
| Clef | 27B | 61.71 | 53.08 | board |
| Jev 1.13 | (API) | 57.96 | 60.11 | board |
| RSI-Jev v6.1-VL | 4B | 50.98 | not listed | self-reported |
| ezjev 4B s2 | 4B | 50.82 | 46.95 | board |
| jiwo 4B | 4B | 45.76 | 42.86 | board |
| Nox 4B | 4B | 44.21 | 44.95 | board |
Our public-suite scores by area:
Table with columns: area, KnowLine-4B-Gen1 (0.3 public), Jev 1.13 (0.2.1)| area | KnowLine-4B-Gen1 (0.3 public) | Jev 1.13 (0.2.1) |
|---|
| Knowledge & reasoning | 37.7 | 51.4 |
| Language | 60.8 | 62.0 |
| Retrieval & routing | 70.9 | 55.4 |
| Tools & agents | 86.8 | 75.1 |
| Arts & taste | 49.5 | 37.7 |
Releases
New versions will follow.
Table with columns: model, date, DI 0.3 public, notes| model | date | DI 0.3 public | notes |
|---|
| KnowLine-4B-Gen4 | 2026-10-10 | 64.90 | fourth release |
| KnowLine-4B-Gen3 | 2026-10-09 | 63.11 | third release |
| KnowLine-4B-Gen2 | 2026-10-08 | 62.54 | second release |
| KnowLine-4B-Gen1 (this model) | 2026-10-07 | 60.47 | first release |
Evaluation
All results are self-run and not verified by a third party.
Other public benchmarks
Table with columns: suite, accuracy| suite | accuracy |
|---|
| C-Eval (4 categories, macro) | 77.9 |
Web operation (Mind2Web official test splits)
Table with columns: split, element selection, operation (balanced)| split | element selection | operation (balanced) |
|---|
| test_task (Cross-Task) | 93.1 | 96.6 |
| test_website (Cross-Website) | 90.7 | 96.5 |
| test_domain (Cross-Domain) | 91.5 | 98.5 |
- So far this is only used to explore the model in RPA-style automation and to check that it generalises to some degree.
- The task is to pick the target element among it and up to 5 other candidates sampled from the page. This is easier
than the original Mind2Web protocol, so do not compare it with the Mind2Web leaderboard.
Disclosures
- Decontamination:
- Every component text (state, instructions, option texts) was checked against every Decision Index row and the public
benchmarks we report.
- Rows that are identical, or share any run of 50 consecutive characters, were removed.
- No model outputs as labels: no output of Jev or any other decision model was used as a training label.
- Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
not all were checked by a human.
- Public suite: the Decision Index public suite is public; the private tests may show a lower score.
Limitations
- Knowledge-heavy reasoning: weaker than larger models. The knowledge area is 37.7 (0.3); MMLU-Pro, GPQA and HLE are
close to the base model.
- Math: answered without reasoning. The rebuilt 0.3 GSM8K scores 29.5.
- Prompt injection: an instruction planted in the state can change the answer. Keep untrusted text clearly
delimited.
- Calibration: for score questions, fit a temperature on your own data. On the Decision Index 0.3 public suite,
computed the board's way, ECE is about 0.08-0.09 and 3.3% of answers are wrong at ≥95% confidence (board median
1.3%): the model is overconfident overall.
- Private tests: the 0.3 private tests may score lower than the public suite.
Citation
@misc{knowline4bgen1, title = {KnowLine-4B-Gen1: a 4B decision model}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-4B-Gen1}}
License
- Weights: Apache-2.0, the same as the base model.
- Code:
knowline_server.py is MIT (see the file header).