What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of KnowLine-4B only need to change the model name,
because the interface is the same.
Table | |
|---|
| Base model | Qwen/Qwen3.5-35B-A3B (Apache-2.0): mixture of experts, 35B parameters in total, about 3B active per token |
| Training | LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16 weights |
| Release format | bf16 weights (about 67 GB), with a readout temperature T = 1.20 folded into the final norm weight (see Calibration). The vision tower and MTP head are unchanged from the base model; the tokenizer is the same as in KnowLine-4B. |
| Serving | the same serve_knowline.sh as KnowLine-4B: SGLang converts the weights to FP8 at load (about 35 GB). We ran every evaluation with one server on one 96 GB H20; plan for at least 48 GB of GPU memory. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-35B-A3B-Gen1 0 8080 # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Front end:
knowline_server.py is a single file that depends only on transformers and requests. It is the same
code as in KnowLine-4B-Gen4.
- Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.
How Gen1 was made
- Base model: Qwen/Qwen3.5-35B-A3B (Apache-2.0).
- Training: LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16
weights. Training examples include decision tasks written and labelled by LLM teachers.
- Readout temperature: a single temperature T = 1.20, fitted on held-out data, is folded into the weights. It
changes the size of the probabilities, but not which option ranks first.
The training data is not included in this repository.
Comparison
Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers
run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.
Table with columns: model, size, DI 0.3 public, DI 0.3 full, source| model | size | DI 0.3 public | DI 0.3 full | source |
|---|
| KnowLine-35B-A3B-Gen1 | 35B-A3B | 69.49 | not yet scored | self-run, official kit |
| KnowLine-4B-Gen4 | 4B | 64.90 | not yet scored | self-run, official kit |
| RSI-Jev v6.1 VL 27B (effort auto) | 27B | 65.64 | 58.66 | board |
| vLLM-SR d3 | 27B | 65.45 | 66.26 | board |
| Torchcast Decision 27B | 27B | 65.10 | 59.91 | board |
| Perplexity Decider v1.1 | 27B | 62.25 | 62.75 | board |
| Clef | 27B | 61.71 | 53.08 | board |
| Jev 1.13 | (API) | 57.96 | 60.11 | board |
| Cloudflare clef-omni | 30B-A3B | 56.54 | 48.28 | board |
| JPT-35B-A3B | 35B-A3B | 52.68 | 51.41 | board |
| decisio 0.4.0 | 35B-A3B | 48.43 | 47.72 | board |
Board rows as of the board's latest update (2026-10-13).
Public-suite scores by area (DI 0.3):
Table with columns: area, KnowLine-35B-A3B-Gen1, KnowLine-4B-Gen4, Perplexity Decider v1.1 (27B), Jev 1.13| area | KnowLine-35B-A3B-Gen1 | KnowLine-4B-Gen4 | Perplexity Decider v1.1 (27B) | Jev 1.13 |
|---|
| Knowledge & reasoning | 52.5 | 44.6 | 52.0 | 53.9 |
| Language | 71.2 | 65.5 | 67.5 | 59.2 |
| Retrieval & routing | 75.2 | 72.5 | 61.3 | 55.4 |
| Tools & agents | 88.1 | 87.6 | 78.9 | 75.1 |
| Arts & taste | 63.4 | 59.2 | 47.0 | 39.1 |
Releases
The model is trained in a self-evolving loop, and new versions will follow.
Table with columns: model, date, DI 0.3 public, notes| model | date | DI 0.3 public | notes |
|---|
| KnowLine-35B-A3B-Gen1 (this model) | 2026-10-12 | 69.49 | first 35B-A3B release; board figures updated 2026-10-13 |
| KnowLine-4B-Gen4 | 2026-10-10 | 64.90 | 4B, fourth release |
| KnowLine-4B-Gen3 | 2026-10-09 | 63.11 | 4B, third release |
| KnowLine-4B-Gen2 | 2026-10-08 | 62.54 | 4B, second release |
| KnowLine-4B-Gen1 | 2026-10-07 | 60.47 | 4B, first release |
Evaluation
All results are self-run and have not been verified by a third party.
Other public benchmarks
Table with columns: suite, 35B-A3B-Gen1, 4B-Gen4| suite | 35B-A3B-Gen1 | 4B-Gen4 |
|---|
| C-Eval (4 categories, macro) | 87.6 | 78.7 |
| JevBench / JevBench-hard | 85.7 / 70.3 | 86.1 / 72.1 |
In a paired sign test against Gen4, the difference on JevBench-hard is not significant.
Web operation (Mind2Web official test splits)
Table with columns: split, 35B-A3B-Gen1 element selection, 35B-A3B-Gen1 operation (balanced), 4B-Gen4 element selection, 4B-Gen4 operation (balanced)| split | 35B-A3B-Gen1 element selection | 35B-A3B-Gen1 operation (balanced) | 4B-Gen4 element selection | 4B-Gen4 operation (balanced) |
|---|
| test_task (Cross-Task) | 94.4 | 97.4 | 92.9 | 97.0 |
| test_website (Cross-Website) | 92.4 | 97.5 | 91.9 | 95.7 |
| test_domain (Cross-Domain) | 93.1 | 98.0 | 91.7 | 97.2 |
- Slightly above Gen4 on all three splits; no request failed.
- So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some
degree.
- The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
leaderboard.
Calibration
Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.
Table with columns: metric, 35B-A3B-Gen1, 4B-Gen4, board median (159 models), Jev 1.13| metric | 35B-A3B-Gen1 | 4B-Gen4 | board median (159 models) | Jev 1.13 |
|---|
| ECE (lower is better) | 0.04-0.05 | 0.04-0.05 | 0.078 | 0.074 |
| Brier score (lower is better) | 0.25-0.27 | 0.29-0.31 | 0.47 | 0.36 |
| Confidence ≥95% but wrong | 1.7% | 1.7% | 1.5% | 2.1% |
| Mean confidence / accuracy | 0.85 / 0.80 | 0.82 / 0.77 | | 0.81 / 0.74 |
- Temperature in the weights: the final norm weight includes a readout temperature T = 1.20, fitted on held-out
data, so every logit is divided by 1.20 before the softmax over the option labels. The top-ranked option does not
change, so neither do the Decision Index scores; the probabilities are just less extreme.
- Why ranges: the board does not publish which 32 benchmarks it uses. We checked our computation against three
board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the
plausible benchmark sets.
- Brier score: lower than every model on the board; the lowest there is 0.284 (Torchcast Decision 27B). Part of
this comes from higher accuracy, not only from calibration.
- Still slightly overconfident: mean confidence is about 5 points above accuracy. Most of the gap comes from HLE,
hard reasoning (MuSR, CRUXEval, GPQA Diamond), cfcolor and New Yorker caption matching.
- Recommendation: if your use depends on probability thresholds, check
calibration on your own data.
Disclosures
- Decontamination: all training data was checked with standard and strict decontamination against the Decision Index
0.3 public suite and the public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM). 10,278 rows that nearly
duplicated an evaluation item, repeated its question or claim, or contained its whole state were removed.
- Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
not all of them were checked by a human.
- Public suite: the Decision Index public suite is public; the board's 0.3 private tests may show a lower score.
Limitations
- Size and speed: it needs much more GPU memory than KnowLine-4B (at least 48 GB with FP8 at load), and each
request takes longer: 51 ms median, one request at a time on one H20, against about 31 ms for KnowLine-4B (see
INFERENCE.md).
- Knowledge-heavy reasoning: better than the 4B line but still the weakest area (52.5, DI 0.3); HLE remains at
chance level.
- Math: answers are given without step-by-step reasoning.
- Prompt injection: the model can be steered by instructions planted in the input. Keep untrusted text clearly
delimited and in data fields of the state, not in the instructions.
- Images: there was no image training; check image use on your own data before relying on it.
- Private tests: the 0.3 private tests may show a lower score than the public suite.
Citation
@misc{knowline35ba3bgen1, title = {KnowLine-35B-A3B-Gen1: a 35B-A3B decision model}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-35B-A3B-Gen1}}
License
- Weights: Apache-2.0, the same as the base model.
- Code:
knowline_server.py is MIT (see the file header).