What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of KnowLine-4B only need to change the model name,
because the interface is the same.
Table | |
|---|
| Base model | Qwen/Qwen3.5-35B-A3B (Apache-2.0): mixture of experts, 35B parameters in total, about 3B active per token |
| Training | LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16 weights |
| Release format | bf16 weights (about 67 GB), with a readout temperature T = 1.20 folded into the final norm weight (see Calibration). The vision tower and MTP head are unchanged from the base model; the tokenizer is the same as in KnowLine-4B. |
| Serving | the same serve_knowline.sh as KnowLine-4B: SGLang converts the weights to FP8 at load (about 35 GB). We ran every evaluation with one server on one 96 GB H20; plan for at least 48 GB of GPU memory. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-35B-A3B-Gen1 0 8080 # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Front end:
knowline_server.py is a single file that depends only on transformers and requests. It is the same
code as in KnowLine-4B-Gen4.
- Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.
How Gen1 was made
-
Why a larger base? The team trained the KnowLine-4B series, most recently updated to
KnowLine-4B-Gen4. Over four rounds, it gained most on retrieval,
tools and taste, while knowledge and reasoning stayed the weakest area (44.6 in Gen4). The agent trained the same data
on a larger base to see how much of that gap comes from model size.
-
Data. Two rounds:
- all training data of the KnowLine-4B line up to Gen4, in one set;
- a second round of inputs no earlier round used: more rows from the same public train sets, format variants of
existing items, about 40 new task families from domains not covered before, and synthetic scenarios, plus a replay
share of the first round (arithmetic, knowledge, Jev-style and injection-invariance items at a higher rate).
All data went through the same decontamination check as Gen4; see Disclosures.
-
Training. The base model was trained on the first round. The result was then trained further on the second round,
and the two were averaged 50/50. A single temperature was fitted and folded into the weights. It changes the size of
the probabilities, but not which option ranks first.
-
We compared 10 candidates (checkpoints of both rounds and two averages), with Decision Index 0.3
public scores between 68.09 and 69.60. Every candidate had to pass the same checks as Gen4, against
KnowLine-4B-Gen4 and against the first-round model: no significant drop on our held-out sets, Open-Jev 1.1 OOD or
JevBench-hard; an injection rate of at most 4%; and no regression beyond set margins on new-domain tasks, sibling
skills, format consistency, unseen injection templates or calibration (measured after applying the temperature).
Gen1 has the highest public score among the candidates that passed: 69.60 before the temperature was folded in, and
69.49 in the fresh run of the released weights (the temperature does not change the ranking, so the difference is
run-to-run noise).
Where the gain over Gen4 comes from
Gen1 and Gen4 were trained on largely the same kinds of tasks. For 33 of the 37 benchmarks, the training data holds
items of the same task or from the same source; only MMLU-Pro, BBH, GPQA Diamond and HLE have none. Most of the 4.59
points therefore come from the larger base model rather than from new benchmark data:
- About 3.9 points come from the 33 benchmarks with training data, and about 0.7 from the four without. Per benchmark,
the four without training data gained more on average.
- Largest gains, skill Gen4 → Gen1: iSarcasmEval 40.3 → 64.2, CRUXEval 43.8 → 63.3, MMLU-Pro 51.6 → 66.6 (no training
data), MMLU 69.9 → 82.2, BBH 53.1 → 64.9 (no training data), ANLI 51.7 → 62.4 and New Yorker caption matching
64.5 → 73.0.
- The largest drops are small: POP909-CL (85.0 → 83.4) and Habermas Machine (48.1 → 46.8).
Comparison
Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers
run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.
Table with columns: model, size, DI 0.3 public, DI 0.3 full, source| model | size | DI 0.3 public | DI 0.3 full | source |
|---|
| KnowLine-35B-A3B-Gen1 | 35B-A3B | 69.49 | not yet scored | self-run, official kit |
| KnowLine-4B-Gen4 | 4B | 64.90 | not yet scored | self-run, official kit |
| RSI-Jev v6.1 VL 27B (effort auto) | 27B | 65.64 | 58.66 | board |
Board rows as of the board's latest update (2026-10-13).
Public-suite scores by area (DI 0.3):
Table with columns: area, KnowLine-35B-A3B-Gen1, KnowLine-4B-Gen4, Perplexity Decider v1.1 (27B), Jev 1.13| area | KnowLine-35B-A3B-Gen1 | KnowLine-4B-Gen4 | Perplexity Decider v1.1 (27B) | Jev 1.13 |
|---|
| Knowledge & reasoning | 52.5 | 44.6 | 52.0 | 53.9 |
| Language | 71.2 | 65.5 | 67.5 | 59.2 |
| Retrieval & routing | 75.2 | 72.5 | 61.3 | 55.4 |
Releases
The model is trained in a self-evolving loop, and new versions will follow.
Table with columns: model, date, DI 0.3 public, golden held-out (en / zh-Hans / zh-Hant), notes| model | date | DI 0.3 public | golden held-out (en / zh-Hans / zh-Hant) | notes |
|---|
| KnowLine-35B-A3B-Gen1 (this model) | 2026-10-12 | 69.49 | 70.3 / 73.8 / 72.1 | first 35B-A3B release; board figures updated 2026-10-13 |
| KnowLine-4B-Gen4 | 2026-10-10 | 64.90 | 69.7 / 70.1 / 65.1 | 4B, fourth release |
| KnowLine-4B-Gen3 |
Evaluation
All results are self-run and have not been verified by a third party.
Held-out and Chinese evaluations
Table with columns: suite, 35B-A3B-Gen1, 4B-Gen4| suite | 35B-A3B-Gen1 | 4B-Gen4 |
|---|
| In-house evaluation set, English | 70.3 | 69.7 |
| In-house evaluation set, Simplified Chinese | 73.8 | 70.1 |
| In-house evaluation set, Traditional Chinese | 72.1 | 65.1 |
| C-Eval (4 categories, macro) | 87.6 | 78.7 |
| Open-Jev 1.1 test / OOD | 91.0 / 90.4 | 88.4 / 87.1 |
| Decision tasks from new domains (our own) |
In a paired sign test against Gen4, the gains on the Simplified and Traditional Chinese in-house sets and on Open-Jev
1.1 OOD are significant (p < 0.001); the differences on the English in-house set and on JevBench-hard are not.
Web operation (Mind2Web official test splits, evaluation only)
Table with columns: split, 35B-A3B-Gen1 element selection, 35B-A3B-Gen1 operation (balanced), 4B-Gen4 element selection, 4B-Gen4 operation (balanced)| split | 35B-A3B-Gen1 element selection | 35B-A3B-Gen1 operation (balanced) | 4B-Gen4 element selection | 4B-Gen4 operation (balanced) |
|---|
| test_task (websites seen in training, new tasks) | 94.4 | 97.4 | 92.9 | 97.0 |
| test_website (new websites) | 92.4 | 97.5 | 91.9 | 95.7 |
| test_domain (new domains) | 93.1 | 98.0 | 91.7 |
- Slightly above Gen4 on all three splits; no request failed.
- So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some
degree.
- The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
leaderboard.
Images (zero-shot)
- No image training: Gen1 keeps the base model's vision tower unchanged and saw no images in training.
knowline_server.py accepts OpenAI-style image / image_url content parts in a chat state and passes them to
SGLang; each question is still one prefill.
- How we measured it: the same reconstruction of the Decision Index 0.3.1 vision public suite as for Gen4: 11
benchmarks, the board's row counts where the sources allow, images capped at 1.6 MP, chance-corrected skill and the
board's weights, run through the release server: 12,192 requests, 0 failed.
Table with columns: benchmark, 35B-A3B-Gen1, stock Qwen3.5-35B-A3B (same front end), 4B-Gen4| benchmark | 35B-A3B-Gen1 | stock Qwen3.5-35B-A3B (same front end) | 4B-Gen4 |
|---|
| CV-Bench | 75.8 | 76.4 | 75.7 |
| BLINK | 59.8 | 66.4 | 58.1 |
| RealWorldQA | 71.9 | 64.7 | 64.7 |
| CharXiv | 66.9 | 56.3 | 66.3 |
- Mixed against the stock model: 3.4 points higher overall. The gains are on charts, infographics, web screenshots,
exam pages and RealWorldQA; BLINK (multi-image perception, −6.6), Hateful Memes, Winoground, KIE and CV-Bench are
slightly lower.
- Indicative only: these are self-run numbers on our reconstruction, not on the board's rows. Our CharXiv distractors
lean towards numeric questions, and our Mind2Web marks and KIE questions are our own construction.
Game harness (KOF '98)
- Setup: single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for the 4B
line.
- Result: 17-0-1 against Jev 1.13, 18-0-0 against StartLux-Decision-35B-A3B and 12-4-2 against KnowLine-4B-Gen4;
47-4-3 overall, a score of 0.898 [0.79, 0.95].
- Play style: it chooses moves by distance: at close range mostly the 623C anti-air uppercut (45%) and heavy punch
(41%), special_2 at mid range (68%), and mostly waiting at long range (64%). Overall, it uses special_2 36% of the
time, the uppercut 16%, heavy punch 14%, special_3 13% and waiting 13%, a more varied mix than Gen4's.
- Caveats:
- 18 games per pair give wide intervals.
- 27 of the 54 games ended at time-out, so those were decided on remaining health. Gen1 won 25 games by K.O. (13
against StartLux, 9 against Gen4, 3 against Jev); its 2 K.O. losses were both against Gen4. Against Jev, 14 of the
17 wins came at time-out.
- This is a measured result in this harness, not general fighting-game skill.
Calibration
Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.
Table with columns: metric, 35B-A3B-Gen1, 4B-Gen4, board median (159 models), Jev 1.13| metric | 35B-A3B-Gen1 | 4B-Gen4 | board median (159 models) | Jev 1.13 |
|---|
| ECE (lower is better) | 0.04-0.05 | 0.04-0.05 | 0.078 | 0.074 |
| Brier score (lower is better) | 0.25-0.27 | 0.29-0.31 | 0.47 | 0.36 |
| Confidence ≥95% but wrong | 1.7% | 1.7% | 1.5% | 2.1% |
- Temperature in the weights: the final norm weight includes a readout temperature T = 1.20, so every logit is
divided by 1.20 before the softmax over the option labels. The top-ranked option does not change, so neither do the
Decision Index scores; the probabilities are just less extreme. T was fitted on a set of our own evaluation items
whose distribution is covered by the training data, not on the held-out sets or on Decision Index rows. On our
held-out sets, ECE with the temperature in the weights is 0.025 (Gen4: 0.043).
- Why ranges: the board does not publish which 32 benchmarks it uses. We checked our computation against three
board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the
plausible benchmark sets.
- Brier score: lower than every model on the board; the lowest there is 0.284 (Torchcast Decision 27B). Part of
this comes from higher accuracy, not only from calibration.
- Still slightly overconfident: mean confidence is about 5 points above accuracy. Most of the gap comes from HLE,
hard reasoning (MuSR, CRUXEval, GPQA Diamond), cfcolor and New Yorker caption matching.
- Recommendation: the temperature was fitted on our data. If your use depends on probability thresholds, check
calibration on your own data.
Disclosures
- Decision Index format training data: in the first round, about 21% of the rows were built in the request format
of Decision Index benchmarks (some with reworded instructions), and another 4% are copies of the same items rendered in
other formats; in the second round, about 8% and 9%. These include train splits of public datasets rewritten in that format (for example GSM8K,
WinoGrande XL, ACOS and Amazon ESCI) and synthetic items written in the same format.
- Decontamination:
- All training data, of both rounds, was checked with the same tool as Gen4 against every Decision Index 0.3 row, all
of our evaluation sets and the other public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM).
- Rows that nearly duplicate an evaluation item, repeat its question or claim, or contain its whole state were
removed: 7,833 of 1,500,018 rows in the first round and 2,445 of 985,519 in the second. Rows that only share a
source passage with an evaluation item, such as the same Wikipedia paragraph or tool description, were kept (31,249
and 5,702 rows).
- BPoMP and cfcolor training items come from the same source datasets as these benchmarks (OEDILF limericks; the
O'Donovan et al. colour ratings), with every benchmark item excluded, as in Gen4.
- Selection on evaluations: of the 10 candidates, we picked the one with the highest Decision Index 0.3 public
score among those that passed all of our release checks.
- Game data: labels come from simulator rollouts or engines. Seeds and start states are disjoint from the evaluation
states.
- No model outputs as labels: no output of Jev or any other decision model was used as a training label.
Limitations
- Size and speed: it needs much more GPU memory than KnowLine-4B (at least 48 GB with FP8 at load), and each
request takes longer: 51 ms median, one request at a time on one H20, against about 31 ms for KnowLine-4B (see
INFERENCE.md).
- Knowledge-heavy reasoning: better than the 4B line but still the weakest area (52.5, DI 0.3); HLE remains at
chance level.
- Math: answers are given without step-by-step reasoning. The rebuilt 0.3 GSM8K scores 69.3; part of this comes
from training on the GSM8K train split rewritten in the same format.
- Prompt injection: on our injection set, planted instructions changed no answers, and about 1% with templates not
seen in training. Other attacks may still work, so keep untrusted text clearly delimited.
- Images: there was no image training. The image scores above are zero-shot and self-run on our own reconstruction of
the suite, so check them on your own data before relying on them.
- Private tests: part of the public-suite score comes from benchmarks whose tasks are in our training data, so the
0.3 private tests may show a lower score.
Citation
@misc{knowline35ba3bgen1, title = {KnowLine-35B-A3B-Gen1: a 35B-A3B decision model}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-35B-A3B-Gen1}}
License
- Weights: Apache-2.0, the same as the base model.
- Code:
knowline_server.py is MIT (see the file header).