What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of other KnowLine models only need to change the model
name, because the interface is the same.
Table | |
|---|
| Base model | google/gemma-4-26B-A4B-it (Apache-2.0): mixture of experts, about 26B parameters in total and about 4B active per token |
| Weights | byte-identical to KnowLine-26B-A4B-Gen2: LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16 weights (about 52 GB), with a readout temperature T = 1.25 folded into the final norm weight and the final logit soft-cap |
| Serving | serve_knowline.sh: SGLang in bf16, as for Gen2 (no FP8 at load for this model); knowline_server.py reads short requests twice (on by default; --no-read-twice gives the Gen2 prompts). One 80 GB GPU is enough; we ran every evaluation with one server on one 96 GB H20. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-26B-A4B-Gen2.5 0 8080 # SGLang (bf16) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Front end:
knowline_server.py with knowline_layout.py next to it; together they depend only on transformers and
requests.
- Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.
What changed in Gen2.5
- Read twice. Gen2 saw each request once: the state, then the instruction, the question and its options. Gen2.5
writes all of that a second time, introduced by the line "Read the same input again before answering:", and then
reads the answer from the label token as before. The rule is mechanical: it applies to every text-only request of at
most 3,000 tokens (94.1% of the Decision Index requests); longer requests and requests with images are sent as in
Gen2.
- A likely reason. With causal attention, the first copy of a long state is read before the model knows the
question; in the second copy every token can also attend to the first copy, question included. This fits where the
gains are: reasoning and inference over long inputs.
- No training. The weights, the temperature and the label-token readout are those of Gen2. We checked the change on
a fresh full run of the Decision Index public suite before release. It is the same
change as in KnowLine-35B-A3B-Gen1.5.
- Credit. The read-twice layout follows FlexDM (github.com/1000luca/FlexDM,
MIT); repeating the input goes back to RE2 (Xu et al., 2023,
arXiv:2309.06275).
knowline_layout.py is our own implementation.
Comparison
Scores on the Decision Index. Since Decision Index 0.4, the board's full score is 60% private tests of the same skills
and 40% private tasks from new domains; the public suite is shown but no longer counts. Gen2.5 has no board entry yet.
Table with columns: model, size, DI 0.3 public, DI 0.4.1 full, source| model | size | DI 0.3 public | DI 0.4.1 full | source |
|---|
| KnowLine-26B-A4B-Gen2.5 | 26B-A4B | 68.21 | not yet scored | self-run, official kit |
| KnowLine-26B-A4B-Gen2 (same weights) | 26B-A4B | 66.94 | 61.34 | board |
| KnowLine-35B-A3B-Gen1.5 | 35B-A3B | 70.72 | not yet scored | self-run, official kit |
| KnowLine-35B-A3B-Gen1 | 35B-A3B | 69.49 | 61.20 | board |
| Fastino GLiDE V1.2 | – | 71.99 | 71.52 | board |
| Jeffan-31b-1710 | 31B | 73.13 | 69.75 | board |
| mammoth-1 | 27B | 75.18 | 68.87 | board |
| Perplexity Decider v1.2 | 27B | 67.74 | 67.98 | board |
| StartLux-Decision v1.1 27B | 27B | 72.88 | 67.92 | board |
| FlexDM · Gemma-4-31B | 31B | 63.63 | 67.56 | board |
| Jev 1.14 | (API) | 60.70 | 60.24 | board |
Board rows from the board's data of 2026-10-21 (22:34 UTC, Decision Index 0.4.1).
Public-suite skill by area (DI 0.3, same weights):
Table with columns: area, Gen2.5, Gen2| area | Gen2.5 | Gen2 |
|---|
| Knowledge & reasoning | 55.1 | 51.9 |
| Language | 71.2 | 69.7 |
| Retrieval & routing | 70.9 | 70.4 |
| Tools & agents | 83.3 | 84.4 |
| Arts & taste | 61.4 | 60.0 |
Releases
The model is trained in a self-evolving loop, and new versions will follow.
Evaluation
All results are self-run and have not been verified by a third party. Every Gen2.5 number below was measured through the
Gen2.5 front end (read twice on). Unless noted, the Gen2 numbers were measured again on the same files.
Other public benchmarks
Table with columns: suite, 26B-A4B-Gen2.5, 26B-A4B-Gen2| suite | 26B-A4B-Gen2.5 | 26B-A4B-Gen2 |
|---|
| C-Eval (4 categories, macro) | 77.9 | 74.8 |
| JevBench / JevBench-hard | 90.5 / 81.1 | 88.3 / 76.6 |
- In a paired test against Gen2, p < 0.001 for C-Eval, p = 0.23 for JevBench and p = 0.18 for JevBench-hard.
- The Gen2 column matches the Gen2 card.
Web operation (Mind2Web official test splits)
Table with columns: split, Gen2.5 element selection, Gen2.5 operation (balanced), Gen2 element selection, Gen2 operation (balanced)| split | Gen2.5 element selection | Gen2.5 operation (balanced) | Gen2 element selection | Gen2 operation (balanced) |
|---|
| test_task (Cross-Task) | 89.8 | 90.3 | 89.5 | 91.6 |
| test_website (Cross-Website) | 89.2 | 94.0 | 88.1 | 94.1 |
| test_domain (Cross-Domain) | 89.6 | 92.9 | 88.8 | 93.4 |
-
Element selection is slightly higher than with Gen2 on all three splits (paired test p = 0.04 on test_domain, p ≥ 0.17
on the other two); operation accuracy is within noise of Gen2 (p ≥ 0.18). No request failed.
-
The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
leaderboard.
Images (zero-shot)
- Unchanged from Gen2: read twice applies only to text-only requests. A request with images is sent exactly as in
Gen2, byte for byte, to the same weights, so Gen2's image results hold for Gen2.5 and we did not rerun them.
- Gen2's result: 63.3 on the official vision public suite (self-run); see the
Gen2 card for the per-benchmark table.
Calibration
Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.
Table with columns: metric, 26B-A4B-Gen2.5, 26B-A4B-Gen2| metric | 26B-A4B-Gen2.5 | 26B-A4B-Gen2 |
|---|
| ECE (lower is better) | 0.013-0.018 | 0.008-0.012 |
| Brier score (lower is better) | 0.24-0.27 | 0.26-0.28 |
| Confidence ≥95% but wrong | 0.9% | 0.8-0.9% |
| Mean confidence / accuracy | 0.82-0.83 / 0.80-0.82 | 0.80-0.81 / 0.79-0.81 |
- Temperature in the weights: unchanged from Gen2: the final norm weight and the final logit soft-cap include
T = 1.25, so every final logit is divided by 1.25 before the softmax over the option labels.
- Why ranges: the board does not publish which benchmarks it uses for calibration, so we report ranges over the
plausible benchmark sets, as for Gen2.
- Recommendation: if your use depends on probability thresholds, check calibration on your own data.
Disclosures
- Training data: the weights are those of Gen2, so the decontamination statement on the
Gen2 card applies unchanged. The training data is not
included in this repository.
- Public suite: the Decision Index public suite is public; the board's private tests may show a lower score.
Limitations
- Slower than Gen2: about 1.54 times the prompt tokens on the Decision Index suite; one request at a time on one H20,
66.3 ms median and 151 ms mean (Gen2: 43.1 ms and 82.8 ms), and 629 ms at the 95th percentile (Gen2: 238 ms). Use
--no-read-twice if latency matters more than these gains.
- Only short requests change: text-only requests over 3,000 tokens, and requests with images, are answered exactly as
by Gen2.
- Not better everywhere: a few benchmarks, including some tool and agent tasks, are slightly lower than with Gen2.
- Everything else as Gen2: Chinese is still clearly below the 35B-A3B (C-Eval); web operation and images are below
the 35B-A3B; prompt-injection attacks may still work, so put untrusted content in data fields of the state, not in the instructions;
the board's private tests may show a lower score than the public suite.
Citation
@misc{knowline26ba4bgen25, title = {KnowLine-26B-A4B-Gen2.5: a 26B-A4B decision model served with read twice}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-26B-A4B-Gen2.5}}
License
- Weights: Apache-2.0, the same as the base model.
- Code:
knowline_server.py, knowline_layout.py and knowline_engine.py are MIT (see the file headers).