What it does
You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:
- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of KnowLine-31B-Gen1 and other KnowLine models only
need to change the model name, because the interface is the same.
Table | |
|---|
| Base model | google/gemma-4-31B-it (Apache-2.0): dense, about 31B parameters |
| Weights | byte-identical to KnowLine-31B-Gen1, a LoRA fine-tune of gemma-4-31B-it (rank 32, alpha 64, language model only, one epoch, merged into bf16 weights); see the Gen1 card for every step |
| Release format | bf16 weights (about 63 GB), with a readout temperature T = 1.44 folded into the final norm weight and the final logit soft-cap (see Calibration); the front end reads score questions at an effective temperature of 1.0 (temperatures.json). The vision tower is unchanged from the base model; the tokenizer is Gemma 4's, not the one of the Qwen-based KnowLine models. |
| Serving | serve_knowline.sh: SGLang in bf16, as for Gen1 (we did not test FP8 at load for this model); knowline_server.py reads short requests twice, as in Gen1 (on by default), and serve_knowline.sh adds --header-when-long 32768, so longer text-only requests are read once with the question header (--header-when-long 0, the default of knowline_server.py started on its own, gives the Gen1 prompts; --no-read-twice reads every request once, without the header). SGLang reserves 85% of the GPU's memory by default (MEM); the weights take about 63 GB. We ran every evaluation with one server on one 96 GB H20. |
| Languages | English, Simplified Chinese, Traditional Chinese |
Quickstart
pip install "sglang==0.5.21" "transformers==5.12.1" requestsbash serve_knowline.sh PelaAI/KnowLine-31B-Gen1.5 0 8080 # SGLang (bf16) on :9080 + /v1/systemone on :8080curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
- Front end:
knowline_server.py with knowline_layout.py next to it; together they depend only on transformers and
requests. Requests that read twice covers, and requests with images, are rendered exactly as by the KnowLine-31B-Gen1
front end, prompt for prompt; only longer text-only requests change.
- Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.
What changed in Gen1.5
- Question header for long requests. Gen1 read a text-only request of more than 3,000 tokens once: the state, then
the instruction, the question and its options. Gen1.5 puts a header before the state that lists every question of the
request, introduced by the line "Questions that will be asked about the conversation or state below:", and still reads
the request once; the answer is read from the label token as before. The rule is mechanical: it applies to every
text-only request that read twice does not cover, as long as its longest prompt with the header is at most 32,768
tokens (5.6% of the Decision Index requests). Requests of at most 3,000 tokens are read twice exactly as in Gen1
(94.1%); requests with images, and the few whose header prompt would be longer than 32,768 tokens (0.3%), are sent as
in Gen1.
- A likely reason. With causal attention, a long state read once is read before the model knows what will be asked;
with the header, the questions come first, so every token of the state can attend to them.
- No training. The weights, the temperature, the per-type temperatures and the label-token readout are those of
Gen1. We checked the change on the Decision Index public suite before release.
- Credit. The question header and the read-twice layout follow FlexDM
(github.com/1000luca/FlexDM, MIT), and Aleria Judge (github.com/ALERIA-ML/aleria-judge, Apache-2.0) uses the same header structure;
repeating the input goes back to RE2 (Xu et
al., 2023, arXiv:2309.06275).
knowline_layout.py is our own implementation.
Comparison
Scores on the Decision Index. Since Decision Index 0.4, the board's full score is 60% private tests of the same skills
and 40% private tasks from new domains; the public suite is shown but no longer counts. Gen1.5 has no board entry yet.
Table with columns: model, size, DI 0.3 public, DI 0.4.1 full, source| model | size | DI 0.3 public | DI 0.4.1 full | source |
|---|
| KnowLine-31B-Gen1.5 | 31B | 70.50 | not yet scored | self-run, official kit |
| KnowLine-31B-Gen1 (same weights) | 31B | 70.13 | 71.86 | board |
| KnowLine-35B-A3B-Gen1.5 | 35B-A3B | 70.72 | 65.77 | board |
| KnowLine-26B-A4B-Gen2.5 | 26B-A4B | 68.21 | 64.57 | board |
| Fastino GLiDE V1.2 | – | 71.99 | 71.52 | board |
| Sieve 31B v3 | 31B | 71.68 | 71.10 | board |
| Turion Jev 31B | 31B | 66.16 | 70.47 | board |
| vLLM-SR d3-ultra | 27B | 71.57 | 70.18 | board |
| Jeffan-31b-1710 | 31B | 73.13 | 69.75 | board |
| H2O Lightning 31B v1.1 | 31B | 67.75 | 68.64 | board |
| WeMM-JEV-27B | 27B | 69.37 | 68.49 | board |
| mammoth-1 | 27B | 75.34 | 68.44 | board |
| Perplexity Decider v1.2 | 27B | 67.74 | 67.98 | board |
| StartLux-Decision v1.1 27B | 27B | 72.88 | 67.92 | board |
| FlexDM · Gemma-4-31B | 31B | 63.63 | 67.56 | board |
| Jev 1.14 | (API) | 60.70 | 60.24 | board |
Board rows from the board's data of 2026-10-29 (17:22 UTC, Decision Index 0.4.1).
Public-suite skill by area (DI 0.3; KnowLine-31B-Gen1.5 with its release serving, the other three served read twice):
Table with columns: area, KnowLine-31B-Gen1.5, KnowLine-31B-Gen1, KnowLine-26B-A4B-Gen2.5, KnowLine-35B-A3B-Gen1.5| area | KnowLine-31B-Gen1.5 | KnowLine-31B-Gen1 | KnowLine-26B-A4B-Gen2.5 | KnowLine-35B-A3B-Gen1.5 |
|---|
| Knowledge & reasoning | 59.4 | 59.4 | 55.1 | 55.0 |
| Language | 72.8 | 72.0 | 71.2 | 72.9 |
| Retrieval & routing | 73.2 | 73.0 | 70.9 | 75.4 |
| Tools & agents | 85.4 | 84.7 | 83.3 | 88.2 |
| Arts & taste | 60.5 | 60.5 | 61.4 | 64.2 |
Releases
The model is trained in a self-evolving loop, and new versions will follow.
Evaluation
All results are self-run and have not been verified by a third party. The new serving changes only text-only requests
over 3,000 tokens; every other request is sent exactly as with KnowLine-31B-Gen1. JevBench was run again in full through
the Gen1.5 release server. The other sets have no request that changes, so their numbers below (C-Eval, Mind2Web and
images) are Gen1's, measured with the Gen1 serving (read twice on). The 31B-Gen1 columns repeat the Gen1 card. The
26B-A4B-Gen2.5 and 35B-A3B-Gen1.5 numbers are from their release evaluations with the read-twice front end.
Other public benchmarks
Table with columns: suite, 31B-Gen1.5, 31B-Gen1, 26B-A4B-Gen2.5, 35B-A3B-Gen1.5| suite | 31B-Gen1.5 | 31B-Gen1 | 26B-A4B-Gen2.5 | 35B-A3B-Gen1.5 |
|---|
| C-Eval (4 categories, macro) | 82.5 | 82.5 | 77.9 | 88.1 |
| JevBench / JevBench-hard | 90.5 / 80.2 | 90.5 / 80.2 | 90.5 / 81.1 | 87.0 / 73.0 |
- In a paired sign test against KnowLine-26B-A4B-Gen2.5, it is higher on C-Eval (p < 0.001); JevBench and
JevBench-hard (p = 1.0) do not differ.
- Against KnowLine-35B-A3B-Gen1.5, it is lower on C-Eval (p < 0.001); JevBench and JevBench-hard (p = 0.06) do not
differ at p < 0.05.
Web operation (Mind2Web official test splits)
Table with columns: split, 31B-Gen1 element selection, 31B-Gen1 operation (balanced), 26B-A4B-Gen2.5 element selection, 26B-A4B-Gen2.5 operation (balanced)| split | 31B-Gen1 element selection | 31B-Gen1 operation (balanced) | 26B-A4B-Gen2.5 element selection | 26B-A4B-Gen2.5 operation (balanced) |
|---|
| test_task (Cross-Task) | 91.5 | 92.8 | 89.8 | 90.3 |
| test_website (Cross-Website) | 91.5 | 95.9 | 89.2 | 94.0 |
| test_domain (Cross-Domain) | 90.4 | 96.6 | 89.6 | 92.9 |
-
Element selection is higher than with KnowLine-26B-A4B-Gen2.5 on all three splits (paired test p = 0.05, 0.01 and
0.11); operation accuracy is within noise (p ≥ 0.09). No request failed. No Mind2Web request is over 3,000 tokens, so
these results, measured with the Gen1 serving, hold for Gen1.5 unchanged.
-
The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
leaderboard.
Images (zero-shot)
- No image training: the vision tower is the base model's, unchanged, and no images were seen in training.
knowline_server.py accepts OpenAI-style image / image_url content parts in a chat state and passes them to
SGLang; each question is still one prefill. Requests with images are read once, without the question header (read
twice and the header are text-only).
- Result: 69.3 on the official vision public suite (12,210 rows, complete, 0 failed requests; self-run with the
official kit, through the Gen1 release server). A request with images is sent exactly as in Gen1, byte for byte, to
the same weights, so this result holds for Gen1.5 and we did not rerun it.
Calibration
Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.
Table with columns: metric, 31B-Gen1.5, 31B-Gen1, 26B-A4B-Gen2.5, 35B-A3B-Gen1.5| metric | 31B-Gen1.5 | 31B-Gen1 | 26B-A4B-Gen2.5 | 35B-A3B-Gen1.5 |
|---|
| ECE (lower is better) | 0.013-0.022 | 0.012-0.021 | 0.013-0.018 | 0.03-0.04 |
| Brier score (lower is better) | 0.23-0.26 | 0.23-0.26 | 0.24-0.27 | 0.24-0.26 |
| Confidence ≥95% but wrong | 1.0-1.1% | 0.9-1.0% | 0.9% | 1.5% |
| Mean confidence / accuracy | 0.84 / 0.81-0.83 | 0.83-0.84 / 0.81-0.83 | 0.82-0.83 / 0.80-0.82 | 0.85-0.86 / 0.81-0.83 |
- Temperature in the weights: unchanged from Gen1: the final norm weight and the final logit soft-cap include a
readout temperature T = 1.44, fitted on held-out data with the read-twice serving, so every final logit is divided by
1.44 before the softmax over the option labels. The top-ranked option does not change.
- Score questions at temperature 1.0 (as in Gen1):
serve_knowline.sh passes temperatures.json
({"score": 0.6944}), so the front end divides the logits of score (rubric) questions by a further 0.6944 and reads
them at an effective temperature of 1.44 × 0.6944 = 1.0; yes/no and choice questions stay at 1.44. Score questions
are often read as an expected value over the scale, which a higher temperature pulls toward the middle. This is a
serving setting, not a weight change, and the KnowLine releases before Gen1 did not use it; starting
knowline_server.py without --temperatures (or knowline_engine.py with --option temperatures=none) reads every
question at 1.44.
- Why ranges: the board does not publish which benchmarks it uses for calibration, so we report ranges over the
plausible benchmark sets, as for the earlier releases.
- Recommendation: if your use depends on probability thresholds, check
calibration on your own data.
Disclosures
- Training data: the weights are those of Gen1, so the decontamination statement on the
Gen1 card applies unchanged. The training data is not
included in this repository.
- Public suite: the Decision Index public suite is public; the board's private tests may show a lower score.
Limitations
- Size and speed: about 63 GB of bf16 weights and dense, so every token uses all 31B parameters; we have only run it
on 96 GB GPUs. One request at a time on one H20, over 700 Decision Index rows sampled in the style of the board's
latency method, requests take 330 ms median and 1,135 ms mean (4,241 ms at the 95th percentile);
KnowLine-26B-A4B-Gen2.5 took 66.3 ms and 151 ms on an earlier sample of 300 rows; see INFERENCE.md.
- Read twice and the question header: text-only requests up to 3,000 tokens are written twice, about 1.54 times the
prompt tokens on the Decision Index suite. Longer ones carry the question header in every question's prompt, so the
summed prompt tokens of the suite are 896 million (323 million with the Gen1 serving); with SGLang's prefix cache,
which prefills the header and the state once per request, the prefilled tokens grow by about 5%. Use
--no-read-twice
if latency matters more (it also turns the header off); we did not evaluate this model without read twice for this
card.
- Planted instructions: this model can be steered by instructions planted in the input; put untrusted content in
data fields of the state, not in the instructions, and keep it clearly delimited.
- C-Eval: below KnowLine-35B-A3B-Gen1.5 (82.5 against 88.1).
- Calibration: probabilities are calibrated on average, not for every domain; check calibration on your own data.
- Images: there was no image training, so check image use on your own data.
- Other attacks: other prompt-injection attacks may still work.
- Private tests: the board's private tests may show a lower score than the public suite.
Citation
@misc{knowline31bgen15, title = {KnowLine-31B-Gen1.5: a 31B decision model served with read twice and a question header}, author = {PelaAI}, year = {2026}, url = {https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5}}
License
- Weights: Apache-2.0, the same as the base model.
- Base model licence: google/gemma-4-31B-it is released under Apache-2.0; see Google's licence page,
ai.google.dev/gemma/docs/gemma_4_license.
- Code:
knowline_server.py, knowline_layout.py and knowline_engine.py are MIT (see the file headers).