tkm-research

tkm-decision-v0.1

README

License: apache-2.0

How a decision is made

Prompt. The state (strings as-is, structured states as compact JSON) is followed by the question's instructions and its options, lettered A., B., … in the order the request lists them, and the line "Answer with the letter only." It goes through the Gemma chat template with thinking off. Choice options show label: description; noul is posed as A. Yes / B. No (with the request's true/false criteria text); score levels are lettered in level order.
Truncation. States longer than 5,000 tokens keep their first 2/3 and last 1/3 of that budget.
Readout. One forward pass. For each option letter the next-token probability of X and X is summed; these are renormalised over the options. Output tokens: 0. usage.input_tokens is the exact prompt length.
Calibration (jevdec/shim/calib.json, fixed in the code):
- choice: softmax(log p / 0.74)
- noul: P(yes) = σ((log p_yes − log p_no − 1.5) / 0.38)
- score: softmax(log p) (identity); the reported score is the expected level.
The calibration code also supports a temperature that grows when the model puts little total probability on the option letters (a_* slopes, computed only from the model's own next-token distribution). It helped the untrained base model; for the shipped fine-tuned model the slopes are 0.

Local results (not board scores)

Scored with a local replica of the JevBench v1.6.1 scorer (jevdec/jevscore.py). The board measures 1,500 decisions, most of them on sealed items, so expect different numbers there.

Table with columns: Set, Model, Capability, Notes
Set	Model	Capability	Notes
231 public JevBench items	untrained base, same readout	73.96	5-fold out-of-fold calibration
231 public JevBench items	this model	84.92	5-fold out-of-fold calibration
231 public JevBench items	this model	86.08	calibration fitted on these items (shipped constants)
Open-Jev-v1.1 test + ood (6,000 items)	untrained base	63.74

Serving (RTX PRO 6000, bf16, serial, after warm-up, 231 public items): p50 0.113 s, p95 0.98 s; mean 635 input tokens per decision; zero output tokens. See docs/results.md for every run, checkpoint and caveat.

Data and disclosures

Training. Four LoRA runs (rank 16, all language-model linear layers; KL weight 1/3/1/2; 150/150/150/300 steps of 16 examples, i.e. 2,400–4,800 examples each, drawn from a pool of 12,000 Open-Jev-v1.1 train items stratified by type; B1 and B2 share a data order) were trained with a soft-target loss over the option letters, a KL term to the frozen base model's own letter distribution, and an ordinal loss on score items, with option order shuffled per example. Open-Jev-v1.1 (ZefanCai/Open-Jev-v1.1) is procedurally generated typed decisions (generated content CC0-1.0; WANLI-derived records CC-BY-4.0). The shipped adapter is the exact average of the four runs' final LoRA deltas (stacked into one rank-64 adapter, safetensors), applied unmerged: merging into bf16 weights lost accuracy to rounding. A fifth run that mixed in natural-language sources (BoolQ, WANLI, CommonsenseQA, ARC, MMLU auxiliary-train, SST-5, prompt-injections, Banking77) was trained, evaluated and dropped. An 8-gram overlap check of the Open-Jev pool against the 231 public JevBench items found no overlap. No JevBench item was used for training.
Use of the public JevBench items. The 231 public items from the jevbench repository (datasets/public/*.jsonl) were used for development decisions: comparing the untrained model with the fine-tuned runs (the ship decision), dropping the mixed-data run, leaving the score calibration at identity, choosing the unmerged adapter over merged weights, and fitting the three calibration constants above (choice temperature on hard-tier choice items; noul temperature and offset), with 5-fold cross-validation to check stability. Open-Jev-v1.1 calibration, test and ood splits were used for development checks. No rule in the code refers to any item.
Prompt wording was fixed before evaluation; two answer-prefix variants ("Answer: ", "The answer is ") were tried on the public hard items and rejected.
The calibration file records the sha256 of the adapter it was fitted for; the shim refuses to start with a different adapter or without one.

Running

BASE is a local copy of google/gemma-4-31B-it at revision 842da3794eaa0b77d5f08bae87a17459d91ff475; ADAPTER is a local copy of this repository (the adapter files sit at its root).

bash

hf download google/gemma-4-31B-it --revision 842da3794eaa0b77d5f08bae87a17459d91ff475 --local-dir gemma-4-31B-ithf download tkm-research/tkm-decision-v0.1 --local-dir tkm-decision-v0.1      # add --revision <commit> to pin a releaseBASE=$PWD/gemma-4-31B-it ADAPTER=$PWD/tkm-decision-v0.1 # docker (offline; weights mounted read-only)docker build -t jevdec "$ADAPTER"docker run --gpus '"device=0"' --network none -v "$BASE":/model:ro -v "$ADAPTER":/adapter:ro -p 8000:8000 jevdeccurl -s localhost:8000/v1/systemone -H 'content-type: application/json' \  -d '{"state": "Order #1182 shipped.", "questions": {"q": {"type": "noul", "instructions": "Shipped?"}}}' # HTTP without docker (POST /v1/systemone, GET /health), from inside $ADAPTERpython -m jevdec.shim.server --model-dir "$BASE" --adapter-dir "$ADAPTER" --port 8000 # in-process, from inside $ADAPTERpython -c "from jevdec.shim import loader; m = loader('$BASE', adapter_dir='$ADAPTER'); \  print(m.predict('Order #1182 shipped.', {'decision': {'type': 'noul', 'instructions': 'Shipped?'}}))"

Requirements: one GPU with ≥ 80 GB (bf16 base ≈ 62 GB + adapter ≈ 2 GB). The Docker image installs transformers==5.19.0, accelerate==1.15.0 and peft==0.21.2 on a digest-pinned PyTorch 2.14.1 / CUDA 13.0 image; trust_remote_code is not used. Questions with more than 26 options get HTTP 422.

Repository contents and licences

Serving code: jevdec/prompt.py, jevdec/readout.py, jevdec/postprocess.py, jevdec/shim/ — the only modules in the Docker image. Everything else is development tooling and is never imported at serve time: the v1.6.1 scorer replica (jevdec/jevscore.py), item loaders (jevdec/items.py), calibration fitting (jevdec/calib.py), training (jevdec/train/), and scripts/.
The evaluation sets are not included. scripts/build_devsets.py rebuilds them from a clone of https://github.com/fstandhartinger/jevbench (MIT License, Copyright (c) 2026 Florian Standhartinger and contributors) and from ZefanCai/Open-Jev-v1.1 on the Hugging Face Hub.
holds four published aggregate rows of the JevBench v1.6.1 results.

Development

jevdec/jevscore.py is a local replica of the JevBench v1.6.1 scorer (its aggregation reproduces published rows exactly; see tests/test_jevscore.py). Scripts in scripts/ build dev sets, cache readouts, fit/evaluate calibration and measure latency. Tests: python -m pytest -q.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

tkm-research

Model Tree

Base

google/gemma-4-31B-it

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality