ninhnv

chon-1.1-31b

README

License: apache-2.0

Results

Our own runs with the public serving code v1.5.3, not official board numbers.

Decision Index, public suite (edition 0.3)

Table

Decision Index	65.47
balanced raw / breadth skill	73.47 / 64.89
rows	140,620 read; all 140,178 scoreable rows answered; 0 unsupported, 0 errors
run	the kit's own runner (`decision_index run --engine chon.dindex:ChonEngine`), the full suite in 4 parts on 4 GPUs, merged with `chon.dindex merge`

Table with columns: area, skill, raw
area	skill	raw
tools	0.791	0.812
language	0.699	0.785
retrieval	0.674	0.748
knowledge	0.555	0.640
arts	0.507	0.681

Latency (board rule: median, mean and 80th percentile each at most 1,000 ms per request on one RTX PRO 6000), every request of the full suite, one request at a time per GPU:

Table with columns: median, mean, 80th percentile, 99th percentile
median	mean	80th percentile	99th percentile
154 ms	725 ms	585 ms	3,782 ms

The slowest requests are those with 20-30 questions about one short state: every question is listed before the state (the "header" layout), so their shared prompt reaches 56,000-73,000 tokens. Prompts over 4,096 tokens fill the cache in chunks, and 32 requests of the run were read a second time in smaller pieces after running out of GPU memory (all answered).

The full (private) Decision Index has not scored this model yet.

JevBench, public items

Table with columns: file, items, accuracy
file	items	accuracy
hard	111	88.3
original	72	100.0
easy	48	100.0

Scored with the board's published formulas (equal type weights, tier weights 10/20/30/40, yes/no answers inside 0.2-0.8 count as wrong; the calibration mapping is not fully published, so calibration is our proxy):

Table with columns: serving, Intelligence, Calibration (proxy), Composite
serving	Intelligence	Calibration (proxy)	Composite
as released	80.6	90.6	85.6
with `--noul-commit 0.199,0.801` (serving code v1.6.0)	84.4	91.9	88.1

The commit band moves a yes/no answer between 0.2 and 0.8 to 0.199 or 0.801, so JevBench does not count it as an abstention; it is off by default and not used on the Decision Index. Expected calibration error on the public items: 0.023. The public set is small (231 items): a few answers move these numbers by a point or more.

Use

bash

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.5.3"python -m chon.serve --model ninhnv/chon-1.1-31b --port 8000      # downloads the base model and this adapter

The first start downloads the base model, google/gemma-4-31B-it, into the Hugging Face cache. --device defaults to auto: an NVIDIA GPU if there is one, else Apple's GPU, else the CPU.

bash

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{  "state": "Order #4411 shipped on 3 March; the customer reports it arrived damaged on 9 March.",  "questions": {"decision": {"type": "choice", "instructions": "Route this ticket.",    "criteria": {"refund": "Refund or replacement", "shipping": "Delivery status", "other": "Anything else"}}}}'

The response holds answers.decision with choice and probabilities for choice, noul (P(yes); also as probabilities keyed true / false) for yes/no, or probabilities over the levels for score. usage reports input_tokens and output_tokens (always 0).

For JevBench, serve with the yes/no commit band (v1.6.0 or newer):

bash

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.6.0"python -m chon.serve --model ninhnv/chon-1.1-31b --port 8000 --noul-commit 0.199,0.801

Python

python

from chon.infer import DecisionModel dm = DecisionModel.load("ninhnv/chon-1.1-31b", device="auto")answers, usage = dm.predict(    "Order #4411 shipped on 3 March; the customer reports it arrived damaged on 9 March.",    {"damaged": {"type": "noul", "instructions": "Did the order arrive damaged?"}})print(answers["damaged"]["noul"])   # P(yes)

When option labels are only ids ("0", "1", ...), add "option_keys": "hide" to the question so the model sees only the option descriptions.

Reproduce the Decision Index run

bash

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.5.3"   # plus the Decision Index kitpython -m chon.dindex shard --suite-dir suite-0.3 --n 4 --out runs/di/shardsfor k in 0 1 2 3; do  CUDA_VISIBLE_DEVICES=$k python -m decision_index run --engine chon.dindex:ChonEngine \    --option model=ninhnv/chon-1.1-31b --compact \    --rows runs/di/shards/shard-$k.jsonl.gz --out runs/di/parts/shard-$k &done; waitpython -m chon.dindex merge --suite-dir suite-0.3 --shards runs/di/parts --out runs/di/chon-1.1-31b

How it reads

Prompt: the model's chat template with thinking off; every question of the request listed first, then the state, then the question and its options as (A) key: description lines; the answer is the letter after Answer: (.
Readout: a softmax over the option letters only, read with the final logit soft cap off; per-type temperatures (choice 0.726, yes/no 0.564, score 0.803) fitted by negative log-likelihood on development items.
Second read: a choice or yes/no answer whose top probability is under 0.9, in a prompt of at most 2,000 tokens, is read again with the request written out twice, and the two reads are averaged.
Shared prefix: the questions of one request share a single read of their common prompt (the same answers as separate passes, up to floating-point rounding).

Training

Method: LoRA (rank 32, alpha 32, no dropout) on the attention and MLP projections of the language model; AdamW, learning rate 5e-5, beta2 0.95; 664 optimizer steps.
Targets: the full distribution over option letters (soft labels where the item has them), with the option order shuffled.
Data: 208,173 typed-decision items: 63,171 from earlier Chon rounds and 145,002 new ones (public datasets turned into typed decisions, multi-question formats, and synthetic decision tasks written and labelled by LLMs, then filtered).
Decontamination: every item was checked against the Decision Index 0.3 suite, the public JevBench files and our held-out sets (exact, near-duplicate and long shared passages), and matches were removed. No Decision Index suite rows and no JevBench items were trained on.

Limitations

Score questions are the weakest type (chance-corrected 68 on JevBench public, against 91 for choice and 94 for yes/no with the commit band).
Requests with many questions about one state are slow in the header layout (seconds to minutes for 20-30 questions).
Timed in bf16 only; FP8 serving is not tested.
The probabilities are calibrated on our own development items; check them on your own data.

What is in this repository

Table with columns: File, What it is
File	What it is
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (PEFT)
`chon_config.json`	serving settings (prompt and calibration) read by the `chon` server
tokenizer files	the base model's tokenizer and chat template, unchanged
`README.md`	this card

Licence

Apache-2.0, as google/gemma-4-31B-it.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ninhnv

Model Tree

Base

google/gemma-4-31B-it

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results

Our own runs with the public serving code v1.5.3, not official board numbers.

Decision Index, public suite (edition 0.3)

Table

Decision Index	65.47
balanced raw / breadth skill	73.47 / 64.89
rows	140,620 read; all 140,178 scoreable rows answered; 0 unsupported, 0 errors
run	the kit's own runner (`decision_index run --engine chon.dindex:ChonEngine`), the full suite in 4 parts on 4 GPUs, merged with `chon.dindex merge`

Table with columns: area, skill, raw
area	skill	raw
tools	0.791	0.812
language	0.699	0.785
retrieval	0.674	0.748
knowledge	0.555	0.640
arts	0.507	0.681

Latency (board rule: median, mean and 80th percentile each at most 1,000 ms per request on one RTX PRO 6000), every request of the full suite, one request at a time per GPU:

Table with columns: median, mean, 80th percentile, 99th percentile
median	mean	80th percentile	99th percentile
154 ms	725 ms	585 ms	3,782 ms

The full (private) Decision Index has not scored this model yet.

JevBench, public items

Table with columns: file, items, accuracy
file	items	accuracy
hard	111	88.3
original	72	100.0
easy	48	100.0

Table with columns: serving, Intelligence, Calibration (proxy), Composite
serving	Intelligence	Calibration (proxy)	Composite
as released	80.6	90.6	85.6
with `--noul-commit 0.199,0.801` (serving code v1.6.0)	84.4	91.9	88.1

Use

bash

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.5.3"python -m chon.serve --model ninhnv/chon-1.1-31b --port 8000      # downloads the base model and this adapter

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.5.3"python -m chon.serve --model ninhnv/chon-1.1-31b --port 8000      # downloads the base model and this adapter

The first start downloads the base model, google/gemma-4-31B-it, into the Hugging Face cache. --device defaults to auto: an NVIDIA GPU if there is one, else Apple's GPU, else the CPU.

bash

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{  "state": "Order #4411 shipped on 3 March; the customer reports it arrived damaged on 9 March.",  "questions": {"decision": {"type": "choice", "instructions": "Route this ticket.",    "criteria": {"refund": "Refund or replacement", "shipping": "Delivery status", "other": "Anything else"}}}}'

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{  "state": "Order #4411 shipped on 3 March; the customer reports it arrived damaged on 9 March.",  "questions": {"decision": {"type": "choice", "instructions": "Route this ticket.",    "criteria": {"refund": "Refund or replacement", "shipping": "Delivery status", "other": "Anything else"}}}}'

For JevBench, serve with the yes/no commit band (v1.6.0 or newer):

bash

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.6.0"python -m chon.serve --model ninhnv/chon-1.1-31b --port 8000 --noul-commit 0.199,0.801

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.6.0"python -m chon.serve --model ninhnv/chon-1.1-31b --port 8000 --noul-commit 0.199,0.801

Python

python

from chon.infer import DecisionModel dm = DecisionModel.load("ninhnv/chon-1.1-31b", device="auto")answers, usage = dm.predict(    "Order #4411 shipped on 3 March; the customer reports it arrived damaged on 9 March.",    {"damaged": {"type": "noul", "instructions": "Did the order arrive damaged?"}})print(answers["damaged"]["noul"])   # P(yes)

from chon.infer import DecisionModel dm = DecisionModel.load("ninhnv/chon-1.1-31b", device="auto")answers, usage = dm.predict(    "Order #4411 shipped on 3 March; the customer reports it arrived damaged on 9 March.",    {"damaged": {"type": "noul", "instructions": "Did the order arrive damaged?"}})print(answers["damaged"]["noul"])   # P(yes)

When option labels are only ids ("0", "1", ...), add "option_keys": "hide" to the question so the model sees only the option descriptions.

Reproduce the Decision Index run

bash

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.5.3"   # plus the Decision Index kitpython -m chon.dindex shard --suite-dir suite-0.3 --n 4 --out runs/di/shardsfor k in 0 1 2 3; do  CUDA_VISIBLE_DEVICES=$k python -m decision_index run --engine chon.dindex:ChonEngine \    --option model=ninhnv/chon-1.1-31b --compact \    --rows runs/di/shards/shard-$k.jsonl.gz --out runs/di/parts/shard-$k &done; waitpython -m chon.dindex merge --suite-dir suite-0.3 --shards runs/di/parts --out runs/di/chon-1.1-31b

pip install "chon[serve] @ git+https://github.com/xoai/chon@v1.5.3"   # plus the Decision Index kitpython -m chon.dindex shard --suite-dir suite-0.3 --n 4 --out runs/di/shardsfor k in 0 1 2 3; do  CUDA_VISIBLE_DEVICES=$k python -m decision_index run --engine chon.dindex:ChonEngine \    --option model=ninhnv/chon-1.1-31b --compact \    --rows runs/di/shards/shard-$k.jsonl.gz --out runs/di/parts/shard-$k &done; waitpython -m chon.dindex merge --suite-dir suite-0.3 --shards runs/di/parts --out runs/di/chon-1.1-31b

How it reads

Prompt: the model's chat template with thinking off; every question of the request listed first, then the state, then the question and its options as (A) key: description lines; the answer is the letter after Answer: (.
Readout: a softmax over the option letters only, read with the final logit soft cap off; per-type temperatures (choice 0.726, yes/no 0.564, score 0.803) fitted by negative log-likelihood on development items.
Second read: a choice or yes/no answer whose top probability is under 0.9, in a prompt of at most 2,000 tokens, is read again with the request written out twice, and the two reads are averaged.
Shared prefix: the questions of one request share a single read of their common prompt (the same answers as separate passes, up to floating-point rounding).

Training

Method: LoRA (rank 32, alpha 32, no dropout) on the attention and MLP projections of the language model; AdamW, learning rate 5e-5, beta2 0.95; 664 optimizer steps.
Targets: the full distribution over option letters (soft labels where the item has them), with the option order shuffled.
Data: 208,173 typed-decision items: 63,171 from earlier Chon rounds and 145,002 new ones (public datasets turned into typed decisions, multi-question formats, and synthetic decision tasks written and labelled by LLMs, then filtered).
Decontamination: every item was checked against the Decision Index 0.3 suite, the public JevBench files and our held-out sets (exact, near-duplicate and long shared passages), and matches were removed. No Decision Index suite rows and no JevBench items were trained on.

Limitations

Score questions are the weakest type (chance-corrected 68 on JevBench public, against 91 for choice and 94 for yes/no with the commit band).
Requests with many questions about one state are slow in the header layout (seconds to minutes for 20-30 questions).
Timed in bf16 only; FP8 serving is not tested.
The probabilities are calibrated on our own development items; check them on your own data.

What is in this repository

Table with columns: File, What it is
File	What it is
`adapter_model.safetensors`, `adapter_config.json`	the LoRA adapter (PEFT)
`chon_config.json`	serving settings (prompt and calibration) read by the `chon` server
tokenizer files	the base model's tokenizer and chat template, unchanged
`README.md`	this card

Licence

Apache-2.0, as google/gemma-4-31B-it.