jtatman

kev-0.8b-code-verify-v1

README

License: apache-2.0

How to ask it

The fine-tune binds these exact strings. One noul question (probability that the statement holds):

json

{"type": "noul", "instructions": "Does the code correctly and completely implement the request, so it would pass a thorough hidden test suite including edge cases? Judge from the request, the code and the checks shown."}

State (an object; Kev renders it as key: value lines):

json

{"request": "<the task, verbatim>",  "code": "<the model's code>",  "checks": {"compiles": true, "defines_requested_function": true, "passes_examples_in_request": true,             "generated_edge_case_tests": "7 of 8 passed",             "note": "edge-case tests were written by a small model from the request alone; some may be wrong"}}

generated_edge_case_tests comes from a small model writing ~8 asserts from the request only, then running them. Serve with Kev's runtime (python -m kev.serve --run jtatman/kev-0.8b-code-verify-v1, a TypeSafe /v1/systemone endpoint) or kev.predictors.LocalPredictor. The checkpoint carries its fitted temperature.

Results

Unseen coder (llama-3.1-8b-instruct, never in training): 488 attempts, pass rate 0.57. "Kept local @X% err" = share of attempts a router can accept, highest score first, with at most X% of them wrong. evidence score = (request examples passed + generated-assert pass rate) / 2.

Table with columns: score, AUROC, Brier, ECE, kept local @2% err, @5%, @10%
score	AUROC	Brier	ECE	kept local @2% err	@5%	@10%
baseline kev	0.935	0.096	0.12	0.23	0.28	0.48
fine-tuned kev	0.959	0.064	0.05	0.23	0.40	0.63

Development (coders seen in training, tasks not seen):

Table with columns: score, AUROC, Brier, ECE, kept local @2% err, @5%, @10%
score	AUROC	Brier	ECE	kept local @2% err	@5%	@10%
baseline kev	0.925	0.029	0.05	0.73	0.89	0.95
fine-tuned kev	0.941	0.025	0.03	0.80	0.89	0.95

Numbers are on small sets; see the project repo for task-bootstrap confidence intervals.

Recommended use

Average this model's probability with the execution evidence score, and tune the accept threshold per cheap-tier coder on a few hundred of that coder's execution-labelled attempts: a fixed threshold's error rate depends on how often the coder fails.

Training

Init: jaredpalmer/kev-0.8b@9a45d25 (LoRA r=16 + pointer head, base Qwen/Qwen3.5-0.8B-Base), Kev's own trainer (kev.train, commit f2bb629d).
Data: 1416 execution-labelled records (task-grouped split; llama-3.1-8b held out) + 2000 public decision-v7 replay records against forgetting.
One epoch, lr 2e-05, batch 4 x accum 2, gradient checkpointing, bf16 autocast, state limit 1664 tokens.
Temperature fitted on a held-out calibration split (min NLL).
Hardware: one NVIDIA L4 (Google Colab).

Training data provenance

Code attempts whose correctness this model learned to judge (labels from executing hidden tests):

Table with columns: source, model, license, training rows
source	model	license	training rows
or:mistral-small-3.2-24b	mistralai/Mistral-Small-3.2-24B-Instruct-2506	Apache-2.0	297
or:qwen3-235b-a22b	Qwen/Qwen3-235B-A22B-Instruct-2507	Apache-2.0	292
or:qwen3-coder-30b-a3b	Qwen/Qwen3-Coder-30B-A3B-Instruct	Apache-2.0	238
qwen2.5-coder:3b	Qwen/Qwen2.5-Coder-3B-Instruct	Qwen Research License

Built with Qwen. Training data includes outputs of Qwen2.5-Coder-3B-Instruct, used under the Qwen Research License Agreement (section 4b). The held-out test coder (Llama 3.1 8B Instruct) was never used for training.

Limitations

Python function-level tasks only (HumanEvalPack); other languages, repos and multi-file changes are untested.
The coder pool is 8-10 models of 3B-235B parameters; accept thresholds do not transfer across coders.
Trained on this question and state layout; other phrasings work less well.

Credits

Kev by Jared Palmer (jaredpalmer/kev, Apache-2.0); Qwen3.5 base (Apache-2.0); HumanEvalPack (bigcode). Project: https://github.com/jtatman/kev-code-verify

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

jtatman

Model Tree

Base

jaredpalmer/kev-0.8b

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

How to ask it

The fine-tune binds these exact strings. One noul question (probability that the statement holds):

json

{"type": "noul", "instructions": "Does the code correctly and completely implement the request, so it would pass a thorough hidden test suite including edge cases? Judge from the request, the code and the checks shown."}

State (an object; Kev renders it as key: value lines):

json

{"request": "<the task, verbatim>",  "code": "<the model's code>",  "checks": {"compiles": true, "defines_requested_function": true, "passes_examples_in_request": true,             "generated_edge_case_tests": "7 of 8 passed",             "note": "edge-case tests were written by a small model from the request alone; some may be wrong"}}

Results

Table with columns: score, AUROC, Brier, ECE, kept local @2% err, @5%, @10%
score	AUROC	Brier	ECE	kept local @2% err	@5%	@10%
baseline kev	0.935	0.096	0.12	0.23	0.28	0.48
fine-tuned kev	0.959	0.064	0.05	0.23	0.40	0.63

Development (coders seen in training, tasks not seen):

Table with columns: score, AUROC, Brier, ECE, kept local @2% err, @5%, @10%
score	AUROC	Brier	ECE	kept local @2% err	@5%	@10%
baseline kev	0.925	0.029	0.05	0.73	0.89	0.95
fine-tuned kev	0.941	0.025	0.03	0.80	0.89	0.95

Numbers are on small sets; see the project repo for task-bootstrap confidence intervals.

Recommended use

Training

Init: jaredpalmer/kev-0.8b@9a45d25 (LoRA r=16 + pointer head, base Qwen/Qwen3.5-0.8B-Base), Kev's own trainer (kev.train, commit f2bb629d).
Data: 1416 execution-labelled records (task-grouped split; llama-3.1-8b held out) + 2000 public decision-v7 replay records against forgetting.
One epoch, lr 2e-05, batch 4 x accum 2, gradient checkpointing, bf16 autocast, state limit 1664 tokens.
Temperature fitted on a held-out calibration split (min NLL).
Hardware: one NVIDIA L4 (Google Colab).

Training data provenance

Code attempts whose correctness this model learned to judge (labels from executing hidden tests):

Table with columns: source, model, license, training rows
source	model	license	training rows
or:mistral-small-3.2-24b	mistralai/Mistral-Small-3.2-24B-Instruct-2506	Apache-2.0	297
or:qwen3-235b-a22b	Qwen/Qwen3-235B-A22B-Instruct-2507	Apache-2.0	292
or:qwen3-coder-30b-a3b	Qwen/Qwen3-Coder-30B-A3B-Instruct	Apache-2.0	238
qwen2.5-coder:3b	Qwen/Qwen2.5-Coder-3B-Instruct	Qwen Research License

Limitations

Python function-level tasks only (HumanEvalPack); other languages, repos and multi-file changes are untested.
The coder pool is 8-10 models of 3B-235B parameters; accept thresholds do not transfer across coders.
Trained on this question and state layout; other phrasings work less well.

Credits

Kev by Jared Palmer (jaredpalmer/kev, Apache-2.0); Qwen3.5 base (Apache-2.0); HumanEvalPack (bigcode). Project: https://github.com/jtatman/kev-code-verify