abhishek085

spark-s1-1.7b-v5

README

License: apache-2.0

What's new in v5

v5 narrows scope on purpose: training and evaluation are now Jev-style decisions only — agent-harness control (tool-call verification, context pruning, response-quality checks, delegation, error recovery, plan alignment), guardrails (prompt-injection and message-manipulation gating, PII/output-leak checks), moderation and ticket routing, retrieval/re-ranking, and structured records decisions (entity resolution, fraud and risk scoring, invoice/insurance/SOC routing). General-purpose text classification is no longer a training goal for this release. Training data grew from v3's 967 rows to 11,792 rows / 19,568 examples across 49 task packs, drawn from os-datagen (LLM-written scenarios, independently verified) plus new code-only rule packs where both the label and the surface text come from deterministic rules, no LLM in the loop. Recipe: LoRA r16 (merged), lr 5e-5, 1 epoch (lower learning rate and fewer epochs than v3), cross-entropy plus a Brier regulariser.

How it reads an answer

The state is rendered once; each question is appended as a menu (A. ..., B. ..., option definitions in the prompt) followed by the Qwen3 non-thinking assistant header. The logits of the first answer token are restricted to the option-letter tokens (A..Z), divided by a per-type temperature and softmaxed. That distribution is the answer. The weights are an ordinary Qwen3 causal LM; the readout is done by the open-spark-Jev repository (https://github.com/abhishek085/open-spark-jev) code (open_spark_jev.model.MenuScorer, open_spark_jev/serve), which also serves the POST /v1/decide API. calibration.json holds the temperatures, fitted for every question type this release: choice 1.792, score 1.112, noul 1.608.

Results

Own held-out splits (locked test / challenge draw from scenario families never seen in training)

Table with columns: split, accuracy, ECE (calibrated)
split	accuracy	ECE (calibrated)
calibration	0.682	0.021
locked test	0.695	0.033
challenge	0.671	0.040

External sets (no training overlap)

Table with columns: source, accuracy, ECE
source	accuracy	ECE
Agent tool-call risk (60-row diagnostic set)	0.900	0.096
Jev-directory (70 questions)	0.643	0.256
Prompt injection, with deployment context	0.813	0.079
Prompt injection, no context	0.837	0.045
Kev decision-v1 (out-of-domain classification, external)	0.693	0.209

JevBench (public tiers only)

Accuracy on Benchmark Heaven's JevBench v1.2, public items only (231 of 534 total; the judge tier and part of each other tier are private and were not run):

Table with columns: tier, n, accuracy
tier	n	accuracy
easy	48	0.979
standard	72	0.778
hard	111	0.378

This is not the official JevBench Score (that also needs the private item set plus Calibration, Speed and Cost axes measured under JevBench's own protocol); it is a same-methodology comparison point across our own releases.

Latency

Batch size 1, idle GPU, one NVIDIA DGX Spark (GB10), bf16, Hugging Face Transformers: 29.6 ms p50, 33.8 decisions/s. Measured on the v3 release on the same backbone size and the same prompt set; LoRA-merged weights don't change inference cost (same architecture, same FLOPs), so this figure carries over unchanged to v5.

Limitations

Trained and evaluated on Jev-style decisions only for this release; general text classification (topic, sentiment, NLI, etc.) is out of scope and untested here.
Boolean/score temperatures are now fitted (unlike v3, where they were left at 1.0) but on a smaller sample for some question types; recalibrate on your own labelled outcomes before relying on thresholds.
An NVFP4 post-training quantization (MLP-only) was tried on this checkpoint and cost real accuracy (roughly 9-13 points across our external sets) with a calibration-temperature blow-up; it is not shipped. bf16 is the recommended precision for this release.
An RLCD-direct pass was tried on a hard-mined subset of unseen rows and made results worse across the board, not better (a real regression, not noise); it was not applied. This release is supervised fine-tuning only.
Answers depend on the state and option definitions you give it; it never explains itself. Do not use it as the only gate on high-impact actions.

Reproducibility

Code: github.com/abhishek085/open-spark-jev. Training: configs/train/sft_v5.yaml (python -m open_spark_jev.train.sft --config configs/train/sft_v5.yaml). Evaluation: python -m open_spark_jev.eval.osdg, python -m open_spark_jev.eval.external. Single seed; no confidence intervals reported yet.

License

Apache-2.0 (the Qwen/Qwen3-1.7B backbone is Apache-2.0). The training data was generated and verified with Gemma-4-26B-A4B, Qwen3.6-35B-A3B, Qwen3.6-27B and NVIDIA-Nemotron-3-Super-120B-A12B (used as a writer for a subset of the harness packs); check those terms for commercial use.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

abhishek085

Model Tree

Base

Qwen/Qwen3-1.7B

Fine-tuned

this model

Input Modalities

Text

Output Modalities