knowlet

Gemma-4-12B-Unified-System-One

README

License: apache-2.0

Intended use and limits

The measured task is passage-grounded, English yes/no classification using the Noul primitive. The implementation also exposes Choice and Score schemas, but this training run does not establish their quality, multilingual quality, multi-question independence, or general assistant performance.

A scalar temperature is fitted on the 256-case BoolQ calibration split only. That temperature does not establish calibration for images, audio, other tasks, or deployment traffic. Native image/audio weights remain present; their results are reported separately below. This is a small, single-seed public-data study, not a production-readiness claim. Foundation-model pretraining overlap with BoolQ is unknown.

Use the decision interface

Clone the source repository and install the pinned S1 runtime. The model weights are downloaded separately from Hugging Face in the Python example.

bash

git clone https://github.com/knowlet/Gemma-4-12B-Unified-System-One.gitcd Gemma-4-12B-Unified-System-Onegit checkout d4ea1ef79af2e22f666374e529fa56c1de9d3f53uv sync --locked --extra inference

Run the following with uv run --no-sync python from that repository root. snapshot_download pins the weights, processor and s1_config.json to the same immutable Hub commit, then UnifiedDecisionModel loads that local snapshot.

python

from huggingface_hub import snapshot_download from s1.contracts import DecisionRequestfrom s1.unified import UnifiedDecisionModel model_directory = snapshot_download(    repo_id="knowlet/Gemma-4-12B-Unified-System-One",    revision="66626de5cbcf8c5fecb8e9b58710805a02a42f67",)model = UnifiedDecisionModel(    model_directory,    device="auto",)request = DecisionRequest.model_validate({    "state": {        "passage": "Paris is the capital of France.",        "question": "Is Paris the capital of France?",    },    "questions": {        "answer": {            "type": "noul",            "instructions": "Answer the question using the supplied passage.",        }    },})result = model.predict(request)print(result["answers"]["answer"])

device="auto" selects CUDA, then Apple MPS, then CPU. The runtime chooses its default weight dtype for that device and OS. The saved s1_config.json temperature is loaded automatically. Calling a generic chat or text-generation runner does not reproduce this prompt construction, selected-head readout, or calibrated candidate probabilities.

Training and artifact provenance

Table with columns: Item, Value
Item	Value
Base checkpoint	`google/gemma-4-12B-it`
Base revision	`707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7`
Updates	One pass over 2,048 examples; 2,048 / 2,048 completed updates
Adaptation	LoRA rank 8, alpha 16; `q_proj`, `k_proj`, `v_proj`, `o_proj`
Objective / learning rate	Cross-entropy / `2e-5`

The adapter is preserved before merging. Calibration is performed after the merged checkpoint is reloaded. The training and evaluation protocol, archived data, exclusions, and artifact receipts are described in the release documentation.

Evaluation

Calibration and fresh test passages are disjoint from the selected training examples, all official BoolQ training passages, and previously sampled local cases. The dataset manifest SHA-256 is e0ab28e3435a910c19abf3a9a7e58df8b5cff7a806b6c7f4f18021db1d3804dd.

Table with columns: Dataset, Scored coverage, Accuracy, NLL, Brier, ECE
Dataset	Scored coverage	Accuracy	NLL	Brier	ECE
Fresh BoolQ test, pinned base	256 / 256	80.078125%	0.440918	0.287375	0.067065
Fresh BoolQ test, trained	256 / 256	89.843750%	0.272659	0.158027	0.052127
Historical native media, trained	52 / 52

The 52 media cases contain 32 MNIST images and 20 FSDD recordings. Image accuracy: 87.5% (28 / 32); audio accuracy: 30.0% (6 / 20). These cases have informed previous experiments and are regression evidence, not fresh held-out evidence. No media labels are used for training or temperature fitting. A separate historical 128-case BoolQ regression result is 92.1875% (118 / 128), with all 128 cases scored. The pinned base has the same media accuracy, but lower media NLL (1.049982), Brier (0.416844), and ECE (0.157409); text-only fine-tuning and calibration did not improve these media probability metrics.

Metrics and hardware/latency scope come from the saved evaluation report: evaluation receipt (evaluation receipt SHA-256 07da695983a341ed1b488dedb7edd610b426fc86681a9fb45fbf9fe29f6fc280). Evaluation ran on an NVIDIA A100-SXM4-80GB. The trained fresh-test request latency was 88.96 ms p50 / 92.60 ms p95; the 52-case media latency was 111.59 ms p50 / 113.65 ms p95. These are the saved single-request evaluation measurements, not throughput or Apple MPS measurements. NLL uses a probability floor of 1e-12; Brier is the sum of squared errors over the legal candidates; ECE uses 15 equal-width confidence bins. No result from the earlier base-model MPS profiling workload is substituted for this evaluation.

Attribution and licenses

The source model is by Google. Its Apache-2.0 license and applicable notices are retained in LICENSE and NOTICE; this derivative identifies the LoRA fine-tuning and merged checkpoint changes.

BoolQ is by Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova (2019), Google Research. The pinned BoolQ source and transformed BoolQ data are CC-BY-SA-3.0. The frozen data archive preserves the attribution, source hashes, transformations, and license links. The media regression data retains the separate MNIST MIT and FSDD CC-BY-SA-4.0 licenses. These dataset licenses are distinct from the model's source license.

The decision implementation builds on system-one-open; the pinned System One implementation defines the inference contract used here.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

knowlet

Model Tree

Base

google/gemma-4-12B-it

Fine-tuned

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality