GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JohnP1
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JohnP1
Model Tree
google/gemma-4-E2B
Input Modalities
Output Modalities
Supported Functionality
| Kev-Gemma4-E2B, 1 epoch | Kev-0.8B base recipe, 2 epochs | Kev-4B base recipe, 2 epochs | |
|---|---|---|---|
| in-distribution accuracy (decision-v7 dev, 1,204 records / 1,264 clean questions) | 0.794 | 0.817 | 0.877 |
| out-of-domain accuracy (transfer-v4 dev, 764 records) | 0.569 | 0.622 | 0.788 |
| ECE, in-distribution, trial temperature | 0.029 | 0.042 | 0.024 |
| confident errors (p ≥ 0.9 and wrong), out of domain, trial temperature | 2.1% | 1.1% | 3.0% |
| permutation flip rate | 0.017 | 0.000 | 0.033 |
| held-out rule pairs, both answers correct | 0.45 | 0.69 | 0.94 |
| question isolation (max Δp with a sibling question, fp32) | 1.8e-5 | 2.3e-4 | 3.4e-4 |
The Qwen columns are the seed-0 trials of the same base recipe (q35-08b/00-trial-0, q35-4b/00-trial-0 in the Kev
repo), before the delta fine-tune the released checkpoints add. Evaluation ran on Apple Silicon (MPS) in fp32.
Where it falls short: generated rule structures (composition tasks 0.31–0.75 in distribution; held-out rule pairs 0.45 against a 0.70 research gate) and out-of-domain paraphrase (PAWS 0.29). The public classification sources are close to Kev-0.8B (dbpedia14 0.96, trec 0.95, imdb 0.91, banking77 0.84). A second epoch is the obvious next run; Kev-0.8B needed two to learn the rule compositions.
head.pt carries a temperature of 1.48, fitted on the development rows with scripts/calibrate_checkpoint.py
(out-of-fold ECE 0.027, 95% interval 0.021–0.054). Out of domain it lowers confident errors from 6.4% to 1.2% and ECE
from 0.152 to 0.105 without changing any answer. KEV_TEMPERATURE=1.0 serves the raw logits.
bash
git clone https://github.com/jonpol01/kev.git && cd kevuv sync --extra serveuv run --extra serve python -m kev.serve --run JohnP1/kev-gemma4-e2b --port 8009Then send System One requests to http://127.0.0.1:8009/v1/systemone (see the Kev README), or point the TypeSafe SDK at it.
Gemma 4 is attention-only, so on Apple Silicon it runs through PyTorch MPS (the MLX backend is only for the hybrid
Qwen3.5 bases). The server defaults to bf16; KEV_DTYPE=fp32 is the path the numbers above use.
<unk>), so it uses the reserved <unused0>–<unused4> rows,
which caller text cannot produce.<bos> is prepended, as Gemma's attention expects. Gemma's own control tokens (<bos>, <pad>, <|turn> ...)
in caller text are escaped.text
kev.train --suite evals/v7/decision-v7 --base google/gemma-4-E2B --base_revision d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f --epochs 1 --lr 1e-4 --batch 4 --accum 2 --dtype bf16 --weights_dtype bf16 --checkpointing 1 --p_none_pair 0.25 --seed 0ccd6e8b, on one NVIDIA L4 (HF Jobs l4x1): 12,506 training records (70 of 12,576 dropped by the
suite's context rule under Gemma's tokenizer), 1,564 optimizer steps, 81 minutes, 14.6 GB peak GPU memory. Trained with
a bf16 backbone; Gemma ships bf16 weights, so head.pt marks the backbone fp32 (the same weights upcast), which makes
the default load the exact path. In bf16 the isolation check is off by up to 0.046, rounding rather than leakage.
Apache-2.0, like Kev and the Gemma 4 base model. Training datasets have their own licenses; see the Kev model cards.
git clone https://github.com/jonpol01/kev.git && cd kevuv sync --extra serveuv run --extra serve python -m kev.serve --run JohnP1/kev-gemma4-e2b --port 8009