GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
PSanni
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
PSanni
Model Tree
Qwen/Qwen3-14B-Base
Input Modalities
Output Modalities
Supported Functionality
bash
git clone https://github.com/jaredpalmer/kev && cd kev && uv sync --extra serveuv run --extra serve python -m kev.serve --run PSanni/mati-14b --port 8011python
from typesafe_sdk import TypeSafeClientclient = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8011", model="kev-latest")Note: kev.serve exposes each checkpoint under the model name kev-latest.
I have benchmarked it on Kev's out-of-domain read for an apples-to-apples comparison. The training and calibration data, however, are different. Every column scores the same items. Each model is compared as it is served. Mati-14B and Kev-9B apply the temperature stored in their checkpoints. Jev is scored on the probabilities its API returns. Results on a custom out-of-domain set will be added here soon (waiting for TREC numbers).
| Mati-14B | Kev-9B | Jev | |
|---|---|---|---|
| accuracy | 0.803 | 0.822 | 0.857 |
| Brier score | 0.276 | 0.264 | 0.211 |
| expected calibration error (ECE) | 0.066 | 0.042 | 0.049 |
| negative log-likelihood | 0.476 | 0.484 | 0.701 |
| mean confidence | 0.867 | 0.832 | 0.903 |
| confident errors (p ≥ 0.9 and wrong, share of all questions) | 4.1% | 4.0% | 3.7% |
| accuracy when p ≥ 0.9 | 0.935 | 0.932 | 0.948 |
| coverage at ≤ 5% error | 0.59 | 0.45 | 0.70 |
| coverage at ≤ 1% error | 0.37 | 0.11 | – |
| option-order flip rate (36 questions) | 2.8% | 2.8% | 0.0% |
| minimal pairs, both siblings correct (64 pairs) | 0.781 | 0.828 | 0.859 |
Definitions:
Accuracy by source (Mati-14B / Kev-9B / Jev; 80 questions each unless noted):
| source | Mati-14B | Kev-9B | Jev |
|---|---|---|---|
| SciQ | 0.988 | 0.963 | 0.988 |
| QNLI | 0.900 | 0.925 | 0.925 |
| PAWS | 0.762 | 0.762 | 0.787 |
| MMLU | 0.750 | 0.738 | 0.900 |
| TweetEval-offensive | 0.725 | 0.775 | 0.812 |
| Emotion | 0.562 | 0.600 | 0.588 |
| deadline, date arithmetic (40) | 0.825 | 0.800 | 0.925 |
| authorization (40) | 1.000 | 1.000 | 1.000 |
| held-out policy: (A or B) and C (32) | 0.906 | 0.906 | 0.906 |
| held-out policy: (A and B) or not C (32) | 0.719 | 0.875 | 0.969 |
| held-out policy: if A then not B else C (32) | 0.844 | 0.906 | 0.781 |
Calibration is built in. head.pt carries a temperature (T = 1.87), as in Kev's contract. The pointer head divides its logits by this temperature at inference, so every loader (kev.serve, kev.benchmark, your own harness) gets calibrated probabilities by default.
Reports come from kev.benchmark with logits recorded and the checkpoint's temperature applied. All 764 of 764 transfer records were evaluated, with none rejected or truncated. Kev-9B's numbers are from its published trial result at its shipped temperature on the same partition (same sha256). Jev's are from the Kev repo's scored run of Jev on the same partition. Scored question counts exclude the robustness variants (options permuted, none-of-the-above inserted), which are reported separately.
Apache-2.0 for the adapter and head. The Qwen3 base is Apache-2.0.
git clone https://github.com/jaredpalmer/kev && cd kev && uv sync --extra serveuv run --extra serve python -m kev.serve --run PSanni/mati-14b --port 8011from typesafe_sdk import TypeSafeClientclient = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8011", model="kev-latest")