GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
KevinDavidHayes
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
KevinDavidHayes
Model Tree
Qwen/Qwen3.5-0.8B
Input Modalities
Output Modalities
Supported Functionality
bash
pip install pinocchio-uqpython
from pinocchio import Pinocchio judge = Pinocchio() # loads Qwen/Qwen3.5-0.8B and this adapter response = client.chat.completions.create(model="gpt-5", messages=messages)p_correct = judge.score(response, messages=messages)The calibrator reads the question and response (plus optional benchmark and model-identity metadata) and predicts a single token, i (incorrect) or ii (correct). The correctness probability is the softmax over those two logits. During training, every text example was paired with a blank placeholder image; the pinocchio-uq package reproduces that input.
| Base model | Qwen/Qwen3.5-0.8B |
| Method | LoRA (rank 16), 3 epochs |
| Source models | Claude Fable 5, Claude Opus 5, GPT-5.6, Kimi 3, GPT-5-mini, GPT-5.2, Qwen3.5-397B |
| Benchmarks | The 9 text benchmarks of the paper's training set: ARC-AGI, BBEH, ChemBench, GPQA Diamond, HLE, LiveBench, OmniMath, PRBench, SimpleQA |
| Split | Question-level held-out split, the same as the paper's largest model |
On held-out text responses from the seven source models (n = 2,293):
| Metric | Value |
|---|---|
| AUROC | 0.867 |
| ECE | 0.080 |
As the paper reports, this lightweight text-only checkpoint matches the AUROC of the largest model. Results for vision-language inputs, transfer to unseen models, and baselines are in the paper.
bibtex
@inproceedings{hayes2026pinocchio, title = {Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models}, author = {Hayes, Kevin David and Pal, Arka and Zhang, Haosong and Goldstein, Tom and Goldblum, Micah}, booktitle = {Conference on Language Modeling (COLM)}, year = {2026}}from pinocchio import Pinocchio judge = Pinocchio() # loads Qwen/Qwen3.5-0.8B and this adapter response = client.chat.completions.create(model="gpt-5", messages=messages)p_correct = judge.score(response, messages=messages)@inproceedings{hayes2026pinocchio, title = {Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models}, author = {Hayes, Kevin David and Pal, Arka and Zhang, Haosong and Goldstein, Tom and Goldblum, Micah}, booktitle = {Conference on Language Modeling (COLM)}, year = {2026}}