Quickstart
pip install "transformers>=5.19" "peft>=0.21" torch acceleratepython quickstart.py
from yue_judge import YueJudge judge = YueJudge(adapter=".") # needs ~26 GB of GPU memory in bf16r = judge.choice( state="喂,我上個禮拜喺你哋網店買咗部風筒,今日先開箱,發現插唔到電,想換過部新嘅。", question="呢個客人想做乜?", options=["查詢送貨進度", "退換有問題嘅貨品", "投訴店員態度", "問點樣用優惠碼"],)r["top"], r["probs"] # '退換有問題嘅貨品', {...: 0.897, ...} judge.yes_no(state, "客服有冇跟規則?") # P(yes)judge.score(state, "呢單維修應該幾急?", levels=[...5 level descriptions...])
yue_judge.py builds the exact prompt the adapter was trained on. Use it rather than writing your own prompt:
the readout depends on the format. Each result also has label_mass, the share of next-token probability on the
option letters. Below about 0.9 the input is far from the training format and the probabilities are less reliable.
Intended use
- Routing and triage of Cantonese messages (intent, topic, urgency).
- Checking text against a rule or a claim (does the reply follow policy, is the claim supported by the passage).
- Judging Cantonese content: naturalness, register, Hong Kong vs Mainland usage, local knowledge, scams, toxicity.
- A cheap, calibrated scorer inside a larger system, where the probability is used, not just the top answer.
Not intended for: generating text, open-ended questions without options, high-stakes decisions about people
without human review, or as the only safety filter in front of users.
Training (summary)
- Start from Gemma 4 12B-it.
- Supervised fine-tuning of a LoRA adapter on curated Cantonese decision items with soft targets: each item
carries a vote share over its options from several independent judges, and the loss matches the model's
option distribution to that share. The mix also has Traditional Chinese and English decision items, prompt-
injection and content-free items (so the model ignores instructions inside the text and stays uniform when the
text says nothing), and option-order permutation pairs (so the answer does not depend on where an option sits).
- Calibration: raw probabilities are the default (see below).
Training ran on Google Cloud TPU (v6e) for early runs and on a local NVIDIA GB10 for the final adapter.
Post-SFT stages that we tried, including RL-style objectives with proper-scoring-rule rewards and a label-free
consistency stage, did not beat this adapter on held-out data, so they are not part of this release.
Evaluation
All evals are held out from training. "Base" is the same Gemma 4 12B-it weights with the adapter off, same prompt,
with its own per-type temperature fit on the calibration split (raw base outputs are ≈ 100% confident and score
much worse). Accuracy = the top option equals the majority vote. 95% CIs from a paired bootstrap over items.
One training seed.
Table with columns: Eval set, n, Model acc [95% CI], Base acc, Δ acc [95% CI], Model NLL, Base NLL| Eval set | n | Model acc [95% CI] | Base acc | Δ acc [95% CI] | Model NLL | Base NLL |
|---|
| Cantonese decisions, test | 359 | 0.847 [0.808, 0.886] | 0.822 | +2.5 pp [−1.4, +6.1] | 0.602 | 0.643 |
| Cantonese decisions, held-out task types | 174 | 0.902 [0.856, 0.943] | 0.851 | +5.2 pp [+0.6, +9.8] | 0.502 | 0.557 |
| Cantonese decisions, test v2 | 584 | 0.896 [0.872, 0.920] | 0.848 | +4.8 pp [+2.2, +7.4] | 0.525 | 0.618 |
| Cantonese decisions, test v2 (larger) | 799 | 0.916 [0.896, 0.935] | 0.867 | +4.9 pp [+2.6, +7.3] | 0.478 | 0.599 |
| Cantonese decisions, hard subset | 174 | 0.937 [0.897, 0.971] | 0.856 | +8.0 pp [+3.4, +12.6] | 0.449 | 0.566 |
| English decisions | 749 | 0.765 [0.734, 0.796] | 0.728 | +3.7 pp [+1.6, +6.1] | 0.640 | 0.705 |
| English NLI, 100 human votes per item | 600 | 0.712 [0.677, 0.747] | 0.718 | −0.7 pp [−3.8, +2.5] | 0.935 | 0.950 |
| Hong Kong knowledge probe (279 items, both option orders) | 558 | 0.919 | 0.896 | | | |
NLL is the soft negative log-likelihood against the vote shares (lower is better), both models with a temperature
fit by NLL on the calibration split.
Calibration. Raw the model probabilities are near-calibrated: top-1 ECE (15 equal-mass bins) 0.041–0.065 on
the sets above. calibration.json therefore defaults to T = 1. It also holds per-type temperatures fit to vote
shares (temperature_vote_share), which spread probability toward minority answers; use them if you compare
against human vote distributions rather than ask "how often is the top answer right".
Robustness.
Table with columns: Check, Model, Base| Check | Model | Base |
|---|
| Instructions injected into the text change the answer (1,200 injected items) | 13.2% | 24.7% |
| … when the injection names a wrong option | 16.3% | 43.0% |
| Content-free text: probability on option A, 4 options (uniform = 0.25) | 0.27 | 0.99 |
| Negated statement pairs with consistent yes/no (within 0.1) | 86% | 92% |
| Same item, options shown in reverse order: answer changes (knowledge probe) | 6.1% | 9.7% |
Limits
- Length. Trained and evaluated on inputs up to 1,024 tokens; most evaluated Cantonese inputs are under 700.
Longer text is cut in the middle by
yue_judge.py (options are never cut). On English items that had to be cut,
accuracy dropped about 6 points.
- Scores are the weakest type. On the test set, score questions reach 0.71 exact-level accuracy against 0.88
for choice and 0.90 for yes/no. Use the expected level (
judge.score(...)["expected"]) rather than the top level.
- Knowledge. Weakest on pronunciation (Jyutping tones, readings of place names) and on time and money words.
- What the training data covers. Mostly judging inputs (messages, posts, passages, tickets). Judging an
assistant's output, refusals, tool-call arguments, speech-recognition transcripts and long multi-turn
dialogues are thin or absent; test on your own data before relying on it there.
- Text only. The base model reads images, audio and video; the adapter was trained and evaluated on text only.
- No refusals, no safety tuning of its own. It answers the question it is given. Robustness to injected
instructions is better than the base model but not complete (see above).
- Not a source of facts. Probabilities reflect what the model learned; they are not evidence.
- One training run; differences of 1–2 points between nearby models are within noise.
Licence and use
Internal preview for hon9kon9ize members only. Do not redistribute or re-upload the weights or the files in this repo.
The base model, google/gemma-4-12B-it, is Apache 2.0.