GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Datchthana
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Datchthana
Model Tree
jaredpalmer/kev-4b
Input Modalities
Output Modalities
Supported Functionality
Locked test split, 735 records (templates and long-form texts by different authors than train/dev; never trained
on). All three systems received byte-identical states and questions (input_sha checked per record).
What the metrics mean. Accuracy: share of answers that are correct. Precision (is_sensitive): of the texts the model flagged, how many really were sensitive (false alarms). Recall: of the truly sensitive texts, how many it caught (misses; the critical number for privacy). F1: harmonic mean of precision and recall, high only when both are. Macro-F1: F1 computed per category then averaged, so rare categories (credentials, s.26 data) count as much as common ones. Risk exact / ±1: risk level exactly right / at most one level off. MAE: average number of levels the risk prediction is off by. Brier: squared distance between the predicted probabilities and the truth (punishes confident mistakes). ECE: calibration, i.e. whether answers given with 90% confidence are right ~90% of the time. For all of these, higher is better except MAE, Brier, ECE and latency.
| Metric (test, n=735) | Kev-4B base (zero-shot) | kev-thai-pdpa-guard | OpenThai-SystemOne 0.8B | Tev1-4B (4.3B, zero-shot) | Jev, TypeSafe hosted (zero-shot) |
|---|---|---|---|---|---|
| is_sensitive F1 | 0.890 | 0.990 | 0.442 | 0.904 | 0.923 |
| is_sensitive precision / recall | 0.876 / 0.905 | 0.993 / 0.988 | 0.968 / 0.287 | 0.866 / 0.945 | 0.864 / 0.991 |
| has_* macro-F1 (8 PII types) | 0.649 | 0.934 | 0.620 | 0.671 | 0.764 |
| primary_category accuracy / macro-F1 | 0.638 / 0.565 | 0.946 / 0.931 | 0.463 / 0.436 | 0.559 / 0.520 | 0.732 / 0.736 |
| risk_level exact accuracy | 0.476 | 0.912 | 0.524 | 0.486 | 0.597 |
| risk_level within ±1 | 0.703 | 0.977 | 0.702 | 0.702 | 0.797 |
| risk_level MAE | 1.033 | 0.125 | 1.083 | 0.913 | 0.735 |
| Brier (all questions, lower is better) | 0.247 | 0.040 | 0.295 | 0.253 | 0.197 |
| ECE (15 bins, lower is better) | 0.032 | 0.011 | 0.063 | 0.039 | 0.023 |
| Mean latency, RTX 4060 Laptop | 1.31 s (4-bit) | 1.28 s (4-bit) | 0.12 s (bf16) | 3.27 s (Ollama Q8_0) | 0.59 s (API) |
Risk accuracy by gold level (n): 0 (313) 0.981 · 1 (55) 0.927 · 2 (60) 0.917 · 3 (87) 0.667 · 4 (220) 0.905 (base: 0.709 / 0.327 / 0.983 / 0.195 / 0.155). Hard negatives: risk exact accuracy 0.966 (base 0.503).
Reading. Fine-tuning lifts every metric; the largest gain is the 0–4 risk score (+43.6 pp), especially PDPA s.26
critical cases (0.155 → 0.905). OpenThai-SystemOne is ten times faster and reasonable on the per-type flags, but it is
zero-shot on this task and misses most sensitive texts (recall 0.287). Tev1-4B (Together AI, the closest model in size and
type) has good recall (0.945) but collapses the risk scale toward level 2 (risk 4 accuracy 0.255, score ECE 0.319). The remaining weakness is risk 3: 23 of its 29
misses are under-predictions where the "three or more PII types → +1" escalation applies (e.g. name + phone + address).
The test split comes from the same synthetic generator family, so absolute numbers are optimistic for real-world
text; the system-to-system comparison on identical items is the reliable signal. Full tables:
results/comparison.md.
Versus the real Jev (TypeSafe, hosted). Jev was run zero-shot on the same 735 records through the Vercel AI Gateway. It is the strongest of the zero-shot systems (second on almost every row) and has the best recall (0.991), but it false-alarms more (precision 0.864), is fooled by hard negatives (risk exact 0.503) and does not know the project's PDPA risk rules (risk exact 0.597; level 4 only 0.473). The fine-tuned 4B model beats it by +31.5 pp on risk_level, +21.4 pp on primary_category and +0.17 has_* macro-F1, while running entirely on a local 8 GB GPU so personal data never leaves the machine. (On general English tasks Jev leads Kev-4B base, per the Kev-4B card: breadth-v1 0.757 vs 0.690.)
| Developer | Datchthana (built with a Claude agent team: Opus PM, Sonnet leads, Haiku developers) |
| Model type | Decision model: causal LM backbone run prefill-only + pointer head over the supplied options |
| Initialised from | jaredpalmer/kev-4b (adapter + head), backbone Qwen/Qwen3.5-4B-Base (frozen) |
| Adapter | LoRA r=16, α=32 on q/k/v/o, gate/up/down and the Gated DeltaNet projections (33.8M trainable params) |
| Quantisation (training & default serving) | bitsandbytes NF4 + double quant, bf16 compute; DeltaNet gates in_proj_a/b kept bf16 |
| Calibration | the temperature stored in head.pt (inherited from Kev-4B, applied at load) |
| Languages | Thai (primary), English |
| Context | trained with states ≤ 1,024 tokens |
| License | Apache-2.0 |
p_sensitive is high, queue the uncertain middle for a human.Out of scope: legal advice or a final determination of PDPA compliance; fully automated decisions with legal
consequences without human review; locating the exact span of each PII item (the model classifies, it does not tag
spans; use the dataset's pii_spans to train a tagger); documents longer than ~1k tokens without chunking.
bash
git clone https://github.com/Datchthana1/kev-thai-pdpa-guard && cd kev-thai-pdpa-guardbash scripts/setup_env.sh # Python 3.13 venv, torch cu128, Kev, bitsandbytes, flash-linear-attentionpython
import sys; sys.path.insert(0, "src")from pdpa_guard.guard import PDPAGuard guard = PDPAGuard("Datchthana/kev-thai-pdpa-guard") # 4-bit backbone, ~3-4 GB VRAMr = guard.check("ผู้ป่วย นายวิชัย ศรีสุข เป็นเบาหวาน ต้องฉีดอินซูลินวันละ 2 ครั้ง โทร 089-123-4567")print(r.is_sensitive, r.p_sensitive) # True 0.99...print(r.risk_level, r.risk_label) # 4 criticalprint(r.primary_category) # healthprint(r.detected()) # ['has_name', 'has_contact', 'has_health_or_special']The 11 questions are fixed (pdpa_guard.guard.canonical_questions). Any TypeSafe-style client can also send its own
choice / score / noul questions through PDPAGuard.raw() or the Kev server (python -m kev.serve --run <this repo>).
| Field | Meaning |
|---|---|
is_sensitive, p_sensitive | personal data of an identifiable person present (threshold 0.5) |
risk_level, risk_label, risk_score, risk_probs | 0 none · 1 low · 2 medium · 3 high · 4 critical; risk_score = expected level |
primary_category, category_probs | none, name, contact, address, other_identifier, national_id, financial, credential, health, special_category |
flags | p(yes) for has_name, has_contact, has_address, has_national_id, has_financial, has_credential, has_health_or_special, has_other_identifier |
| Level | Rule (computed from the PII present) |
|---|---|
| 0 | no personal data of an identifiable person (company info, public figures in news, anonymous statistics, masked values) |
| 1 | weak identifier only: a name alone, LINE ID, plate, DOB without name |
| 2 | contactable/locatable: phone, email or home address |
| 3 | government/financial identifier: national ID, passport, bank account, card number |
| 4 | PDPA s.26 data about an identifiable person, credentials, full card data (number + expiry/CVV), or national ID + financial data |
+1 (max 4) when ≥3 distinct PII types co-occur.
Datchthana/thai-pdpa-pii-synthetic: 2,741 train /
450 dev / 735 test synthetic records in 15 genres; 56% positive, 25% hard negatives (checksum-failing 13-digit
numbers, Luhn-failing card numbers, public figures, company contacts, masked values, anonymous health statistics),
19% clean negatives; 506 free-form long texts. Labels are computed by code from the recorded PII spans. The test split
uses templates and long-form texts written by different authors from train/dev; 0 cross-split near-duplicates.
| Recipe | kev.train (official Kev trainer) + QLoRA patch (src/kev_qlora), --init_from jaredpalmer/kev-4b |
| Epochs / steps | 1 epoch, 343 optimizer steps |
| Optimiser | AdamW, lr 2e-5, OneCycle (10% warm-up), weight decay 0.01 |
| Batch | 1 record × 8 accumulation (each record = 11 questions); --shared_prefix 1, --length_sort 1 |
| Loss | cross-entropy over each question's options (Kev default) |
| Precision | NF4 backbone, bf16 autocast, fp32 LoRA/head masters, gradient checkpointing |
| Kernels | flash-linear-attention 0.5.2 + triton-windows 3.4 (≈10× faster than the torch fallback on Windows) |
| Hardware / time | Intel i5-12500H, 16 GB RAM, RTX 4060 Laptop 8 GB; peak VRAM 4.55 GB; ≈ 2 h 55 min |
Training loss (25-step means): 0.352 → 0.268 → 0.148 → 0.112 → 0.087 → 0.063 → … → ≈0.03–0.06 at the end.
Built on Kev (Jared Palmer, Apache-2.0) and Qwen3.5-4B-Base (Apache-2.0); compared with iapp/OpenThai-SystemOne. Architecture diagrams by archify.
git clone https://github.com/Datchthana1/kev-thai-pdpa-guard && cd kev-thai-pdpa-guardbash scripts/setup_env.sh # Python 3.13 venv, torch cu128, Kev, bitsandbytes, flash-linear-attentionimport sys; sys.path.insert(0, "src")from pdpa_guard.guard import PDPAGuard guard = PDPAGuard("Datchthana/kev-thai-pdpa-guard") # 4-bit backbone, ~3-4 GB VRAMr = guard.check("ผู้ป่วย นายวิชัย ศรีสุข เป็นเบาหวาน ต้องฉีดอินซูลินวันละ 2 ครั้ง โทร 089-123-4567")print(r.is_sensitive, r.p_sensitive) # True 0.99...print(r.risk_level, r.risk_label) # 4 criticalprint(r.primary_category) # healthprint(r.detected()) # ['has_name', 'has_contact', 'has_health_or_special']