Jakevin

taiwan-mainland-wording-kev-4b

README

License: apache-2.0

Current task: two wording cue shares

Table with columns: Output key, Intended meaning
Output key	Intended meaning
`taiwan`	Estimated Taiwan share of assigned regional cue occurrences
`mainland`	Estimated mainland share of assigned regional cue occurrences

The two shares sum to one; the frontend multiplies them by 100. These are cue shares, not classification confidence or shares of every word. The current task has no mixed/shared sentence-label outputs. Shared words are outside the target denominator; a genuinely cue-free sentence has no defined count target even though the neural API emits scores.

台灣／大陸用語比例實驗模型。例如已確認的地域線索全部屬台灣，目標是 100% 台灣；兩側各一個線索，目標是 50%／50%。這是目標定義，並不保證模型每次預測正確。四分類表及舊準確率已移到歷史紀錄。

Read the current report · Download the PDF · Evaluation scope and future panel · Archived four-class model card

Run on another device

Use Python 3.12 or 3.13. The first launch also downloads the pinned Qwen base (~8 GB). This repository contains a LoRA adapter and custom Kev pointer head, not a full standalone backbone.

bash

python3.13 -m venv .venv.venv/bin/pip install huggingface-hub==1.32.0.venv/bin/hf download Jakevin/taiwan-mainland-wording-kev-4b \  --revision wording-ratio-v0.2.0-experimental --local-dir wording-ratio.venv/bin/pip install -r wording-ratio/demo/requirements.txt.venv/bin/python wording-ratio/demo/app.py --host 127.0.0.1 --port 8008

Open http://127.0.0.1:8008. Apple Silicon uses MLX automatically. For a reverse proxy running on the same device, point it to port 8008. To serve over the local network, set --host 0.0.0.0. Deployment and API details include frontend migration instructions. No Hugging Face Space or paid GPU is required for local deployment.

Output and meaning

Request example: example-request.json; actual paired response: example-response.json. API shapes were checked against the official Clef output schema; the input model selector is adapted for this local Kev checkpoint.

v0.2.0 removes the vocabulary admission veto and adopts Jev / System One I/O. Every valid input reaches the unchanged neural model. The frontend calls POST /v1/systemone directly. Requests contain model, state, questions; responses contain model, answers keyed by question ID, and usage. The local model selector is kev-wording-ratio (not Cloudflare clef). The bundled example-request.json preserves the exact trained ratio question. choice, noul, score and JSON state are supported as formats, but broad decision-task capability has not been evaluated.

/api/ratio and /api/classify are aliases using the same Jev schema. Previous {"text":"…"} inputs and flat percentage outputs are removed. Update the frontend and server together. Local limits: rendered state 1–400 characters, request ≤16 KiB, 1–64 questions; no image/video support and no silent truncation. See deployment/API details.

Training targets are fractions of regional cue occurrences: Taiwan count divided by Taiwan plus mainland count. Shared words do not enter the denominator; repeated cues count again and overlapping terms use the longest match. The frontend converts the trained two-option values in answers.wording.probabilities to estimated shares, not the response's confidence. These are not calibrated probabilities of linguistic correctness or the writer's identity.

There is no finite lexicon gate in this release. For unmatched vocabulary the model still returns scores summing to one; they are unverified estimates, not established regional cue counts. If a sentence truly has no regional cue, its count target is undefined (zero denominator). Do not use these scores as evidence of regional vocabulary or fabricate a 50/50 gold target for genuinely cue-free sentences. Relative preference, place names and writing system do not establish regional exclusivity. A 100% target means all counted regional cues are Taiwan cues, not all words uniquely Taiwanese.

This API/frontend fix uses byte-identical adapter and head files to the prior ratio release. It is not new training, better accuracy, or verified vocabulary generalization. The earlier four-class task is archived separately; it is not the current output contract.

Training and measured limitations

Continued from our historical strict four-class intermediate checkpoint, with 160 synthetic cue-list sentences, soft targets, one epoch / 20 optimizer updates, learning rate 5e-6, microbatch 1 and accumulation 8. Head temperature is 1.0; no calibration was fitted. The base is Qwen/Qwen3.5-4B-Base@1001bb4d826a52d1f399e183466143f4da7b741b. Bundled runtime source is Kev commit 84847f0a883d900f7de5b7a57eaa341ca7f9a6b4.

The development and test partitions each contain 40 constructed sentences, split by complete cue combination and template, with eight rows per target share. These use familiar digital vocabulary; they are not new-word generalization or independently annotated linguistic gold. Both checkpoints were scored with the same two-option question and MLX bf16 / temperature 1.0.

Table with columns: Partition / metric, Our four-class intermediate, Ratio candidate
Partition / metric	Our four-class intermediate	Ratio candidate
Development MAE, percentage points (n=40)	19.81	19.46
Test MAE, percentage points (n=40)	27.33	22.85
Test RMSE, percentage points (n=40)	36.02	29.31
Test within ±5 percentage points	10/40	6/40

Candidate test MAE by true Taiwan share: 0% → 24.25 pp; 25% → 35.13 pp; 50% → 23.39 pp; 75% → 21.31 pp; 100% → 10.19 pp (n=8 each). The rule teacher has zero error by construction, so these numbers measure imitation of synthetic counting targets, not regional linguistic correctness.

The frozen gate required test MAE ≤10 pp, improvement ≥2 pp, and every target-bin MAE ≤15 pp. The candidate failed and was not promoted. Its lower mean error does not establish reliable composition. Natural sentences can be substantially wrong, including cases containing both regional cues. There is no verified accuracy for broad vocabulary, no guarantee that examples targeted at 50/50 produce 50/50, and no independently verified natural-text or new-vocabulary ratio result. CPU/CUDA deployment options are provided but only the Apple Silicon MLX path was smoke-tested here; their outputs may differ.

evaluation-summary.json contains the aggregate metrics. release.json links the source checkpoint hashes to the exported hashes. The adapter is byte-identical to the candidate; the head's learned tensors are identical, but its exported file hash differs because local paths were replaced with portable provenance names. SHA256SUMS.json pins the distributed artifacts; the server checks it before loading.

Official Kev-4B versus our adapted checkpoints (spent-panel diagnosis)

The previous pilot table's Parent is our historical four-class intermediate, not official Kev. We now score official jaredpalmer/kev-4b@6cfce5c2fa4b4bd64026336ab649c5ca78857d52, that intermediate, and the published ratio checkpoint with the same exact two-option cue-share question, option order, sentence rendering, pinned Qwen backbone, MLX bf16 backend and temperature 1.0. This uses the already inspected 40-row test in 20 combination groups, with eight rows per target share; it is not a new blind test. The protocol was frozen before this comparison's inference, but after prior test inspection. No training, calibration or prompt selection was performed.

Table with columns: Model / baseline, MAE (pp) ↓, RMSE (pp) ↓, Within ±5 pp, Within ±10 pp, Signed Taiwan-share bias (pp)
Model / baseline	MAE (pp) ↓	RMSE (pp) ↓	Within ±5 pp	Within ±10 pp	Signed Taiwan-share bias (pp)
Official Kev-4B, zero-shot ratio prompt	32.15	38.83	4/40	6/40	−19.67
Our four-class intermediate	27.33	36.02	10/40	10/40	−14.92
Our ratio experimental checkpoint

All metrics here use Jev shares serialized to four decimal places. Re-running our two checkpoints reproduces every previously frozen prediction after the same rounding. The ratio checkpoint's MAE is 9.2945 pp lower than official Kev, a 28.91% relative error reduction on this panel, but it still fails the original quality gate. Its ±5-pp count is lower than our four-class intermediate and the constant baseline. These are synthetic count-target errors, not linguistic accuracy percentages; “28.91%” must not be reported as an accuracy gain.

Table with columns: Target Taiwan share (n=8 each), Official Kev MAE (pp), Our intermediate MAE (pp), Our ratio MAE (pp)
Target Taiwan share (n=8 each)	Official Kev MAE (pp)	Our intermediate MAE (pp)	Our ratio MAE (pp)
0%	26.05	0.63	24.25
25%	5.37	32.47	35.14
50%	20.36	26.87	23.39
75%	40.61	40.88	21.31

Improvement is uneven: the ratio model is worse than official Kev at the 25% and 50% targets. Official Kev underestimates Taiwan shares on average; the ratio version overestimates them. A paired percentile bootstrap resampling all 20 combination groups (both script variants together, 10,000 draws, seed 20261005) gives a descriptive 95% interval of −4.02 to +22.90 pp for official-minus-ratio MAE. It includes zero and does not establish statistically confirmed improvement, population generalization or a causal effect of one training stage. This is a single adaptation trajectory on a small spent constructed panel.

Official Kev's choice probabilities under the ratio prompt are a zero-shot baseline; they have not been independently validated as cue fractions. Known vocabulary, provisional cue assignments and constructed targets limit every comparison. The rule's zero error is definitional. Natural-text and vocabulary absent from fine-tuning still require fresh prediction-blind span annotation. No NAER / Chinese Linguipedia data enter this experiment. Only aggregate results and protocol/checkpoint hashes are published; raw sentences and per-row predictions stay local.

Publication scope and license

This release contains weights, tokenizer, inference runtime, frontend, aggregate synthetic metrics and provenance. It does not contain training/evaluation sentences, author annotation files, NAER / Chinese Linguipedia data, or author-probe predictions. Only aggregate ratio records are public. NAER was not used in this ratio training and remains validation-only and local-only. The current report leads with the two-share task and retains four-class studies in a labeled historical chapter. Historical classification results are not ratio performance.

Model weights and runtime are released under Apache-2.0, subject to upstream notices; see LICENSE, LICENSE-Qwen and NOTICE. Original reference corpora and tables are not included or relicensed. Parent training-source limitations remain documented in the historical model card.

Maintainer: Hsuchen Lo (Jakevin). Independent research prototype; no upstream endorsement.

Final Jev regression and publication status

The already inspected 40-row synthetic test was re-scored through the final Jev API: MAE 22.85375 pp, RMSE 29.30882 pp, 6/40 within ±5 pp, 11/40 within ±10 pp. Serialized shares exactly match frozen predictions rounded to four decimals. This is an interface regression, not new training or a blind natural-text experiment. Aggregate regression record.

The paper is an experimental technical report, not peer reviewed or accepted by arXiv. It now presents the current cue-share task first. Four-class adaptation and distillation diagnostics retain their original metrics in a historical chapter and the archived model card. Hsuchen Lo is a Ph.D. student in the Department of Industrial Engineering and Management, National Taipei University of Technology.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Jakevin

Model Tree

Base

Qwen/Qwen3.5-4B-Base

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality