ThakiCloud

K-MT-7B-v0.1

README

License: apache-2.0

0. How to use it — the Korean constraint is set in the prompt

The behaviour does not appear on its own. You state the speech level and address term you want. This is the exact format used in training and evaluation.

python

from transformers import AutoModelForCausalLM, AutoTokenizer mid = "ThakiCloud/K-MT-7B-v0.1"tok = AutoTokenizer.from_pretrained(mid, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16",                                             trust_remote_code=True, device_map="auto") prompt = ("Please translate the following text into Korean. Note that the translation "          "style must strictly conform to [종결어미는 합니다체로 통일 · 청자 호칭은 "          "'본부장님'을(를) 사용]:\n"          "Tell him the deployment finished and ask if we should roll back.") text = tok.apply_chat_template([{"role": "user", "content": prompt}],                               tokenize=False, add_generation_prompt=True)out = model.generate(**tok(text, return_tensors="pt").to(model.device),                     do_sample=False, max_new_tokens=256)      # greedy — same as evaluationprint(tok.decode(out[0], skip_special_tokens=True))

Constraint syntax — inside square brackets, separated by ·. These four were evaluated:

Table with columns: Constraint, How to write it
Constraint	How to write it
Speech level	`종결어미는 합니다체로 통일` (also 해요체, 하십시오체, …)
Address term	`청자 호칭은 '본부장님'을(를) 사용`
Relationship	`화자와 청자의 관계는 부하→상사` (subordinate → superior)
Terminology	prepend `Reference the following translations:\ndeployment translates to 배포`

Give no instruction and it translates normally — that is the intended behaviour. We measured separately that it does not insert address terms when none were requested.

⚠️ The hunyuan_v1_dense architecture needs trust_remote_code=True, and the chat template is the repo's chat_template.jinja (same as the base). All evaluation ran through transformers.

Intended use and scope

Built for: EN↔KO business translation where speech level, address term and relationship have to match an instruction. It is a drop-in replacement for the base model — general translation quality is tied with it (§1).
Not verified: per-direction quality for language pairs without Korean — retention was checked as a 28-direction FLORES aggregate, and no per-direction superiority is claimed. The domain is business documents; literary, medical and legal text are untested. This is a translation model, not a chat model.

What actually changes

Every pair below is a real evaluation output — left is the base model, right is this model, same prompt, same greedy decoding. Of the 461 constraint units, 346 are cases where the base failed the instruction and K-MT passed; zero go the other way.

(a) Address term — the instruction says who is being spoken to, and the base ignores it

Table with columns: Instruction, Source, Hy-MT2-7B (base), K-MT-7B
Instruction	Source	Hy-MT2-7B (base)	K-MT-7B
호칭 `정 책임님` · 해요체	CSAT held up pretty well even with the outage we had last week.	지난주에 발생한 서비스 중단에도 불구하고 CSAT 점수는 꽤 양호한 수준을 유지했어요.	정 책임님, 지난주에 발생한 장애가 있었음에도 CSAT가 꽤 잘 유지되었어요.
호칭 `박 매니저님` · 해요체	Attrition dropped a lot after we rolled out the retention program.	유지 프로그램을 도입한 이후로 이탈률이 상당히 줄었어요.	박 매니저님, 유지 프로그램을 도입한 후 이탈률이 크게 줄었어요.
호칭 `대표님` · 합니다체	I will send the report after the meeting ends.	회의가 끝난 후에 보고서를 보내드리겠습니다.

The base translations are not wrong — they simply left out the address term they were told to use. That is the whole gap this model closes.

(b) Speech level — the ending form has to match the level that was asked for

Table with columns: Instruction, Source, Hy-MT2-7B (base), K-MT-7B
Instruction	Source	Hy-MT2-7B (base)	K-MT-7B
호칭 `팀장님` · 합니다체	Did you finish reviewing the migration plan?	이주 계획 검토를 마치셨나요, 팀장님? ← 해요체	팀장님, 마이그레이션 계획 검토를 완료하셨습니까?

The base put the address term in, then ended the sentence in 해요체 when 합니다체 was requested. Speech level is a separate axis from address term, and both are checked.

(c) Relationship — the same person is addressed differently depending on the direction

Table with columns: Instruction, Source, Hy-MT2-7B (base), K-MT-7B
Instruction	Source	Hy-MT2-7B (base)	K-MT-7B
호칭 `최 매니저` · 관계 `상사→부하`	Escalate that urgent ticket to whoever is on call right now.	지금 근무 중인 담당자에게 그 긴급한 티켓을 즉시 전달해 주세요.	최 매니저, 그 긴급 티켓을 지금 교대 중인 사람에게 전달해 주세요.

Note the form: 최 매니저, not 최 매니저님. Superior-to-subordinate drops the honorific suffix, and the model follows the form it was given rather than politening it.

(d) Terminology — a term marked do-not-translate stays in English

Table with columns: Instruction, Source, Hy-MT2-7B (base), K-MT-7B
Instruction	Source	Hy-MT2-7B (base)	K-MT-7B
`egress` 보존 · 호칭 `대표님`	Cross-region egress alone accounted for 31% of last month's cloud bill.	교차 지역 외부 전송 비용만으로도 31%를 차지했습니다.	대표님, 지난달 클라우드 비용 중 교차 지역 egress만으로도 31%를 차지했습니다.
`CAPEX` 보존 · 호칭 `부장님`	GPU CAPEX overran the budget by ₩340,000,000 this quarter.	이번 분기에 GPU 관련 자본지출이 예산을 340,000,000원 초과했습니다.	부장님, 이번 분기 GPU CAPEX가 예산을 3억 4천만 원 초과했습니다.
보존 · 호칭

In a Korean finance or infra team, CAPEX, egress and accrual are the words people actually say. Translating them into 자본지출 or 외부 전송 is not an error in the dictionary sense — it just is not what the reader uses, and a glossary is how you say so.

(e) No instruction, no insertion — the negative control

Table with columns: Instruction, Source, Hy-MT2-7B (base), K-MT-7B
Instruction	Source	Hy-MT2-7B (base)	K-MT-7B
(none)	Can you check if the fleet manager has scheduled the weekend delivery routes?	직원님, 긴급 주문들을 위한 주말 배송 경로가 … 확인해 주실 수 있나요?	fleet manager가 주말 배송 경로를 계획했는지 확인해 주실 수 있나요?
(none)	The quarterly report deadline has been extended by two weeks.	분기별 보고서 제출 기한이 2주 연장되었습니다.	분기별 보고서 마감일이 2주 연장되었습니다.

⚠️ A model trained to insert address terms could learn to insert them everywhere. It does not — where no address term was requested, none appears. This axis was measured separately (n=296), and is the reason the §2 number is a compliance rate rather than an insertion rate.

1. General translation quality — a guardrail, not a win

This table exists to show that specializing for Korean did not break anything else. Point estimates are corpus-level.

Table with columns: Model, Params, Decoding, WMT24++ EN→KO, FLORES EN→KO, FLORES KO→EN
Model	Params	Decoding	WMT24++ EN→KO	FLORES EN→KO	FLORES KO→EN
K-MT-7B	7B	greedy	0.8894	0.9100	0.8891
Hy-MT2-7B (base)	7B	greedy	0.8874	0.9100	0.8890
LMT-60-8B	8B

⛔ Decoding differs by row. The primary numbers use uniform greedy (temp 0.0 · top_p 1.0 · top_k −1 · seed 0). LMT-60-8B is also reported at beam 5 because that is its official recommendation — measuring a model only under a weaker setting than it recommends would be unfair to it. Do not read the beam row and the greedy rows as one series.

Paired-item bootstrap (B=1000, seed 7), K-MT − base:

Table with columns: Benchmark, COMET-22 Δ, chrF++ Δ, BLEU Δ
Benchmark	COMET-22 Δ	chrF++ Δ	BLEU Δ
WMT24++ EN→KO (n=960)	+0.0020 [+0.0005, +0.0035]	+1.033 [+0.547, +1.540]	+0.792 [+0.286, +1.337]
FLORES EN→KO (n=1,012)	+0.0000 [−0.0008, +0.0009]	+0.480 [+0.116, +0.847]	+0.318 (tied)
FLORES KO→EN (n=1,012)	+0.0001 (tied)	−0.206 (tied)	−0.229 (tied)

⛔ Do not read the WMT24++ COMET +0.0020 as an improvement. The interval excludes zero, but it is 0.2 COMET points, and across nine comparisons against the base one crossing is what you expect by chance at α=0.05. This table says "not broken", nothing more.

⚠️ These confidence intervals are computed against the base only. Differences versus LMT-60-8B and TranslateGemma-4B are point estimates with no interval — this table does not support "K-MT beats those two", and we do not claim it.

2. Korean constraint compliance — in-domain

⚠️ This is a failure-mined benchmark. The eval set was mined from the base model's own failures, and the training data came from the same mining. Do not weigh it like §1.

Table with columns: Model, Korean constraint compliance, Semantic error (J1), FLORES Δ vs base
Model	Korean constraint compliance	Semantic error (J1)	FLORES Δ vs base
K-MT-7B (released merged weights)	98.92% (in-domain, not externally comparable)	3.69%	+0.085
K-MT-7B adapter, seeds 1·2·3	98.92 · 98.48 · 98.05%	3.04 · 3.25 · 1.74%	+0.087 · +0.045 · +0.034
Hy-MT2-7B (base)	23.86%	5.64%	—
variant without replay	99.6%	3.0%

"Constraint compliance" = the requested address term matches exactly and the speech level is not violated (deterministic checker, n=461). Semantic error is an independent judge's semantic-role verdict. FLORES Δ is the mean over 28 translation directions (n=28,336).

Replay is the mechanism, and the source of the replay text matters. Remove replay and Korean goes up but FLORES drops 0.43. Replace the replay source text with base-generated sentences and constraint compliance drops 1.66pp (interval excludes zero). Real human sentences as the source, with the base model's own outputs as the targets, is the combination that holds both axes.

3. Measured but not claimed

IFMTBench Korean (n=66) — only the rule-based axes, scored on this checkpoint. Read against the positive control (reference text submitted as the response): glossary 25/34 → 0.93 of ceiling, structured 9/16 → 0.75 (base: 0.81 · 0.75). Paired intervals include zero on both axes (glossary Δ+0.059 [0.000, 0.147] · structured Δ−0.063 [−0.188, 0.000]) — one item is worth 2.9pp and 6.3pp, so no ordering can be claimed.

⛔ The style and background axes were not scored for this checkpoint. They need an LLM judge, and using a self-hosted judge instead of the official default makes those numbers incomparable with published scores for other models, so they are omitted. Note also that this benchmark was built by the base model's authors, and its Korean items are not EN→KO-sourced.

Broad Korean SFT — a control trained broadly on public parallel corpora reached constraint compliance of 2.60 · 0.65 · 1.74% across three seeds, below the untrained base (23.9%). Pouring in more broad Korean data is not the answer. ⛔ We do not write "targeted wins at equal budget" — dataset tokens, consumed tokens and optimizer updates cannot all be matched at once, so no pair is fully matched.

4. Limitations

§1's public benchmarks do not contain what this model was built to fix. WMT24++ and FLORES have no items demanding address-term or speech-level instructions. The "tie" there is a fact about the benchmarks, not about the model. We have never demonstrated instruction-following superiority on public evidence — the OOD benchmark in §0 is meant to fill that gap.
No humans in the evaluation loop (machine-only): deterministic checkers plus a judge. ⛔ The two judges do not receive the same instructions — J2 gets the rubric, J1 gets the rubric plus seven calibration rules. Their κ (0.45–0.48) is therefore not an inter-annotator agreement, and J1∧J2 is used only as a conservative lower bound.
The Korean eval sets are n=461 (constraints) and n=66 (IFMTBench Korean).
The release is bf16 merged weights. Merging rounds away part of the LoRA delta — 33 of 288 free-generation items (11.5%) diverge lexically from the adapter path. The constraint verdict did not change on any of the 461 units, and the §2 numbers were measured on this released file, not on the adapter. ⚠️ Semantic error is 3.04% (adapter) vs 3.69% (merged) — 14 → 17 of 461. Paired-item bootstrap (B=1000, seed 7) puts that at +0.65pp [−0.65, +1.95], and McNemar over the 9 discordant pairs (6 merged-only, 3 adapter-only) gives p=0.51. The merged file is not measurably worse. ⛔ That is not the same as "no difference" — 9 discordant pairs cannot rule out a small one.
The base lineage reports training on OPUS/ParaCrawl/WMT-family corpora. WMT24++ extends WMT24, so both the base and this model may benefit on that benchmark — contamination in our favour.

5. Data provenance

Training: failure-driven targeted Korean data (self-generated, dual-judged) plus capability replay.
Replay source text is the Tatoeba CC0 export (CC0-1.0); the targets are the base model's own greedy outputs. ⛔ No FLORES text was used in training — FLORES is evaluation-only. Contamination check between training text and eval probes: 0 hits, 0 contained.
⛔ Rows derived from non-redistributable licensed corpora are rejected by the data builder (ContaminationError); none are in this model.
Evaluation: WMT24++ (google/wmt24pp, Apache-2.0) · FLORES-200 (CC-BY-SA-4.0) · IFMTBench (tencent/IFMTBench, CC-BY-4.0).

⚠️ We make no determination about whether model weights are a derivative work of training data. What is stated here is only which corpora were used under which licence, and what was not used.

6. Reproduction

Where each §2 row comes from: the released merged weights are the round-6 judgment; adapter seed 1 is round 4 (replay-source ablation, arm R1-s1), seeds 2·3 are round 5, the no-replay variant is arm B1, and the base-generated-replay variant is arm R2-s1. All §1 competitor numbers were measured by us on the same engine (transformers) and the same benchmark files — none are quoted from elsewhere.

The prompt format and greedy settings in §0 are exactly what produced the §2 numbers.

⚠️ Per-item scores and the eval set are not released. The Korean eval set (n=461) was mined from the base model's failures; releasing it would let the next model be trained against it. So §2 is not directly reproducible by a reader — what is reproducible is §1 (WMT24++, FLORES), which should give the same values under the decoding settings above. The forthcoming public OOD benchmark is planned to ship with its items.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider