GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
rikai-translate
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
rikai-translate
Model Tree
cyberagent/CAT-Translate-0.8b
Input Modalities
Output Modalities
Supported Functionality
| Repo | Format | Size | Use |
|---|---|---|---|
| rikai jp<->eng Bench 265 .8B (this repo, root) | safetensors, fp16 | ~1.5 GB | transformers, with the prefix mask from Usage |
rikai jp<->eng Bench 265 .8B mlx-8bit (this repo, mlx-8bit/) | MLX, 8-bit affine, group size 64 | 812 MB | mlx_lm on Apple silicon. All of our own numbers below are for this build, decoded with the prefix mask. |
rikAI .8B seems to perform between CAT-Translate-1.4B and CAT-Translate-3.3B on most benchmarks.
| rikai jp<->eng Bench 265 .8B | CAT-0.8B | CAT-1.4B | CAT-3.3B-β | CAT-3.3B | CAT-7B | |
|---|---|---|---|---|---|---|
| 26-domain gate, CometKiwi ↑ | 0.8384 | 0.8304 | 0.8353 | 0.8420 | 0.8430 | 0.8430 |
| 26-domain gate, MetricX-24-QE error ↓ | 2.85 | 3.29 | 3.04 | 2.89 | 2.76 | 2.78 |
| Gate cells where rikai jp<->eng Bench 265 .8B is significantly better / worse (MetricX-24-QE + COMET-22, of 52) | — | 32 / 0 | 11 / 0 | 4 / 3 | 3 / 10 | 1 / 14 |
| 26-domain gate, MetricX-24 error ↓ | 3.34 | 3.83 | 3.51 | 3.31 | 3.14 | 3.14 |
| 26-domain gate, BLEU, mean over sets ↑ | 22.02 | 21.13 | 19.41 | 23.14 | 23.22 | 26.07 |
| 26-domain gate, chrF++, mean over sets ↑ | 36.88 | 35.62 | 35.32 | 38.06 | 38.31 | 40.17 |
| Blind 500, CometKiwi ↑ | 0.8020 | 0.7944 | 0.7992 | 0.8064 | 0.8046 | 0.8061 |
| Blind 500, MetricX-24-QE (1 − err/25) ↑ | 0.8586 | 0.8391 | 0.8493 | 0.8562 | 0.8604 | 0.8663 |
| Blind 500 en→ja, MetricX-24-QE ↑ | 0.8436 | 0.8101 | 0.8279 | 0.8286 | 0.8428 | 0.8503 |
CAT-3.3B-β vs CAT-3.3B. These are two separate CyberAgent releases of the same 3.3B size. In this table the β release trails CAT-3.3B on 7 of the 8 score rows, so matching β is not the same as matching CAT-3.3B.
This is our main test battery: 1,550 human-referenced items in 26 sets.
| Direction | Sets |
|---|---|
| ja→en (820 items) | BSD business dialogue (whole documents and single lines), court interpreting, EJMMT medical, FLORES-200 (sentences and multi-sentence articles), WMT23 (ads, e-commerce, news, Q&A, user reviews), Tatoeba, OpenMantra manga pages |
| en→ja (730 items) | BSD, EJMMT, FLORES-200, NTREX-128, WMT23 (Mastodon, news, speech, user reviews), WMT24++ paragraphs (literary, social, news, speech), Tatoeba |
How it was run:
Per-domain results for every metric →
The only thing benchmaxing here is me, so this model was also trained with an LLM judge.
Judge: Claude Sonnet 5.5, on 500 blind items it never saw in training (350 ja→en, 150 en→ja).
Prompt: it sees the source, a reference marked as "one acceptable translation", and one model's translation, never two side by side. It scores it 0–100 (100 perfect, 75 minor errors, 50 a major error, 25 meaning wrong, 0 unusable) and lists any errors as minor, major or critical. Every item is judged twice, in differently shuffled batches, and the two scores are averaged.
| Model | Score (0–100) | Translations with a critical error |
|---|---|---|
| rikai jp<->eng Bench 265 .8B | 60.3 | 6.2% |
| CAT-0.8B | 58.5 | 7.4% |
| CAT-1.4B | 60.6 | 7.8% |
| LFM2-350M-ENJP-MT | 51.8 | 13.2% |
| LFM2.5-1.2B-JP | 50.6 | 17.2% |
| TinySwallow-1.5B | 52.0 | 15.6% |
| Gemma-2-2B-jpn | 52.2 | 19.6% |
| Gemma-2-2B-jpn-translate | 55.4 | 14.0% |
| HY-MT1.5-1.8B | 58.0 | 7.2% |
The prompt and chat template are the same as CAT-Translate's:
text
Translate the following {src_lang} text into {tgt_lang}.{src_text}It runs on a custom prefix mask, so the prompt gets read in both directions before the first word comes out. Here's an MLX example (Apple silicon, 8-bit build):
python
import mlx.core as mx, mlx_lm.models.llama as llamafrom mlx_lm import loadfrom mlx_lm.models.cache import make_prompt_cachefrom huggingface_hub import snapshot_download path = snapshot_download("rikai-translate/rikai-jp-eng-Bench-265-0.8B", allow_patterns="mlx-8bit/*") + "/mlx-8bit"model, tok = load(path)causal_mask = llama.create_attention_mask def translate(text, src="Japanese", tgt="English", max_tokens=1024): msg = [{"role": "user", "content": f"Translate the following {src} text into {tgt}.\n\n{text}"}] ids = tok.apply_chat_template(msg, add_generation_prompt=True) cache = make_prompt_cache(model) llama.create_attention_mask = lambda *a, **k: None # prompt[:-1] attends in both directions model(mx.array([ids[:-1]]), cache=cache) llama.create_attention_mask = causal_mask # generation stays causal logits, out = model(mx.array([ids[-1:]]), cache=cache), [] for _ in range(max_tokens): t = mx.argmax(logits[0, -1]).item() if t in tok.eos_token_ids: break out.append(t); logits = model(mx.array([[t]]), cache=cache) return tok.decode(out).strip() print(translate("駅前の新しいパン屋、朝七時から開いてるらしいよ。"))# The new bakery in front of the station seems to be open from 7 a.m.MIT.
If you use this model, please cite the base model:
bibtex
@misc{jinnai2026cattranslatebuildingcompactopensource, title={CAT-Translate: Building Compact Open-Source Models for Japanese-English Translation}, author={Yuu Jinnai}, year={2026}, eprint={2606.21413}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2606.21413},}current Bench PR 265
$5 USD if anyone can build a decoder model that beats rikai bench 265 .8B at .8B or less.
import mlx.core as mx, mlx_lm.models.llama as llamafrom mlx_lm import loadfrom mlx_lm.models.cache import make_prompt_cachefrom huggingface_hub import snapshot_download path = snapshot_download("rikai-translate/rikai-jp-eng-Bench-265-0.8B", allow_patterns="mlx-8bit/*") + "/mlx-8bit"model, tok = load(path)causal_mask = llama.create_attention_mask def translate(text, src="Japanese", tgt="English", max_tokens=1024): msg = [{"role": "user", "content": f"Translate the following {src} text into {tgt}.\n\n{text}"}] ids = tok.apply_chat_template(msg, add_generation_prompt=True) cache = make_prompt_cache(model) llama.create_attention_mask = lambda *a, **k: None # prompt[:-1] attends in both directions model(mx.array([ids[:-1]]), cache=cache) llama.create_attention_mask = causal_mask # generation stays causal logits, out = model(mx.array([ids[-1:]]), cache=cache), [] for _ in range(max_tokens): t = mx.argmax(logits[0, -1]).item() if t in tok.eos_token_ids: break out.append(t); logits = model(mx.array([[t]]), cache=cache) return tok.decode(out).strip() print(translate("駅前の新しいパン屋、朝七時から開いてるらしいよ。"))# The new bakery in front of the station seems to be open from 7 a.m.@misc{jinnai2026cattranslatebuildingcompactopensource, title={CAT-Translate: Building Compact Open-Source Models for Japanese-English Translation}, author={Yuu Jinnai}, year={2026}, eprint={2606.21413}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2606.21413},}