oddadmix

50M-Egyptian-Translation-v1

README

License: apache-2.0

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False), scored with sacreBLEU:

Table with columns: Metric, Score
Metric	Score
sacreBLEU	24.35
chrF	52.38

The saved weights are the best checkpoint by validation loss (eval_loss=1.271, epoch 2 of 3).

Example translations

Real greedy-decoded outputs from the held-out set (English → model output):

Table with columns: English, Model output (Egyptian Arabic)
English	Model output (Egyptian Arabic)
Man, these things happen. Sometimes Liverpool loses the match, and other times Real Madrid wins it.	يا عم الحاجات دي بتحصل. ساعات ليفربول يخسر الماتش، وساعات ريال مدريد يكسبها.
I want grilled chicken, kofta on charcoal, and Pepsi.	أنا عايز فراخ مشوية وكفتة على الفحم وبابسي.
So, do we have to pay here?	يعني لازم ندفع هنا؟
I will be fine, don't worry	أنا هبقى كويس، متقلقش
What's my standard anyway? Forget about his standards, the awards' standards, and all that stuff.	إيه المعيار بتاعي أصلاً؟ سيبك من درجاته، معايير الجوائز، وكل الكلام ده.
Check yourself, Mr. Harry. You destroyed Tottenham Club—looks like you were the problem.	بص على نفسك يا أستاذ هاري. انت دمرت نادي توتنهام، شكلك كنت المشكلة.
Let's go, I'll get up, go to Rouh El Nagham, and call you on the phone.	يلا بينا، أنا هقوم، هقوم لروح النغم، وأكلمك في التليفون.
My question is, what kind of work is Guardiola doing with this one and that one? Where did he succeed?	سؤالي هو ايه الشغل اللي جوارديولا بيعمله مع ده وده؟ هو نجح فين؟

A larger set of 20 examples (with references) is included in eval_translation.json.

Usage

The model uses a ChatML prompt format with a fixed system instruction.

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/50M-Egyptian-Translation-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYSTEM = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللهجة المصرية العامية."

def translate(english: str) -> str:
    prompt = (
        f"<|im_start|>system\n{SYSTEM}<|im_end|>\n"
        f"<|im_start|>user\n{english.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("Man, these things happen. Sometimes Liverpool loses the match."))
# → يا عم الحاجات دي بتحصل. ساعات ليفربول يخسر الماتش.

Training

Base model: oddadmix/50M-2048-Emhotob (Llama arch, ~51.8M params)
Dataset: oddadmix/egyptian-translation-dataset-2.9-openai-batch (135K English/Egyptian-Arabic pairs)
Method: HuggingFace Trainer, ChatML format, prompt-masked cross-entropy (loss only on the Arabic assistant turn). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and the embeddings resized.
Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
Split: 132,282 train / 3,000 deterministic held-out eval (seed=42).

Out-of-domain evaluation — Egyptian Arabic Translation Benchmark

The results above are in-domain: a held-out split of the same corpus this model was trained on. The numbers below are out-of-domain — the same model scored on the Egyptian Arabic Translation Benchmark (oddadmix/egyptian-arabic-translation-benchmark, 319 English→Egyptian pairs written by a different annotator with different orthographic conventions).

Expect these to be substantially lower than the in-domain scores. That gap is the generalization penalty, not a regression — both numbers are real, they measure different things.

Table with columns: Metric, Score
Metric	Score
BLEU (`evaluate`, 0–1 — leaderboard metric)	0.1261
BLEU (sacrebleu, 0–100)	12.61
chrF	46.13
METEOR	0.3559

Decoding is deterministic greedy (do_sample=False, no repetition penalty), using the exact ChatML prompt format the model was trained with — the same protocol as every other number in this study.

Where this rung sits

Table with columns: Model, Params, BLEU (hf), BLEU (sacre), chrF, METEOR
Model	Params	BLEU (hf)	BLEU (sacre)	chrF	METEOR
5M v1	5.2M	0.0000	0.19	13.00	0.0558
5M v2	5.2M	0.0108	1.08	22.19	0.1352
10M v1	11.2M	0.0313

Reading these numbers

BLEU understates quality on this set. Scoring is against a single reference, so a correct translation that picks a different valid word is penalized — e.g. فريش vs the reference's طازة for "fresh", or التليفون اللي ضاع vs تليفونها الضايع for "her lost phone". Both are good Egyptian; only one matches the reference. chrF and METEOR track perceived quality more closely here.

At 319 rows, differences of roughly 1–2 BLEU between adjacent rungs are within noise.

These are small models — 5M to 50M parameters, orders of magnitude below the large systems typically evaluated on this benchmark. The result of interest is the scaling curve and per-parameter efficiency, not absolute rank against models 100–1000× the size.

Limitations

A 50M model: expect errors on rare / technical vocabulary and occasional repetition loops on longer generations (mitigate with repetition_penalty and no_repeat_ngram_size at inference).
Trained on conversational Egyptian; other Arabic dialects or formal MSA are out of scope.
Gender is disambiguated only from context, so it may default to masculine for ambiguous English inputs.

License

Apache-2.0, inherited from the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

oddadmix

Model Tree

Base

oddadmix/50M-2048-Emhotob

Fine-tuned

this model

Input Modalities

Text

Output Modalities