oddadmix

Nawah-50M-RAG-Support-2K

README

License: apache-2.0

Quick numbers

Judged by Gemma-4-31B on 500 held-out rows whose companies appear nowhere in training. Score is 0–2 over answerable rows; refusal is measured separately on the 54 unanswerable ones.

Table with columns: SFT, GRPO r1, this model (GRPO r2)
	SFT	GRPO r1	this model (GRPO r2)
judge score (0–2)	1.41	1.57	1.68
hallucination rate	16.0%	8.8%	4.6%
refusal on unanswerable	98.1%	98.1%	96.3%
answered when answerable	98.2%	98.2%	98.7%
chrF++ vs gold answer	65.79	68.07	70.12

For scale: the 31B teacher that generated the training data scores 1.96 and 66.47 chrF++ on the same split. This model reaches 86% of the teacher's judge score at 0.17% of its parameter count, and beats it on chrF++ — the latter because it was trained to mirror the corpus's answer style, not because it is the better model.

Usage

The contract is ChatML: the system turn carries the support preamble plus numbered passages, the user turn carries the customer's question.

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "oddadmix/Nawah-50M-RAG-Support-2K"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.float32).eval()

chunks = [
    "يمكن للعميل طلب إغلاق الحساب بتقديم طلب كتابي قبل 15 يوماً من نهاية الشهر الجاري، مع سداد كافة المستحقات المتأخرة.",
    "تمنح الشركة خصماً قدره 15% على الفاتورة السنوية عند تفعيل الدفع التلقائي عبر البطاقة البنكية.",
]
system = (
    "أنت مساعد خدمة عملاء. أجب عن سؤال العميل بالفصحى اعتماداً فقط على المعلومات "
    "التالية. إذا لم تكن الإجابة موجودة في المعلومات، فقل ذلك بأدب واعرض تحويل "
    "العميل إلى أحد موظفي خدمة العملاء."
    "\n\n" + "\n".join(f"[{i}] {c}" for i, c in enumerate(chunks, 1))
)

messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "كيف أغلق حسابي؟"},
]
enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=128, do_sample=False, pad_token_id=1)
print(tok.decode(out[0][enc["input_ids"].shape[-1]:], skip_special_tokens=True))

The system string above is verbatim the one used in training — the preamble, a blank line, then passages numbered [1], [2], … one per line. Grounding degrades if you paraphrase the preamble, drop the numbering, or interleave the question with the passages, so build it exactly this way every time.

GGUF quantizations

Two llama.cpp builds ship in gguf/ for CPU and on-device inference. The ChatML template is embedded, so llama.cpp's /v1/chat/completions works without extra configuration.

Table with columns: file, size, vs bf16
file	size	vs bf16
`gguf/nawah-rag-grpo-q8_0.gguf`	57.0 MB	−45%
`gguf/nawah-rag-grpo-q4_k_m.gguf`	38.2 MB	−63%

bash
llama-server -m nawah-rag-grpo-q8_0.gguf -c 2048

What quantization costs

The judged eval re-run on the same 500 rows with greedy decoding and no repetition penalty in every variant, so the only difference is the weights. These numbers are therefore not comparable to the sampled-decoding table above — compare them only to each other.

Table with columns: size, hallucination, judge score, refusal, chrF++, replies identical to bf16
	size	hallucination	judge score	refusal	chrF++	replies identical to bf16
bf16 (the safetensors here)	103.6 MB	4.2% (21)	1.70	96.3% (52/54)	70.79	—
F16 GGUF (reference only)	105.5 MB	3.4% (17)	1.70	96.3% (52/54)	70.68

Q8_0 is free — identical judge score and refusal accuracy at 45% of the size.

Q4_K_M is probably fine, but this eval cannot prove it is harmless. The F16 conversion is near-lossless (92% of replies byte-identical to bf16) yet still moved the hallucination count by 4 rows, in the opposite direction — so the noise floor on 500 rows is roughly ±4 rows, and Q4's 2-row change sits inside it. The one metric that moved consistently is refusal on unanswerable questions (52/54 → 50/54). Two rows on a 54-row denominator is not significant alone, but refusal is the behaviour GRPO exists to install, and it coincides with Q4 diverging from the reference on a third of all replies. Prefer Q8_0 unless the extra 19 MB matters; if it does, an imatrix-guided Q4 is the next thing to try.

Speed on CPU was the least interesting axis — 16.1 / 17.4 / 19.1 rows/s for F16 / Q8_0 / Q4_K_M. These prompts run ~1,000 tokens with short answers, so the work is dominated by prefill, where quantization helps least. Measured under load from another job; treat it as a lower bound.

How it was built

Pretraining. oddadmix/50M-2048-Emhotob — a Llama-architecture Arabic base model trained from scratch at a 2,048-token context.
Supervised fine-tuning on arabic-rag-support-25K: 27,427 grounded question/passage/answer rows over 6,873 fictional companies, distilled from Gemma-4-31B. 12% of rows are refusals — a plausible question whose answer is deliberately not in the passages.
GRPO round 1 on 1,990 fresh scenarios, company-disjoint from both the SFT train split and the eval split.
GRPO round 2 — this checkpoint — continued from the round-1 policy on a 9,959-row pool.

Both GRPO rounds used programmatic rewards with no judge in the loop:

number grounding — every number in the reply must occur in the gold passages or the question; numbers that appear only in distractor passages, or nowhere at all, are penalized
refusal correctness — refuse if and only if the answer is absent
chrF to the gold answer — anchors content and MSA fluency
sanity — Arabic-script prose of reasonable length

All GRPO training data was generated at scenario indices unused by SFT, and never drawn from the eval set.

The companion multi-turn corpora, arabic-rag-chat-30K and arabic-rag-chat-grpo-5K, are released alongside this model. They did not train this checkpoint — they train the multi-turn members of the same family, and are published so the line is reproducible end to end.

Training ran on a single consumer GPU.

Limitations

Single-turn only. It has no dialogue training. Give it one question and one set of passages. It will not track a conversation, resolve pronouns across turns, or handle a customer correcting themselves.
2,048 tokens is a hard ceiling — roughly 8 passages. There is no graceful degradation past it; positions beyond 2,048 were never trained.
Modern Standard Arabic only. The corpus is MSA. Dialectal input is out of distribution.
It answers, it does not compute. The model was trained to quote figures from the passages, never to derive new ones. Do not ask it to add up a bill.
4.6% of answers still contain an ungrounded number. This is a 51.8M- parameter model. Do not put it in front of customers without a human path.
Retrieval quality is the ceiling. Everything above assumes the gold passage is among the ones you supply. This model does not retrieve.
The judge is one model's opinion. Every score here comes from Gemma-4-31B scoring 500 rows. The hallucination and refusal columns are the more literal measurements; treat the 0–2 score as a comparison between rows of the table, not as an absolute.

Provenance

These weights are the GRPO round-2 policy, published from a pinned, verified revision. model.safetensors has SHA-256 fad8746c8a2ca82907fa09e2ac9c60534ba0ac31a94fa9c72c1832499e2a35fc.

A live demo of this model runs at oddadmix/Nawah-50M-RAG-Support-Demo.

Intended use

Research and prototyping of grounded Arabic question answering: retrieval evaluation harnesses, on-device support assistants, and a baseline for anyone studying how small a grounded RAG answerer can get. It has no general instruction tuning, no safety alignment, and no knowledge of its own — outside the passages you give it, it has nothing to say.

Citation

bibtex
@misc{nawah50mrag2k,
  title  = {Nawah-50M-RAG-Support-2K: a 51.8M-parameter grounded Arabic RAG answerer},
  author = {Wasfy, Ahmed},
  year   = {2026},
  url    = {https://huggingface.co/oddadmix/Nawah-50M-RAG-Support-2K}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

oddadmix

Model Tree

Base

oddadmix/50M-2048-Emhotob

Quantized

this model

Input Modalities

Text

Output Modalities