Nizam-LLM-7B-v0.1 API & Inference Endpoint

What's new in v0.3: it knows when to say "the texts don't say"

Many Saudi regulations leave a detail to someone else: "the minister sets the limit by decision", "as the regulation specifies". v0.2 read that sentence as if it were the answer. v0.3 sees that the detail is not there, says so, and still tells you what the article does settle.

Question: وفق نظام العمل، ما الحد الأقصى لعدد ساعات العمل الإضافية المسموح بها في السنة؟

Table with columns: v0.2, v0.3
v0.2	v0.3
يحدد الوزير بقرار منه الحد الأقصى لساعات العمل الإضافية التي يسمح بها في السنة، وفقاً للمادة (106) من نظام العمل.	لا تتضمن النصوص المرفقة الحد الأقصى لساعات العمل الإضافية. وتقرر المادة (106) أن يحدد الوزير بقرار منه الحد الأقصى لساعات العمل الإضافية التي يسمح بها في السنة … ولا يعني عدم وروده هنا أنه غير مقرر في مصدر آخر.
Sounds like an answer, but gives no number.	Clear that the number is set elsewhere, and by whom.

Why it worked: v0.3 adds a small, focused set of training examples for exactly this situation, each verified against its source article and reviewed by hand. That one change took correct declines from 11 to 24 out of 30, while answering accuracy held steady.

How we got here

Every version was trained, then tested on the same 286 questions. All scores below use today's grading, so they compare directly.

Table with columns: Run, What changed, GPU and precision, Score, Says "not in texts"
Run	What changed	GPU and precision	Score	Says "not in texts"
Base	ALLaM-7B, no training	–	16%	2/30
3	First training run	2× T4, 4-bit QLoRA (fp16)	56%	9/30
4 · v0.1	Questions that need two articles	2× T4, 4-bit QLoRA (fp16)	70%	11/30
5 · v0.2	Recovered Labor Law articles, full multi-paragraph answers	2× T4, 4-bit QLoRA (fp16)	74%	11/30
5b	Same data as run 5, full precision	1× H100, bf16 LoRA	74%	15/30
6	Stronger search model (e5-large)	1× H100, bf16 LoRA	71%	13/30
7 · v0.3	Examples where the detail is set elsewhere	1× L4, 4-bit QLoRA (bf16)	77%	24/30

Before run 3 came two runs on a smaller model (Qwen3-1.7B) and a test of three untrained base models. ALLaM was chosen because it covered the most legal content and handles Arabic text most efficiently.

What we learned: better data moved the model; bigger hardware did not. Runs 5, 5b and 6 used three set-ups, from T4s to an H100, and scored within noise of each other. Each real jump (runs 4, 5 and 7) came from new training examples aimed at one weakness the tests had shown.

We keep improving it

Nizam-LLM is trained and tested continuously. Every version is measured on the same test, with the same grading, against every version before it, and a new release ships only when it holds up.

How we test

143 questions across all 24 regulations, each asked twice: with just the right articles, and with them among related ones (286 cases).
Graded by fixed rules, not by another AI. An answer passes only if it states every required point, cites the right article, names the regulation, and declines when it should. Partial answers fail.
No leaks. Every training question is checked against the test questions before training. None overlap.
Two settings. The fixed set gives every model the same articles; the live set lets the search model pick them, as in real use.

Evaluation results

Every run on the same 286 cases, graded the same way.

Table with columns: Base, Run 3, v0.1, v0.2, Run 5b, Run 6, v0.3
	Base	Run 3	v0.1	v0.2	Run 5b	Run 6	v0.3
Overall pass	16.1%	56.3%	69.6%	74.1%	73.8%	70.6%	76.6%
Answerable questions (256)	17.2%	59.4%	73.4%	78.5%	76.6%	73.8%	76.2%
Correct declines (30)	2	9	11	11	15	13	24
Wrong declines on answerable	2	3	1	0	0	0	2
Right article cited	25.4%	91.0%	98.0%	97.3%	97.3%	96.5%	96.9%
Regulation named	50.4%	98.8%	99.6%	100%	100%	100%	100%
Required points covered	0.902	0.866	0.918	0.941	0.932	0.921	0.929

Table with columns: By question type, Cases, Base, Run 3, v0.1, v0.2, Run 5b, Run 6, v0.3
By question type	Cases	Base	Run 3	v0.1	v0.2	Run 5b	Run 6	v0.3
Direct rule	88	18.2%	68.2%	79.5%	84.1%	87.5%	84.1%	78.4%
Applied scenario	84	23.8%	60.7%	72.6%	73.8%	66.7%	63.1%	72.6%
Boundary condition	54	5.6%	59.3%	63.0%	75.9%	75.9%	77.8%	79.6%
Two-article	30	16.7%	30.0%	76.7%	80.0%	73.3%	66.7%	73.3%
No answer in the texts	30	6.7%	30.0%	36.7%	36.7%	50.0%	43.3%	80.0%

With live search (143 questions, the top 5 articles found by e5-large):

Table with columns: Run 6, v0.3
	Run 6	v0.3
Overall pass	78.3%	79.7%
Answerable questions (128)	81.3%	79.7%
Correct declines (15)	8	12
Right article cited	97.7%	98.4%

Against v0.2, the gain of v0.3 on questions without an answer is clear on a paired test (p = 0.001); on answerable questions the two differ within noise (p = 0.41).

Quick start

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "muhammaduxair/Nizam-LLM-7B"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")# On a T4 (no bf16) use dtype=torch.float16. To pin this release: revision="v0.3". system = ("أنت مساعد قانوني متخصص في أنظمة العمل في المملكة العربية السعودية. "          "بين يديك نصوص نظامية مرفقة. "          "أجب بإيجاز وبالعربية الفصحى معتمداً على هذه النصوص وحدها. "          "اذكر رقم المادة التي تستند إليها بصيغة «المادة (رقم)» مع اسم النظام. "          "إذا كان الحكم قد أُلغي أو حُذف، فقل ذلك صراحة. "          "إذا لم تتضمن النصوص المرفقة إجابة السؤال، فقل ذلك ولا تُجب من معرفتك "          "الخاصة ولا تُخمّن رقم مادة.")articles = ("المادة (53) — نظام العمل، المادة (53)\n"            "إذا كان العامل خاضعاً للتجربة، وجب النص على ذلك صراحة في عقد العمل، "            "وتحديد مدتها بوضوح، على ألا يزيد مجموع المدة في جميع الأحوال على مائة "            "وثمانين يوماً. ...")question = "وفق نظام العمل، ما الحد الأقصى لمدة التجربة؟" messages = [{"role": "system", "content": system},            {"role": "user", "content": f"النصوص النظامية:\n\n{articles}\n\nالسؤال: {question}"}]inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,                                       return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=512, do_sample=False)print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use the system prompt exactly as above: it is the one the model was trained on. Find the articles with multilingual-e5-large (revision 3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3), top 5; the corpus card has the full search code. Keep prompts within 3,072 tokens and decode greedily. The model needs about 16 GB of GPU memory.

Same behaviour as our notebook: the guardrail

The model is trained only to answer from the attached articles, or to say they don't hold the answer. Everything else runs in a small guardrail around it, published here as guardrail.py (standard-library Python, used unchanged by our notebook and demo):

questions outside Saudi labor and workforce regulations are refused before the model is called;
an answer that wrongly calls an attached article missing is asked again once;
a bare "the texts don't say" becomes a clear "no complete information" message;
citations of articles that were not in the prompt get a note.

Continuing the example above:

python

import importlib.utilfrom huggingface_hub import hf_hub_downloadfrom sentence_transformers import SentenceTransformer spec = importlib.util.spec_from_file_location("guardrail", hf_hub_download(repo, "guardrail.py"))guardrail = importlib.util.module_from_spec(spec)spec.loader.exec_module(guardrail)e5 = SentenceTransformer("intfloat/multilingual-e5-large",                         revision="3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3") def generate(turn):    chat = [{"role": "system", "content": system}, {"role": "user", "content": turn}]    inputs = tokenizer.apply_chat_template(chat, add_generation_prompt=True,                                           return_tensors="pt", return_dict=True).to(model.device)    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)    return tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip() embedding = e5.encode([f"query: {question}"], normalize_embeddings=True)[0]turn = f"النصوص النظامية:\n\n{articles}\n\nالسؤال: {question}"outcome = guardrail.guard(question, embedding, turn, ["المادة (53)"], generate)print(outcome.text, *outcome.notes, sep="\n")

The last list holds the headers of the articles in the prompt; with search, pass the headers of the five it found, and reuse the same e5 embedding for both. The scores on this card measure the model without the guardrail.

Good to know

It only knows what you give it. With the wrong articles in the prompt it cannot answer correctly, and it is not built to answer from memory.
About 1 answer in 4 misses a condition or point. If an answer cites an article that was not in the prompt, treat it as unsupported.
The texts are archived copies. The model does not know later amendments, or whether a provision is still in force.
Scope: Saudi labor and related workforce regulations, Arabic only. Off-topic questions are refused by the guardrail, not by the model.
Intended for research and prototypes in which a qualified person checks each answer. Not for legal advice to the public, or for automated decisions about anyone's rights.

Technical details

Data: about 4,800 verified, open-book training examples written from 845 of the corpus articles, including examples whose answer is deliberately not in the texts. Every example was checked against its source and reviewed before training, and none overlaps the test.
Training (v0.3): QLoRA on a 4-bit NF4 base with bf16 compute, prompts up to 3,072 tokens.
Release: the adapter was merged into the unquantized base in bf16 and checked token by token against the evaluated model before upload.
Evaluation: rule-based grading, greedy decoding, up to 1,024 new tokens. Earlier runs were re-graded from their saved answers with the current rules. The test is specific to this project, so its scores do not compare with other benchmarks.

Citation

bibtex

@misc{uzair2026nizamllm,  title  = {Nizam-LLM-7B: Grounded Question Answering over Saudi Labor Regulations},  author = {Muhammad Uzair},  year   = {2026},  howpublished = {\url{https://huggingface.co/muhammaduxair/Nizam-LLM-7B}},  note   = {Version 0.3}}

License and attribution

Apache-2.0, the license of the base model; see LICENSE. Built on ALLaM-7B-Instruct-preview by the National Center for Artificial Intelligence (SDAIA), now published by HUMAIN. The source texts are official publications of the Saudi Ministry of Human Resources and Social Development.

v0.2

v0.3

يحدد الوزير بقرار منه الحد الأقصى لساعات العمل الإضافية التي يسمح بها في السنة، وفقاً للمادة (106) من نظام العمل.

لا تتضمن النصوص المرفقة الحد الأقصى لساعات العمل الإضافية. وتقرر المادة (106) أن يحدد الوزير بقرار منه الحد الأقصى لساعات العمل الإضافية التي يسمح بها في السنة … ولا يعني عدم وروده هنا أنه غير مقرر في مصدر آخر.

Sounds like an answer, but gives no number.

Clear that the number is set elsewhere, and by whom.

Run

What changed

GPU and precision

Score

Says "not in texts"

Base

ALLaM-7B, no training

–

16%

2/30

First training run

2× T4, 4-bit QLoRA (fp16)

56%

9/30

4 · v0.1

Questions that need two articles

2× T4, 4-bit QLoRA (fp16)

70%

11/30

5 · v0.2

Recovered Labor Law articles, full multi-paragraph answers

2× T4, 4-bit QLoRA (fp16)

74%

11/30

Same data as run 5, full precision

1× H100, bf16 LoRA

74%

15/30

Stronger search model (e5-large)

1× H100, bf16 LoRA

71%

13/30

7 · v0.3

Examples where the detail is set elsewhere

1× L4, 4-bit QLoRA (bf16)

77%

24/30

Base

Run 3

v0.1

v0.2

Run 5b

Run 6

v0.3

Overall pass

16.1%

56.3%

69.6%

74.1%

73.8%

70.6%

76.6%

Answerable questions (256)

17.2%

59.4%

73.4%

78.5%

76.6%

73.8%

76.2%

Correct declines (30)

Wrong declines on answerable

Right article cited

25.4%

91.0%

98.0%

97.3%

96.5%

96.9%

Regulation named

50.4%

98.8%

99.6%

100%

Required points covered

0.902

0.866

0.918

0.941

0.932

0.921

0.929

By question type

Cases

Base

Run 3

v0.1

v0.2

Run 5b

Run 6

v0.3

Direct rule

18.2%

68.2%

79.5%

84.1%

87.5%

84.1%

78.4%

Applied scenario

23.8%

60.7%

72.6%

73.8%

66.7%

63.1%

72.6%

Boundary condition

5.6%

59.3%

63.0%

75.9%

77.8%

79.6%

Two-article

16.7%

30.0%

76.7%

80.0%

73.3%

66.7%

73.3%

No answer in the texts

6.7%

30.0%

36.7%

50.0%

43.3%

80.0%

Run 6

v0.3

Overall pass

78.3%

79.7%

Answerable questions (128)

81.3%

79.7%

Correct declines (15)

Right article cited

97.7%

98.4%

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "muhammaduxair/Nizam-LLM-7B"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")# On a T4 (no bf16) use dtype=torch.float16. To pin this release: revision="v0.3". system = ("أنت مساعد قانوني متخصص في أنظمة العمل في المملكة العربية السعودية. " "بين يديك نصوص نظامية مرفقة. " "أجب بإيجاز وبالعربية الفصحى معتمداً على هذه النصوص وحدها. " "اذكر رقم المادة التي تستند إليها بصيغة «المادة (رقم)» مع اسم النظام. " "إذا كان الحكم قد أُلغي أو حُذف، فقل ذلك صراحة. " "إذا لم تتضمن النصوص المرفقة إجابة السؤال، فقل ذلك ولا تُجب من معرفتك " "الخاصة ولا تُخمّن رقم مادة.")articles = ("المادة (53) — نظام العمل، المادة (53)\n" "إذا كان العامل خاضعاً للتجربة، وجب النص على ذلك صراحة في عقد العمل، " "وتحديد مدتها بوضوح، على ألا يزيد مجموع المدة في جميع الأحوال على مائة " "وثمانين يوماً. ...")question = "وفق نظام العمل، ما الحد الأقصى لمدة التجربة؟" messages = [{"role": "system", "content": system}, {"role": "user", "content": f"النصوص النظامية:\n\n{articles}\n\nالسؤال: {question}"}]inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=512, do_sample=False)print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

import importlib.utilfrom huggingface_hub import hf_hub_downloadfrom sentence_transformers import SentenceTransformer spec = importlib.util.spec_from_file_location("guardrail", hf_hub_download(repo, "guardrail.py"))guardrail = importlib.util.module_from_spec(spec)spec.loader.exec_module(guardrail)e5 = SentenceTransformer("intfloat/multilingual-e5-large", revision="3d7cfbdacd47fdda877c5cd8a79fbcc4f2a574f3") def generate(turn): chat = [{"role": "system", "content": system}, {"role": "user", "content": turn}] inputs = tokenizer.apply_chat_template(chat, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) out = model.generate(**inputs, max_new_tokens=512, do_sample=False) return tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip() embedding = e5.encode([f"query: {question}"], normalize_embeddings=True)[0]turn = f"النصوص النظامية:\n\n{articles}\n\nالسؤال: {question}"outcome = guardrail.guard(question, embedding, turn, ["المادة (53)"], generate)print(outcome.text, *outcome.notes, sep="\n")

@misc{uzair2026nizamllm, title = {Nizam-LLM-7B: Grounded Question Answering over Saudi Labor Regulations}, author = {Muhammad Uzair}, year = {2026}, howpublished = {\url{https://huggingface.co/muhammaduxair/Nizam-LLM-7B}}, note = {Version 0.3}}

Nizam-LLM-7B-v0.1

README

What's new in v0.3: it knows when to say "the texts don't say"

How we got here

We keep improving it

How we test

Evaluation results

Quick start

Same behaviour as our notebook: the guardrail

Good to know

Technical details

Citation

License and attribution

Explore FriendliAI today

README

What's new in v0.3: it knows when to say "the texts don't say"

How we got here

We keep improving it

How we test

Evaluation results

Quick start

Same behaviour as our notebook: the guardrail

Good to know

Technical details

Citation

License and attribution