xmanii

Mizan-1.5-4B

README

License: apache-2.0

What's new in 1.5

v1's mistakes were found on about 18k real Persian records, taken from public datasets' training splits and never from the benchmark. A blind LLM review diagnosed each one: is the model wrong, the dataset label wrong, or the case ambiguous? Noisy labels were dropped. For every real model mistake, 3 new examples teaching the same lesson were written:

the same failure in new wording;
a near-identical text whose answer flips (a contrast pair);
a fresh question wording.

That added 1,362 new examples plus 454 confirmed hard real rows.

Fixed weak spots:

Treating political criticism as offensive.
Answering positive or negative for aspects a review never mentions.
Calling a claim "supported" when it swaps a detail.
Confusing alarms and reminders.

Table with columns: Task, v1, v1.5
Task	v1	v1.5
Aspect questions (ParsiNLU)	71.5	80.0
Intent routing (MASSIVE)	72.5	78.5
Offensive content (Pars-OFF)	80.8	86.5
Claim support (FarsTail)	83.5	86.5
Article topics (Digikala Mag, never used for training)	80.8	81.5
Mean	78.7	82.0
Calibration error (ECE)	0.110

Digikala Mag was deliberately kept out of the whole process, as an overfitting check. It went up too.

v1 is still available under the v1 tag of this repo.

Quick look

json

{  "state": {"message": "سلام، دو بار از کارتم پول کم شده ولی سفارشم ثبت نشده. لطفا سریع پیگیری کنید",            "customer": {"orders": 14, "vip": true}},  "questions": {    "team":    {"type": "choice", "instructions": "کدام تیم باید این پیام را بررسی کند؟",                "criteria": {"billing": "مالی و بازگشت وجه", "delivery": "ارسال و پیک",                             "technical": "خطای اپ یا سایت", "other": "سایر موارد"}},    "urgency": {"type": "score", "instructions": "این درخواست چقدر فوری است؟", "criteria": ["عادی", "مهم", "خیلی فوری"]},    "refund":  {"type": "noul",  "instructions": "آیا مشتری درخواست بازگشت وجه دارد؟"}  }}

Each answer comes back with probabilities over every option: choice returns choice, confidence and probabilities; score returns score, legend and probabilities; noul returns the probability of "yes". Questions can be in Persian or English, and they can be new questions the model has never seen.

Inference

llama.cpp (recommended, CPU or GPU)

Needs llama.cpp b11443 or newer.

bash

huggingface-cli download xmanii/Mizan-1.5-4B-GGUF mizan-1.5-4b-Q4_K_M.gguf --local-dir .llama-server -m mizan-1.5-4b-Q4_K_M.gguf -c 8192 -b 8192 -ub 8192 -ngl 99 --port 8080

bash

curl http://localhost:8080/v1/systemone -H 'Content-Type: application/json' -d '{  "state": {"review": "غذا سرد رسید و پیک هم خیلی بداخلاق بود"},  "questions": {"sentiment": {"type": "choice", "instructions": "حس کلی مشتری چیست؟",                "criteria": {"positive": "راضی", "negative": "ناراضی"}}}}'

A decision reads the whole input in one batch, so -ub sets the longest input you can send. The GGUF carries the decision head and the calibrated temperatures.

Python (Unsloth)

python

from unsloth import FastDecisionModel model, processor = FastDecisionModel.from_pretrained("xmanii/Mizan-1.5-4B", max_seq_length=4096)FastDecisionModel.for_inference(model)answers = FastDecisionModel.predict(model, processor, state, questions)print(answers["team"]["choice"], answers["team"]["probabilities"])

This applies the calibrated temperatures, so it gives the same answers as the GGUF. Tested with unsloth==2026.10.3.

Plain transformers

python

import sysfrom huggingface_hub import snapshot_download path = snapshot_download("xmanii/Mizan-1.5-4B")sys.path.insert(0, path)from joint_schema_model import load_release_model, systemone model, processor = load_release_model(path, device="cuda")print(systemone(model, processor, {"model": "mizan", "state": state, "questions": questions}))

This path returns uncalibrated probabilities: the top answer is the same, but confidences are sharper. To calibrate them, divide each question's logits by unsloth_decision_config.json's temperature for its type (order: choice, score, noul).

Evaluation on real Persian data

3,000 real, human-labeled records from 7 public Persian datasets. None was used for training, and every model gets identical rows and questions.

Table with columns: Task, Mizan 1.5, Mizan 1.5 Q4, Mizan 1.0, Clef-Flash 9B, Clef 27B, Jev (API), Laya 322M
Task	Mizan 1.5	Mizan 1.5 Q4	Mizan 1.0	Clef-Flash 9B	Clef 27B	Jev (API)	Laya 322M
Food-review sentiment (SnappFood)	87.5	87.5	87.0	88.0	85.8	86.0	80.8
Product recommend (Digikala)	73.8	76.0	74.8

per task size vs accuracy

Reading it honestly:

Mizan 1.5 is within 1–2 points of models 2–7× its size.
It matches or beats Jev on aspect questions, intent routing and article topics.
Its biggest remaining gap is strict claim checking (FarsTail).
The mistake mining used the training splits of the same public datasets, so part of the gain on those six tasks may come from familiarity with their style. The untouched Digikala Mag result suggests most of it is real.
In the blind review, about half of v1's SnappFood and Digikala "errors" turned out to be wrong dataset labels, so roughly 87% on SnappFood is close to that dataset's real ceiling.
The datasets are public, so the larger models may have seen some of them in training.

Confidence gating works. Auto-decide the records Mizan is sure about, and send the rest to a human:

gating

Training

Table

Base	`Qwen/Qwen3.5-4B` (via `unsloth/Qwen3.5-4B`), plain LM turned into a decision model
Head	Clef joint-schema head (4 layers, width 512, 2 evidence-routing layers), trained from scratch
Method	Unsloth `FastDecisionModel` + `DecisionTrainer`; LoRA r=64 on all linear layers (lr 1e-4), head lr 3e-4; LoRA merged after training
Schedule	1 epoch, 1,091 steps, batch 16, cosine, max length 4,096
Calibration	Per-question-type temperatures fitted on a held-out slice
Compute	1× H100, 37 min on Modal

Data (17.6k decisions, about 70% English and 30% Persian):

English: Unsloth's decision mixture (BANKING77, CLINC150, MNLI, SNLI, WANLI, BoolQ, AG News, SST-5, MMLU, CommonsenseQA, ARC, prompt injections) plus typed-decisions, to keep the model general.
Persian business workflows: 6,660 records written to read like real Iranian users (colloquial text, typos, Finglish, emoji), with realistic context such as order data, seller stats, a bot policy or a job's requirements. Workflows: comment moderation, product reviews, app feedback, support routing, SMS fraud, marketplace listings, sales leads, chatbot guardrails, RAG relevance and CV screening.
- Labels: each record was labeled blind by a hosted decision model, and disagreements with the intended answer went to a blind LLM review. The main question trains on the reviewed intended answer.
- Question variety: 6 Persian paraphrases per question and rotated option IDs, plus 1,568 custom, record-specific questions.
New in 1.5: 1,362 examples written around v1's real mistakes, plus 454 confirmed hard real rows (see above).
Leak control: each record and its copies are split together, training rows are decontaminated against every eval set, and benchmark rows never enter the mistake search.

Limitations

Most Persian training data is synthetic. Evaluate on 200–300 of your own labeled examples before automating.
Weakest area: strict entailment (is a claim fully supported?). Large intent taxonomies are better but still behind Clef-Flash.
Text only. Image inputs are coming in Mizan v2.

License

Apache-2.0, the same as the Qwen3.5 base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

xmanii

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What's new in 1.5

the same failure in new wording;
a near-identical text whose answer flips (a contrast pair);
a fresh question wording.

That added 1,362 new examples plus 454 confirmed hard real rows.

Fixed weak spots:

Treating political criticism as offensive.
Answering positive or negative for aspects a review never mentions.
Calling a claim "supported" when it swaps a detail.
Confusing alarms and reminders.

Table with columns: Task, v1, v1.5
Task	v1	v1.5
Aspect questions (ParsiNLU)	71.5	80.0
Intent routing (MASSIVE)	72.5	78.5
Offensive content (Pars-OFF)	80.8	86.5
Claim support (FarsTail)	83.5	86.5
Article topics (Digikala Mag, never used for training)	80.8	81.5
Mean	78.7	82.0
Calibration error (ECE)	0.110

Digikala Mag was deliberately kept out of the whole process, as an overfitting check. It went up too.

v1 is still available under the v1 tag of this repo.

Quick look

json

{  "state": {"message": "سلام، دو بار از کارتم پول کم شده ولی سفارشم ثبت نشده. لطفا سریع پیگیری کنید",            "customer": {"orders": 14, "vip": true}},  "questions": {    "team":    {"type": "choice", "instructions": "کدام تیم باید این پیام را بررسی کند؟",                "criteria": {"billing": "مالی و بازگشت وجه", "delivery": "ارسال و پیک",                             "technical": "خطای اپ یا سایت", "other": "سایر موارد"}},    "urgency": {"type": "score", "instructions": "این درخواست چقدر فوری است؟", "criteria": ["عادی", "مهم", "خیلی فوری"]},    "refund":  {"type": "noul",  "instructions": "آیا مشتری درخواست بازگشت وجه دارد؟"}  }}

{  "state": {"message": "سلام، دو بار از کارتم پول کم شده ولی سفارشم ثبت نشده. لطفا سریع پیگیری کنید",            "customer": {"orders": 14, "vip": true}},  "questions": {    "team":    {"type": "choice", "instructions": "کدام تیم باید این پیام را بررسی کند؟",                "criteria": {"billing": "مالی و بازگشت وجه", "delivery": "ارسال و پیک",                             "technical": "خطای اپ یا سایت", "other": "سایر موارد"}},    "urgency": {"type": "score", "instructions": "این درخواست چقدر فوری است؟", "criteria": ["عادی", "مهم", "خیلی فوری"]},    "refund":  {"type": "noul",  "instructions": "آیا مشتری درخواست بازگشت وجه دارد؟"}  }}

Inference

llama.cpp (recommended, CPU or GPU)

Needs llama.cpp b11443 or newer.

bash

huggingface-cli download xmanii/Mizan-1.5-4B-GGUF mizan-1.5-4b-Q4_K_M.gguf --local-dir .llama-server -m mizan-1.5-4b-Q4_K_M.gguf -c 8192 -b 8192 -ub 8192 -ngl 99 --port 8080

bash

curl http://localhost:8080/v1/systemone -H 'Content-Type: application/json' -d '{  "state": {"review": "غذا سرد رسید و پیک هم خیلی بداخلاق بود"},  "questions": {"sentiment": {"type": "choice", "instructions": "حس کلی مشتری چیست؟",                "criteria": {"positive": "راضی", "negative": "ناراضی"}}}}'

A decision reads the whole input in one batch, so -ub sets the longest input you can send. The GGUF carries the decision head and the calibrated temperatures.

Python (Unsloth)

python

from unsloth import FastDecisionModel model, processor = FastDecisionModel.from_pretrained("xmanii/Mizan-1.5-4B", max_seq_length=4096)FastDecisionModel.for_inference(model)answers = FastDecisionModel.predict(model, processor, state, questions)print(answers["team"]["choice"], answers["team"]["probabilities"])

This applies the calibrated temperatures, so it gives the same answers as the GGUF. Tested with unsloth==2026.10.3.

Plain transformers

python

import sysfrom huggingface_hub import snapshot_download path = snapshot_download("xmanii/Mizan-1.5-4B")sys.path.insert(0, path)from joint_schema_model import load_release_model, systemone model, processor = load_release_model(path, device="cuda")print(systemone(model, processor, {"model": "mizan", "state": state, "questions": questions}))

Evaluation on real Persian data

3,000 real, human-labeled records from 7 public Persian datasets. None was used for training, and every model gets identical rows and questions.

Table with columns: Task, Mizan 1.5, Mizan 1.5 Q4, Mizan 1.0, Clef-Flash 9B, Clef 27B, Jev (API), Laya 322M
Task	Mizan 1.5	Mizan 1.5 Q4	Mizan 1.0	Clef-Flash 9B	Clef 27B	Jev (API)	Laya 322M
Food-review sentiment (SnappFood)	87.5	87.5	87.0	88.0	85.8	86.0	80.8
Product recommend (Digikala)	73.8	76.0	74.8

per task size vs accuracy

Reading it honestly:

Mizan 1.5 is within 1–2 points of models 2–7× its size.
It matches or beats Jev on aspect questions, intent routing and article topics.
Its biggest remaining gap is strict claim checking (FarsTail).
The mistake mining used the training splits of the same public datasets, so part of the gain on those six tasks may come from familiarity with their style. The untouched Digikala Mag result suggests most of it is real.
In the blind review, about half of v1's SnappFood and Digikala "errors" turned out to be wrong dataset labels, so roughly 87% on SnappFood is close to that dataset's real ceiling.
The datasets are public, so the larger models may have seen some of them in training.

Confidence gating works. Auto-decide the records Mizan is sure about, and send the rest to a human:

gating

Training

Table

Base	`Qwen/Qwen3.5-4B` (via `unsloth/Qwen3.5-4B`), plain LM turned into a decision model
Head	Clef joint-schema head (4 layers, width 512, 2 evidence-routing layers), trained from scratch
Method	Unsloth `FastDecisionModel` + `DecisionTrainer`; LoRA r=64 on all linear layers (lr 1e-4), head lr 3e-4; LoRA merged after training
Schedule	1 epoch, 1,091 steps, batch 16, cosine, max length 4,096
Calibration	Per-question-type temperatures fitted on a held-out slice
Compute	1× H100, 37 min on Modal

Data (17.6k decisions, about 70% English and 30% Persian):

English: Unsloth's decision mixture (BANKING77, CLINC150, MNLI, SNLI, WANLI, BoolQ, AG News, SST-5, MMLU, CommonsenseQA, ARC, prompt injections) plus typed-decisions, to keep the model general.
Persian business workflows: 6,660 records written to read like real Iranian users (colloquial text, typos, Finglish, emoji), with realistic context such as order data, seller stats, a bot policy or a job's requirements. Workflows: comment moderation, product reviews, app feedback, support routing, SMS fraud, marketplace listings, sales leads, chatbot guardrails, RAG relevance and CV screening.
- Labels: each record was labeled blind by a hosted decision model, and disagreements with the intended answer went to a blind LLM review. The main question trains on the reviewed intended answer.
- Question variety: 6 Persian paraphrases per question and rotated option IDs, plus 1,568 custom, record-specific questions.
New in 1.5: 1,362 examples written around v1's real mistakes, plus 454 confirmed hard real rows (see above).
Leak control: each record and its copies are split together, training rows are decontaminated against every eval set, and benchmark rows never enter the mistake search.

Limitations

Most Persian training data is synthetic. Evaluate on 200–300 of your own labeled examples before automating.
Weakest area: strict entailment (is a claim fully supported?). Large intent taxonomies are better but still behind Clef-Flash.
Text only. Image inputs are coming in Mizan v2.

License

Apache-2.0, the same as the Qwen3.5 base model.