Model details
Table with columns: Property, Value| Property | Value |
|---|
| Base model | Qwen/Qwen3.5-0.8B |
| Architecture | Qwen3.5 conditional-generation model |
| Fine-tuning | LoRA, rank 16; adapter merged into base weights |
| Training strategy | One epoch of LoRA fine-tuning |
| Training data | Deduplicated Arabic decision and reasoning datasets |
| License | Apache-2.0 (same as the base model) |
Evaluation
The held-out test set contains 10,419 examples. The results below use
constrained candidate-label scoring: for each example, the evaluator scores the
allowed labels and selects the highest-scoring one. This measures decision
accuracy when the candidate labels are known; it is not free-form generation
accuracy.
The primary evaluation prompt does not include dialect metadata.
Accuracy by question type
Table with columns: Type, Correct, Examples, Accuracy| Type | Correct | Examples | Accuracy |
|---|
| Choice | 6,901 | 7,016 | 98.36% |
Yes/no (noul) | 2,265 | 2,772 | 81.71% |
| Score | 601 | 631 | 95.25% |
The yes/no subset is the main weakness. Consider this when choosing the model
for arithmetic-heavy or binary decisions.
vLLM latency sample
A separate 100-example sample was benchmarked using vLLM 0.32.0 on an NVIDIA
A40. Requests were sequential, used bfloat16 and constrained candidate labels,
and ran locally after five warm-up requests.
Table with columns: Metric, Result| Metric | Result |
|---|
| Mean latency | 61.3 ms/request |
| Median latency | 59.9 ms/request |
| 95th-percentile latency | 71.3 ms/request |
| Sequential throughput | 16.3 requests/second |
These are local, single-request measurements; latency and throughput vary with
hardware, prompt length, concurrency, and serving configuration. The sample's
93/100 accuracy is not the full held-out test score above.
Quick start
Install the runtime dependencies (use Transformers 5.17 or newer for Qwen3.5):
pip install "transformers>=5.17" torch
Load Ara-JEV directly with standard Transformers. The predict function keeps
prompt construction and constrained option scoring together:
import torchfrom transformers import AutoModelForImageTextToText model_id = "AhmedReda25300/Ara-JEV"model = AutoModelForImageTextToText.from_pretrained( model_id, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,).eval() result = model.predict( state="الاجتماع يبدأ الساعة العاشرة صباحًا. دلوقتي الساعة تسعة.", question="هل بدأ الاجتماع؟", options=["نعم", "لا"],)print(result) # لا score = model.predict( state=( "بلاغ الدعم بيقول إن بيانات شخصية ظهرت لمستخدم مش صاحبها. " "التواصل كان عن طريق تذكرة من لوحة التحكم، والعميل طلب تحديث لما يظهر جديد." ), question="قيّم أولوية التعامل مع البلاغ.", options=[ "منخفضة: يتابع في المسار المعتاد", "متوسطة: محتاجة متابعة قريبة", "عالية: تتصعّد بسرعة بسبب الأثر", ],)print(score) # 2 (integer index of the selected option)
The prediction helper is built into the Transformers model class, so prompt
construction and option scoring are handled internally. No dialect metadata is
required. Arabic Yes/No options return the chosen option text; other option
lists return the selected option's zero-based integer index. trust_remote_code
is required to load Ara-JEV's custom predict method from the model repository.
vLLM
vllm serve AhmedReda25300/Ara-JEV \ --dtype bfloat16 \ --max-model-len 8192
The checkpoint was tested with vLLM 0.32.0. Use the chat-completions API and
constrain the response to the labels allowed for each task when supported by
your serving client.
Intended use and limitations
- Intended for Arabic decision and routing tasks similar to the evaluation data.
- Performance on dialects, domains, and tasks not represented in evaluation has
not been established.
- For the reported evaluation, candidate labels were known and predictions were
selected from those labels. Do not interpret that score as open-ended reasoning
or general Arabic understanding.
- Test accuracy is an aggregate; results vary by question type, as shown above.
Validate on representative data before deployment.
- The model is not a substitute for expert review in high-impact decisions.