Results
KITAB-Bench ocr-eval, 13 datasets, 3,760 images, final-answer scoring with the
benchmark's unmodified metrics.
Table with columns: amad-vlm5, amad-vlm6 | amad-vlm5 | amad-vlm6 |
|---|
| CER ↓ (macro) | 0.254 | 0.235 |
| CHrF ↑ (macro) | 81.05 | 81.68 |
| CER ↓ (10 leak-free datasets) | 0.321 | 0.236 |
| CHrF ↑ (10 leak-free datasets) | 77.35 | 82.46 |
Largest per-dataset gains: khattparagraph CER 0.711 → 0.088, evarest
0.291 → 0.142, hindawi 0.241 → 0.137.
What the gain is. Error-type decomposition (see the report) shows the
improvement is behavioral: the merge cuts amad-vlm5's thinking rate on
page-level inputs from 45–98% to 5–18% and removes its truncated and empty
outputs, so insertions and deletions collapse. Reading accuracy itself
(substitution rate) is worse on 10 of 13 datasets and better only on adab
and evarest. Hallucination is not cancelled but moved: on isippt and muharaf the
merged model over-generates where amad-vlm5 did not. Four per-dataset wins and
three losses are statistically significant (bootstrap); five are not.
Files
Table with columns: Repository, Format, Use| Repository | Format | Use |
|---|
| this repo | bf16 safetensors | Transformers, vLLM, further fine-tuning |
| amad-vlm6-GGUF | Q8_0 + mmproj | llama.cpp, LM Studio |
Usage
import re, torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
repo = "amad-iq/amad-vlm6"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
repo, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [
{"type": "image", "image": "page.png"},
{"type": "text", "text": "Extract the text in the image. Give me the final text, nothing else."}]}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, do_sample=False, repetition_penalty=1.05)
raw = processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(re.sub(r"<think>.*?</think>", "", raw, flags=re.S).strip())
The model inherits a thinking behaviour from one parent: on dense pages it may
emit <think>…</think> before the transcription. Keep only the text after the
last </think>, and allow 4,096 new tokens — a 500-token budget truncates
dense pages.
Limitations
- Reading regresses on 10 of 13 datasets (substitution rate); the CER
wins come from output discipline, not perception. Macro WER is worse
(0.361 → 0.463), driven by three datasets with catastrophic rows (historyar,
muharaf, isippt), not by a general segmentation problem.
- Two large regressions from imported over-generation: isippt CER
0.054 → 0.541 (insertions 0.006 → 0.471, 500 images) and muharaf 0.046 →
0.600 (insertions 0.012 → 0.423, plus memorized reading lost). Avoid on
presentation-slide and muharaf-style handwritten inputs. onlinekhatt and
khatt (amad-vlm5's other training-overlap sets) also regress.
- Not yet shown to beat the cheaper fix. The natural control — amad-vlm5
with a repetition penalty and a think budget at inference — is being run;
until it lands, merging is not proven necessary for this gain.
- One benchmark is a narrow window. These numbers cover transcription
accuracy on 13 Arabic document styles. They say nothing about photographed
pages in poor light, skew, mixed Arabic–Latin text, tables and layout,
latency, or instruction following — where this model may hold unmeasured
strengths and unmeasured regressions.
Reproduction
merge_method: ties
base_model: Qwen/Qwen2.5-VL-7B-Instruct
models:
- model: amad-iq/amad-vlm5
parameters: {weight: 1.0, density: 0.5}
- model: AhmedZaky1/DIMI-Arabic-OCR-V2
parameters: {weight: 1.0, density: 0.5}
parameters: {lambda: 1.0, normalize: false}
dtype: bfloat16
License
Apache 2.0, inherited from the base model and both parents.
Author: Murtadha · amad-iq