Hatim2221

Mubsir-Qwen-2B-VL

README

License: apache-2.0

Model Details

Base Architecture: Qwen/Qwen2-VL-2B-Instruct
Task: Optical Character Recognition (OCR) / Handwritten Text Recognition (HTR) for Arabic text.
Precision: torch.float16. The trained LoRA weights were merged into the base model entirely in 16-bit precision. This avoids the mathematical rounding errors and accuracy degradation commonly associated with merging weights directly into 4-bit quantized base models.
Parameters: 2 Billion.

Training Data

The model was fine-tuned using the KHATT (KFUPM Handwritten Arabic TexT) dataset, a comprehensive collection of diverse Arabic handwritten text lines designed for training high-accuracy recognition systems.

Training Infrastructure & Methodology

Hardware: 2x NVIDIA T4 GPUs (Kaggle cluster).
Methodology: Parameter-Efficient Fine-Tuning (PEFT) via QLoRA.
VRAM Optimization: The training pipeline utilized PyTorch expandable_segments for strict memory management, coupled with dynamic image resolution capping and tailored batch sizing to bypass standard Out-Of-Memory (OOM) failures during multimodal processing.

Use Cases

Example 1: Arabic Handwriting Transcription

Input:

Usage and Inference

To run this model, ensure you have the required Hugging Face libraries installed, including the Qwen vision utilities.

Installation

bash
pip install -q -U transformers accelerate qwen-vl-utils torchvision torchao

Inference code

python
import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model_id = "Hatim2221/Mubsir-Qwen-2B-VL"

# Load the model and processor in native FP16 precision
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

# Define the multimodal input payload
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "path_to_your_arabic_handwriting_image.jpg"}, 
            {"type": "text", "text": "Transcribe this Arabic handwriting:"}
        ]
    }
]

# Process text and vision inputs using the Qwen utility
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt"
).to("cuda")

# Generate the transcription
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=128)

# Isolate the generated output from the input tokens
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)

print("Transcription Result:")
print(output_text[0])

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Hatim2221

Model Tree

Base

Qwen/Qwen-VL

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Base Architecture: Qwen/Qwen2-VL-2B-Instruct
Task: Optical Character Recognition (OCR) / Handwritten Text Recognition (HTR) for Arabic text.
Precision: torch.float16. The trained LoRA weights were merged into the base model entirely in 16-bit precision. This avoids the mathematical rounding errors and accuracy degradation commonly associated with merging weights directly into 4-bit quantized base models.
Parameters: 2 Billion.

Training Data

Training Infrastructure & Methodology

Hardware: 2x NVIDIA T4 GPUs (Kaggle cluster).
Methodology: Parameter-Efficient Fine-Tuning (PEFT) via QLoRA.
VRAM Optimization: The training pipeline utilized PyTorch expandable_segments for strict memory management, coupled with dynamic image resolution capping and tailored batch sizing to bypass standard Out-Of-Memory (OOM) failures during multimodal processing.

Use Cases

Example 1: Arabic Handwriting Transcription

Input:

Usage and Inference

To run this model, ensure you have the required Hugging Face libraries installed, including the Qwen vision utilities.

Installation

bash
pip install -q -U transformers accelerate qwen-vl-utils torchvision torchao

Inference code

python
import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model_id = "Hatim2221/Mubsir-Qwen-2B-VL"

# Load the model and processor in native FP16 precision
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

# Define the multimodal input payload
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "path_to_your_arabic_handwriting_image.jpg"}, 
            {"type": "text", "text": "Transcribe this Arabic handwriting:"}
        ]
    }
]

# Process text and vision inputs using the Qwen utility
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt"
).to("cuda")

# Generate the transcription
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=128)

# Isolate the generated output from the input tokens
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)

print("Transcription Result:")
print(output_text[0])