GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
kshitizjangra
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
kshitizjangra
Model Tree
Qwen/Qwen2-VL-2B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| base model | Qwen/Qwen2-VL-2B-Instruct |
| method | LoRA (r=16, α=32, dropout=0.05) on q_proj,k_proj,v_proj,o_proj |
| starting weights | continued from prior adapter (v1) — not from scratch |
| dataset | 1500 OMR sheets × 3 fields, human-corrected via Django labeling tool |
| split | 80/20 by sheet (no leakage) → 3414 train / 859 eval rows |
| epochs | 3 |
| batch size | 1 (per-device) × 8 grad-accum = effective 8 |
| learning rate | 5e-5 (lower than v1 since starting from a trained adapter) |
| warmup | 0.03 |
| precision | fp16 on Apple Silicon MPS |
| gradient checkpointing | on |
| total steps | 1281 |
| final train loss | 0.014 (running avg) |
| wall-clock | ~6 hours on M-series MPS |
python
from peft import PeftModelfrom transformers import AutoProcessor, Qwen2VLForConditionalGenerationfrom PIL import Imageimport torch BASE = "Qwen/Qwen2-VL-2B-Instruct"ADAPTER = "kshitizjangra/qwen2vl-omr-lora-v2" processor = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)base = Qwen2VLForConditionalGeneration.from_pretrained(BASE, dtype=torch.float16, trust_remote_code=True)model = PeftModel.from_pretrained(base, ADAPTER).to("mps").eval() img = Image.open("path/to/roll_no.jpg").convert("RGB")messages = [{"role": "user", "content": [ {"type": "image", "image": img}, {"type": "text", "text": "Read the handwritten value. Output only the value."},]}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[text], images=[[img]], return_tensors="pt").to("mps")with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=64, do_sample=False)print(processor.tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))Internal tool for digitizing university OMR sheets at a fixed template (Part-D). The model expects a single section crop (registration_no / roll_no / course_code) and returns the handwritten value as a string.
from peft import PeftModelfrom transformers import AutoProcessor, Qwen2VLForConditionalGenerationfrom PIL import Imageimport torch BASE = "Qwen/Qwen2-VL-2B-Instruct"ADAPTER = "kshitizjangra/qwen2vl-omr-lora-v2" processor = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)base = Qwen2VLForConditionalGeneration.from_pretrained(BASE, dtype=torch.float16, trust_remote_code=True)model = PeftModel.from_pretrained(base, ADAPTER).to("mps").eval() img = Image.open("path/to/roll_no.jpg").convert("RGB")messages = [{"role": "user", "content": [ {"type": "image", "image": img}, {"type": "text", "text": "Read the handwritten value. Output only the value."},]}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[text], images=[[img]], return_tensors="pt").to("mps")with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=64, do_sample=False)print(processor.tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))