GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
iisys-hof
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
iisys-hof
Model Tree
Qwen/Qwen2.5-VL-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This release contains the checkpoint selected by the author as the best-performing adapter from the project experiments.
| Property | Value |
|---|---|
| Base model | Qwen/Qwen2.5-VL-7B-Instruct |
| Format | PEFT LoRA, Safetensors |
| LoRA rank / alpha | 96 / 96 |
| LoRA dropout | 0.05 |
| Bias | None |
| Stored weight precision | BF16 |
| Stored LoRA parameters | 310,947,840 |
| Target modules | 358 |
The saved LoRA weights modify all 28 language-model layers and the two visual-merger projections, visual.merger.mlp.0 and visual.merger.mlp.2. The checkpoint also contains LoRA tensors for 160 vision-encoder projections in visual.blocks.*; their LoRA-B matrices are zero and contribute no weight update through those adapter modules. The configuration preserves the full set of 358 targets represented in the checkpoint.
The repository provides adapter_config.json, adapter_model.safetensors, and this model card. The example uses the tokenizer, processor, and chat template from the base model.
Install PyTorch and torchvision for your hardware using the official PyTorch installation instructions, then install the inference dependencies:
bash
pip install -U transformers==4.51.3 peft==0.15.2 accelerate safetensors pillow qwen-vl-utilsThese examples pin Transformers and PEFT to match the module names stored in this adapter. They are example dependencies, not a record of the training environment.
The example follows the original Qwen2.5-VL quickstart and adds loading of this adapter. It uses the base model's default image preprocessing and greedy decoding. These settings are an inference example; the study's evaluation procedure is described in the paper.
Replace uml_diagram.png with your diagram image and change the question as needed:
python
from pathlib import Path import torchfrom PIL import Imagefrom peft import PeftModelfrom qwen_vl_utils import process_vision_infofrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration BASE_ID = "Qwen/Qwen2.5-VL-7B-Instruct"ADAPTER_ID = "iisys-hof/Qwen2.5-VL-7B-Instruct-uml-fine-tuned"IMAGE_PATH = Path("uml_diagram.png")QUESTION = ( "List the names of all classes shown in this UML class diagram. " "Return a comma-separated list.") image = Image.open(IMAGE_PATH).convert("RGB")processor = AutoProcessor.from_pretrained(BASE_ID)base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained( BASE_ID, torch_dtype="auto", device_map="auto",)model = PeftModel.from_pretrained( base_model, ADAPTER_ID, is_trainable=False,).eval() messages = [{ "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": QUESTION}, ],}]prompt = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor( text=[prompt], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt",).to(model.device) with torch.inference_mode(): output_ids = model.generate( **inputs, max_new_tokens=256, do_sample=False, ) answer_ids = output_ids[:, inputs["input_ids"].shape[1]:]answer = processor.batch_decode( answer_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False,)[0]print(answer)To create a standalone model locally, merge this adapter into the original Qwen2.5-VL base model using PEFT's merge and export workflow. Install the dependencies from Quickstart and run the following in a fresh Python session or notebook kernel. While this repository is private, authenticate with a Hugging Face account that has access before downloading the adapter.
The example loads the unquantized base model on CPU in BF16. Allow enough system RAM for the full model and merging, and enough disk space for the base-model download and the exported checkpoint.
python
from pathlib import Path import torchfrom peft import PeftModelfrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration BASE_ID = "Qwen/Qwen2.5-VL-7B-Instruct"ADAPTER_ID = "iisys-hof/Qwen2.5-VL-7B-Instruct-uml-fine-tuned"OUTPUT_DIR = Path("Qwen2.5-VL-7B-Instruct-uml-merged") base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained( BASE_ID, torch_dtype=torch.bfloat16, device_map="cpu",)model = PeftModel.from_pretrained( base_model, ADAPTER_ID, is_trainable=False,).eval() merged_model = model.merge_and_unload(safe_merge=True)merged_model.save_pretrained( OUTPUT_DIR, safe_serialization=True, max_shard_size="5GB",)processor = AutoProcessor.from_pretrained(BASE_ID)processor.save_pretrained(OUTPUT_DIR)print(f"Merged model saved to {OUTPUT_DIR.resolve()}")The output directory contains the full model weights and configuration, plus the tokenizer, image processor, and chat template. In a new Python session, load both Qwen2_5_VLForConditionalGeneration and AutoProcessor from that directory, using the same dependency versions as above.
The supplied UML training annotation file contains 604 diagram records referencing 604 image paths and 13,121 question-answer pairs. Each record contains id, image, and conversations; the conversation alternates between human questions and gpt reference answers in a LLaVA-style format.
The complete training-data release is available as iisys-hof/uml-class-diagrams-train. Download its train.zip for all 604 diagram images and both annotation formats; train_data.jsonl is also available separately for inspecting the 13,121 QA pairs. Training data is optional for inference.
The public evaluation dataset is iisys-hof/uml-class-diagrams. Its test split contains 1,596 image-question-answer examples across 17 question types, including examples with multiple valid reference answers and examples whose expected answer is None.
These weights correspond to the best-performing fine-tuned checkpoint in the accompanying study, achieving 88.3% accuracy. The study reports the following comparison (Table 2):
| Model | Reported accuracy |
|---|---|
| Qwen2.5-VL-7B-Instruct base | 44.4% |
| Qwen3-VL-8B | 53.9% |
| Qwen3.5-27B | 65.1% |
| This adapter: best fine-tune | 88.3% |
The study evaluates answers with exact matching that accommodates the order of list items while preserving identifier capitalization. See the paper for the experiment settings and analysis by question type. The table summarizes the published study comparison.
The 88.3% accuracy was obtained after correcting the evaluation handling of multiple valid reference answers (multi-gold). Earlier scoring was approximately one percentage point lower before this correction.
None.The fine-tuning implementation was based on 2U1/Qwen2-VL-Finetune by Yuwon Lee, now maintained as Qwen-VL-Series-Finetune, and adapted for UML class-diagram question answering with task-specific training configurations.
We acknowledge the Qwen team for the base model and Hugging Face Transformers and PEFT for the model-loading and adapter frameworks.
If you use this adapter in research, cite the accompanying study and the base model using the citation provided in the Qwen model card.
bibtex
@misc{naboichenko2026uml, title = {Unlocking UML Class Diagram Understanding in Vision Language Models}, author = {Artem Naboichenko and Ren{\'e} Peinl}, year = {2026}, eprint = {2605.11634}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2605.11634}}Adapter: iisys-hof/Qwen2.5-VL-7B-Instruct-uml-fine-tuned. License: Apache-2.0.
pip install -U transformers==4.51.3 peft==0.15.2 accelerate safetensors pillow qwen-vl-utilsfrom pathlib import Path import torchfrom PIL import Imagefrom peft import PeftModelfrom qwen_vl_utils import process_vision_infofrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration BASE_ID = "Qwen/Qwen2.5-VL-7B-Instruct"ADAPTER_ID = "iisys-hof/Qwen2.5-VL-7B-Instruct-uml-fine-tuned"IMAGE_PATH = Path("uml_diagram.png")QUESTION = ( "List the names of all classes shown in this UML class diagram. " "Return a comma-separated list.") image = Image.open(IMAGE_PATH).convert("RGB")processor = AutoProcessor.from_pretrained(BASE_ID)base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained( BASE_ID, torch_dtype="auto", device_map="auto",)model = PeftModel.from_pretrained( base_model, ADAPTER_ID, is_trainable=False,).eval() messages = [{ "role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": QUESTION}, ],}]prompt = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor( text=[prompt], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt",).to(model.device) with torch.inference_mode(): output_ids = model.generate( **inputs, max_new_tokens=256, do_sample=False, ) answer_ids = output_ids[:, inputs["input_ids"].shape[1]:]answer = processor.batch_decode( answer_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False,)[0]print(answer)from pathlib import Path import torchfrom peft import PeftModelfrom transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration BASE_ID = "Qwen/Qwen2.5-VL-7B-Instruct"ADAPTER_ID = "iisys-hof/Qwen2.5-VL-7B-Instruct-uml-fine-tuned"OUTPUT_DIR = Path("Qwen2.5-VL-7B-Instruct-uml-merged") base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained( BASE_ID, torch_dtype=torch.bfloat16, device_map="cpu",)model = PeftModel.from_pretrained( base_model, ADAPTER_ID, is_trainable=False,).eval() merged_model = model.merge_and_unload(safe_merge=True)merged_model.save_pretrained( OUTPUT_DIR, safe_serialization=True, max_shard_size="5GB",)processor = AutoProcessor.from_pretrained(BASE_ID)processor.save_pretrained(OUTPUT_DIR)print(f"Merged model saved to {OUTPUT_DIR.resolve()}")@misc{naboichenko2026uml, title = {Unlocking UML Class Diagram Understanding in Vision Language Models}, author = {Artem Naboichenko and Ren{\'e} Peinl}, year = {2026}, eprint = {2605.11634}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2605.11634}}