GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
GRAI-UNSTPB
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
GRAI-UNSTPB
Model Tree
unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit
Input Modalities
Output Modalities
Supported Functionality
You can load and run inference on this model either using Unsloth (for 2x faster inference) or standard Hugging Face transformers and peft.
bash
pip install torch transformers peft pillow datasets# If you want to use Unsloth for faster inference, install it via:# pip install unsloth
python
from unsloth import FastVisionModelimport torchfrom datasets import load_dataset# Load model and processormodel, processor = FastVisionModel.from_pretrained(model_name="GRAI-UNSTPB/llava-v1.6-mistral-7b-4bit-RoVQA-lora-v1",load_in_4bit=True)FastVisionModel.for_inference(model)# Load a test sample directly from the Flickr30K-RoQA-v1 datasetdataset = load_dataset("GRAI-UNSTPB/Flickr30K-RoQA-v1", split="test")sample = dataset[0]image = sample["image"]question = sample["question"]messages = [{"role": "user","content": [{"type": "image", "image": image},{"type": "text", "text": f"Răspunde la întrebare: {question}"}]}]input_text = processor.apply_chat_template(messages, add_generation_prompt=True)inputs = processor(images=image, text=input_text, return_tensors="pt").to("cuda")# Generate responseoutputs = model.generate(**inputs, max_new_tokens=64)# Decode only the generated response (excluding the prompt)prompt_len = inputs.input_ids.shape[1]response = processor.decode(outputs[0][prompt_len:], skip_special_tokens=True)print(response)
python
from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessorfrom peft import PeftModelimport torchfrom PIL import Image# Load base model and processorbase_model_id = "unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit"processor = LlavaNextProcessor.from_pretrained(base_model_id)model = LlavaNextForConditionalGeneration.from_pretrained(base_model_id,torch_dtype=torch.float16,device_map="auto")# Load LoRA adaptermodel = PeftModel.from_pretrained(model, "GRAI-UNSTPB/llava-v1.6-mistral-7b-4bit-RoVQA-lora-v1")# Inference setup follows the same structure as above...
The model was evaluated on the Flickr30K-RoQA test set and on Romanian Image Captioning (zero-shot transfer task without captioning fine-tuning).
| Model | BERTScore F1 (%) | ROUGE-L F1 (%) | BLEU |
|---|---|---|---|
| LLaVA-v1.6-Mistral-7B (Base) | 69.96 | 38.30 | 8.25 |
| LLaVA-v1.6-Mistral-7B + RoVQA LoRA (Ours) | 71.43 | 52.82 | 26.05 |
| Model | BERTScore F1 (%) | ROUGE-L F1 (%) | BLEU |
|---|---|---|---|
| LLaVA-v1.6-Mistral-7B (Base) | 59.90 | 16.70 | 2.62 |
| LLaVA-v1.6-Mistral-7B + RoVQA LoRA (Ours) | 68.54 | 41.45 | 15.38 |
If you use this model or adapter in your research, please cite the following paper:
bibtex
@article{dima2025parameter,title={Parameter-Efficient Multimodal Instruction Tuning for Romanian Vision--Language Models},author={Dima, George-Andrei and Smădu, Răzvan-Alexandru and Cercel, Dumitru-Clementin},journal={arXiv preprint arXiv:2512.14926},year={2025}}