GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Barath
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Barath
Model Tree
openbmb/MiniCPM-V-4
Input Modalities
Output Modalities
Supported Functionality
| JSON valid | Precision | Recall | F1 | |
|---|---|---|---|---|
| MiniCPM-V-4 zero-shot | 32% | 0.027 | 0.010 | 0.015 |
| LoRA, LLM-only (300 steps, 800 samples) | 16% | 0.0 | 0.0 | 0.0 |
| LoRA + vision tuning (800 steps, 3.3k samples) | 40% | 0.439 | 0.060 | 0.105 |
The decisive factor was unfreezing the vision tower: LLM-only LoRA learned the output format but stayed image-blind (high-confidence repetition of dataset priors). With vision tuning, precision rose 16x over zero-shot — the model genuinely grounds boxes in the drawing. Recall remains the open weakness (long element lists; outputs sometimes truncate before the JSON closes).
openbmb/MiniCPM-V-4, official MiniCPM-V finetune harness
(llm_type ChatML; note: the harness's qwen2 target-span detection needs a
patch for V-4's tokenizer — spans located by token-id lookup of
'<|im_start|>'/'assistant' only exist in Qwen2's vocab)python
import torchfrom transformers import AutoModel, AutoTokenizerfrom peft import PeftModel base = "openbmb/MiniCPM-V-4"tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)model = AutoModel.from_pretrained(base, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda")model = PeftModel.from_pretrained(model, "Barath/minicpmv4-floorplan-lora")model = model.merge_and_unload().eval() from PIL import Imageimg = Image.open("floorplan.png").convert("RGB")prompt = ('Detect the architectural elements in this floor plan (walls, doors, ' 'windows, stairs, fixtures, furniture). Return only JSON: ' '{"elements": [{"type": str, "bbox": [x1, y1, x2, y2]}]} with integer ' 'coordinates normalized to [0, 1000].')out = model.chat(msgs=[{"role": "user", "content": [img, prompt]}], tokenizer=tokenizer, sampling=False, max_new_tokens=1500, repetition_penalty=1.1)print(out)Trained for the Hugging Face Build Small Hackathon 2026.
import torchfrom transformers import AutoModel, AutoTokenizerfrom peft import PeftModel base = "openbmb/MiniCPM-V-4"tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)model = AutoModel.from_pretrained(base, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda")model = PeftModel.from_pretrained(model, "Barath/minicpmv4-floorplan-lora")model = model.merge_and_unload().eval() from PIL import Imageimg = Image.open("floorplan.png").convert("RGB")prompt = ('Detect the architectural elements in this floor plan (walls, doors, ' 'windows, stairs, fixtures, furniture). Return only JSON: ' '{"elements": [{"type": str, "bbox": [x1, y1, x2, y2]}]} with integer ' 'coordinates normalized to [0, 1000].')out = model.chat(msgs=[{"role": "user", "content": [img, prompt]}], tokenizer=tokenizer, sampling=False, max_new_tokens=1500, repetition_penalty=1.1)print(out)