The model wraps hallucinated spans with typed <hallucination> tags:
<Tagged_Text>
The <hallucination type="Color_Attribute">bright red</hallucination>
<hallucination type="Object">sports</hallucination> car is
<hallucination type="Spatial_Attribute">parked near a lake</hallucination>.
</Tagged_Text>
Hallucination Taxonomy
12 fine-grained types across two categories:
Table with columns: Category, Types| Category | Types |
|---|
| Perception | Object, OCR, Numerical_Attribute, Color_Attribute, Shape_Attribute, Spatial_Attribute |
| Reasoning | Logical_Error, Calculation_Error, Knowledge_Error, Query_Misunderstanding, Numerical_Relation, Spatial_Relation |
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
model = AutoModelForImageTextToText.from_pretrained(
"wkinglin/HalluScope-4B", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("wkinglin/HalluScope-4B")
messages = [{
"role": "user",
"content": [
{"type": "image", "image": Image.open("example.jpg")},
{"type": "text", "text": "Analyze the response and tag hallucinated spans:\n<response to diagnose>"},
],
}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt",
).to(model.device)
out = model.generate(**inputs, max_new_tokens=2048)
print(processor.batch_decode(out, skip_special_tokens=True)[0])
For high-throughput inference, serve the model with vLLM and query it through
the OpenAI-compatible API.
Citation
@inproceedings{jin2026halluscope,
title = {HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models},
author = {Jin, Weilin and Wang, Mingyu and Li, Wenbo and Huang, Haoyang and Wu, Yifan and Li, Ying and Huang, Gang and Wu, Zhonghai},
booktitle = {Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
year = {2026}
}