GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
llava-hf
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
llava-hf
Model Tree
Input Modalities
Output Modalities
Supported Functionality
LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. LLaVA 1.6 improves on LLaVA 1.5 BY:
You can use the raw model for tasks like image captioning, visual question answering, multimodal chatbot use cases. See the model hub to look for other versions on a task that interests you.
Here's the prompt template for this model but we recomment to use the chat templates to format the prompt with processor.apply_chat_template().
That will apply the correct template for a given checkpoint for you.
text
"[INST] <image>\nWhat is shown in this image? [/INST]"To run the model with the pipeline, see the below example:
python
from transformers import pipeline pipe = pipeline("image-text-to-text", model="llava-hf/llava-v1.6-mistral-7b-hf")messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"}, {"type": "text", "text": "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"}, ], },] out = pipe(text=messages, max_new_tokens=20)print(out)>>> [{'input_text': [{'role': 'user', 'content': [{'type': 'image', 'url': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg'}, {'type': 'text', 'text': 'What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud'}]}], 'generated_text': 'Lava'}]You can also load and use the model like following:
python
from transformers import LlavaNextProcessor, LlavaNextForConditionalGenerationimport torchfrom PIL import Imageimport requests processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf") model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True) model.to("cuda:0") # prepare image and text prompt, using the appropriate prompt templateurl = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"image = Image.open(requests.get(url, stream=True).raw) # Define a chat history and use `apply_chat_template` to get correctly formatted prompt# Each value in "content" has to be a list of dicts with types ("text", "image") conversation = [ { "role": "user", "content": [ {"type": "text", "text": "What is shown in this image?"}, {"type": "image"}, ], },]prompt = processor.apply_chat_template(conversation, add_generation_prompt=True) inputs = processor(images=image, text=prompt, return_tensors="pt").to("cuda:0") # autoregressively complete promptoutput = model.generate(**inputs, max_new_tokens=100) print(processor.decode(output[0], skip_special_tokens=True))From transformers>=v4.48, you can also pass image url or local path to the conversation history, and let the chat template handle the rest.
Chat template will load the image for you and return inputs in torch.Tensor which you can pass directly to model.generate()
python
messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], },] inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=50)bitsandbytes libraryFirst make sure to install bitsandbytes, pip install bitsandbytes and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:
diff
model = LlavaNextForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True,+ load_in_4bit=True)First make sure to install flash-attn. Refer to the original repository of Flash Attention regarding that package installation. Simply change the snippet above with:
diff
model = LlavaNextForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True,+ use_flash_attention_2=True).to(0)bibtex
@misc{liu2023improved, title={Improved Baselines with Visual Instruction Tuning}, author={Haotian Liu and Chunyuan Li and Yuheng Li and Yong Jae Lee}, year={2023}, eprint={2310.03744}, archivePrefix={arXiv}, primaryClass={cs.CV}}from transformers import pipeline pipe = pipeline("image-text-to-text", model="llava-hf/llava-v1.6-mistral-7b-hf")messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"}, {"type": "text", "text": "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"}, ], },] out = pipe(text=messages, max_new_tokens=20)print(out)>>> [{'input_text': [{'role': 'user', 'content': [{'type': 'image', 'url': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg'}, {'type': 'text', 'text': 'What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud'}]}], 'generated_text': 'Lava'}]from transformers import LlavaNextProcessor, LlavaNextForConditionalGenerationimport torchfrom PIL import Imageimport requests processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf") model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True) model.to("cuda:0") # prepare image and text prompt, using the appropriate prompt templateurl = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"image = Image.open(requests.get(url, stream=True).raw) # Define a chat history and use `apply_chat_template` to get correctly formatted prompt# Each value in "content" has to be a list of dicts with types ("text", "image") conversation = [ { "role": "user", "content": [ {"type": "text", "text": "What is shown in this image?"}, {"type": "image"}, ], },]prompt = processor.apply_chat_template(conversation, add_generation_prompt=True) inputs = processor(images=image, text=prompt, return_tensors="pt").to("cuda:0") # autoregressively complete promptoutput = model.generate(**inputs, max_new_tokens=100) print(processor.decode(output[0], skip_special_tokens=True))messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], },] inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt")output = model.generate(**inputs, max_new_tokens=50)model = LlavaNextForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True,+ load_in_4bit=True)model = LlavaNextForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True,+ use_flash_attention_2=True).to(0)@misc{liu2023improved, title={Improved Baselines with Visual Instruction Tuning}, author={Haotian Liu and Chunyuan Li and Yuheng Li and Yong Jae Lee}, year={2023}, eprint={2310.03744}, archivePrefix={arXiv}, primaryClass={cs.CV}}