GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nomic-ai
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
nomic-ai
Model Tree
Qwen/Qwen2.5-VL-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Model | Avg. | ESG Restaurant Human | Econ Macro Multi. | AXA Multi. | MIT Bio | ESG Restaurant Synth. | ESG Restaurant Synth. Multi. | MIT Bio Multi. | AXA | Econ. Macro |
|---|---|---|---|---|---|---|---|---|---|---|
| ColNomic Embed Multimodal 7B | 62.7 | 73.9 | 54.7 | 61.3 | 66.1 | 57.3 | 56.7 | 64.2 | 68.3 | 61.6 |
| ColNomic Embed Multimodal 3B | 61.2 | 65.8 | 55.4 | 61.0 | 63.5 | 56.6 | 57.2 | 62.5 | 68.8 | 60.2 |
| T-Systems ColQwen2.5-3B | 59.9 | 72.1 | 51.2 | 60.0 | 65.3 | 51.7 | 53.3 | 61.7 | 69.3 | 54.8 |
| Nomic Embed Multimodal 7B | 59.7 | 65.7 | 57.7 | 59.3 | 64.0 | 49.2 | 51.9 | 61.2 | 66.3 | 63.1 |
| GME Qwen2 7B | 59.0 | 65.8 | 56.2 | 55.4 | 64.0 | 54.3 | 56.7 | 55.1 | 60.7 | 62.9 |
| Nomic Embed Multimodal 3B | 58.8 | 59.8 | 57.5 | 58.8 | 62.5 | 49.4 | 49.4 | 58.6 | 69.6 | 63.5 |
| Llama Index vdr-2b-multi-v1 | 58.4 | 63.1 | 52.8 | 61.0 | 60.6 | 50.3 | 51.2 | 56.9 | 68.8 | 61.2 |
| Voyage Multimodal 3 | 55.0 | 56.1 | 55.0 | 59.5 | 56.4 | 47.2 | 46.2 | 51.5 | 64.1 | 58.8 |
To use colnomic-embed-multimodal-7b, please install colpali from source
bash
pip install git+https://github.com/illuin-tech/colpali.gitpython
import torchfrom PIL import Imagefrom transformers.utils.import_utils import is_flash_attn_2_available from colpali_engine.models import ColQwen2_5, ColQwen2_5_Processor model_name = "nomic-ai/colnomic-embed-multimodal-7b" model = ColQwen2_5.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="cuda:0", # or "mps" if on Apple Silicon attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None,).eval() processor = ColQwen2_5_Processor.from_pretrained(model_name) # Your inputsimages = [ Image.new("RGB", (128, 128), color="white"), Image.new("RGB", (64, 32), color="black"),]queries = [ "What is the organizational structure for our R&D department?", "Can you provide a breakdown of last year’s financial performance?",] # Process the inputsbatch_images = processor.process_images(images).to(model.device)batch_queries = processor.process_queries(queries).to(model.device) # Forward passwith torch.no_grad(): image_embeddings = model(**batch_images) query_embeddings = model(**batch_queries) scores = processor.score_multi_vector(query_embeddings, image_embeddings)Nomic Embed Multimodal 7B seamlessly integrates with Retrieval Augmented Generation (RAG) workflows:
The model excels at handling real-world document retrieval scenarios that challenge traditional text-only systems:
ColNomic Embed Multimodal 7B was developed through several key innovations:
Sampling From the Same Source: Forcing sampling from the same dataset source creates harder in-batch negatives, preventing the model from learning dataset artifacts.
Multi-Vector Configuration: Providing a multi-vector variant that achieves higher performance than the dense variant.
If you find this model useful in your research or applications, please consider citing:
bibtex
@misc{faysse2024colpaliefficientdocumentretrieval, title={ColPali: Efficient Document Retrieval with Vision Language Models}, author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo}, year={2024}, eprint={2407.01449}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2407.01449}, }@misc{ma2024unifyingmultimodalretrievaldocument, title={Unifying Multimodal Retrieval via Document Screenshot Embedding}, author={Xueguang Ma and Sheng-Chieh Lin and Minghan Li and Wenhu Chen and Jimmy Lin}, year={2024}, eprint={2406.11251}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2406.11251}, }@misc{nomicembedmultimodal2025, title={Nomic Embed Multimodal: Interleaved Text, Image, and Screenshots for Visual Document Retrieval}, author={Nomic Team}, year={2025}, publisher={Nomic AI}, url={https://nomic.ai/blog/posts/nomic-embed-multimodal},}pip install git+https://github.com/illuin-tech/colpali.gitimport torchfrom PIL import Imagefrom transformers.utils.import_utils import is_flash_attn_2_available from colpali_engine.models import ColQwen2_5, ColQwen2_5_Processor model_name = "nomic-ai/colnomic-embed-multimodal-7b" model = ColQwen2_5.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="cuda:0", # or "mps" if on Apple Silicon attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None,).eval() processor = ColQwen2_5_Processor.from_pretrained(model_name) # Your inputsimages = [ Image.new("RGB", (128, 128), color="white"), Image.new("RGB", (64, 32), color="black"),]queries = [ "What is the organizational structure for our R&D department?", "Can you provide a breakdown of last year’s financial performance?",] # Process the inputsbatch_images = processor.process_images(images).to(model.device)batch_queries = processor.process_queries(queries).to(model.device) # Forward passwith torch.no_grad(): image_embeddings = model(**batch_images) query_embeddings = model(**batch_queries) scores = processor.score_multi_vector(query_embeddings, image_embeddings)@misc{faysse2024colpaliefficientdocumentretrieval, title={ColPali: Efficient Document Retrieval with Vision Language Models}, author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo}, year={2024}, eprint={2407.01449}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2407.01449}, }@misc{ma2024unifyingmultimodalretrievaldocument, title={Unifying Multimodal Retrieval via Document Screenshot Embedding}, author={Xueguang Ma and Sheng-Chieh Lin and Minghan Li and Wenhu Chen and Jimmy Lin}, year={2024}, eprint={2406.11251}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2406.11251}, }@misc{nomicembedmultimodal2025, title={Nomic Embed Multimodal: Interleaved Text, Image, and Screenshots for Visual Document Retrieval}, author={Nomic Team}, year={2025}, publisher={Nomic AI}, url={https://nomic.ai/blog/posts/nomic-embed-multimodal},}