Leveraging Milvus and Friendli Serverless Endpoints for Advanced RAG and Multi-Modal Queries

Introduction

Link to Colab: Leveraging Milvus and Friendli Serverless Endpoints for Advanced RAG and Multi-Modal Queries.ipynb

• ​​FriendliAI specializes in generative AI infrastructure, offering solutions that enable organizations to efficiently deploy and manage large language models (LLMs) and other generative AI models with optimized performance and reduced cost. Users have the ability to choose from production-ready conventional LLMs accessible through APIs, or custom fine-tuned LLMs deployed on the hardware of the user’s choice, whether on the public cloud or on private on-premise clusters.

• Milvus is an open-source vector database that stores, indexes, and searches billion-scale unstructured data through high-dimensional vector embeddings. It is perfect for building modern AI applications such as retrieval augmented generation (RAG), semantic search, multimodal search, and recommendation systems.

In this article, we'll explore how to use Milvus with Friendli Serverless Endpoints to perform Retrieval-Augmented Generation (RAG) on particular documents and materials, as well as to execute multi-modal queries that incorporate images and other visual content. This powerful combination allows for more sophisticated and context-aware AI applications.

Understanding RAG and Multi-Modal Models

Retrieval-Augmented Generation (RAG)

RAG is a technique that enhances language models by providing them with relevant information, primarily retrieved from a vector database-powered knowledge base. This approach allows AI models to generate more accurate and contextually appropriate responses by referencing designated external data sources.

Multi-Modal Models

Multimodal models can process and understand multiple types of input data, such as text, images, and audio. They can analyze and generate responses based on diverse information sources, enabling more comprehensive and nuanced interactions.

Why Incorporate RAG and Multi-modal models together?

The combination of RAG and multi-modal capabilities significantly improves AI systems by providing the following features simultaneously:

1. Allowing for more diverse and rich input types of the user’s choice
2. Providing up-to-date information
3. Enhancing accuracy and relevance of responses
4. Enabling context-aware interactions

Hands-On Implementation

Let's dive into the practical implementation of RAG and multi-modal queries using the Milvus vector database and Friendli Serverless Endpoints.

Step 1: Install Prerequisites and Download Milvus Docs

First, we'll install the necessary libraries and download the Milvus documentation that we'll use for our RAG job:

Step 2: Process Documentation Files

Next, we'll read the Milvus documentation files and use a simple file-splitting strategy to treat each text line as an individual chunk:

Step 3: Prepare Embeddings

We'll use the Hugging Face embeddings library to use a simple `all-MiniLM-L6` model to create vector representations of our text:

Step 4: Set Up Milvus Client

Now, let's prepare the Milvus client for our RAG implementation. In this simple example, we use Milvus Lite, which runs locally and materializes a file in a local file. You can also consider other Milvus deployment options:

• If you only need a local vector database for small scale data or prototyping, setting the uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.
• For larger scale data and traffic in production, you can set up a Milvus server on Docker or Kubernetes. In this setup, please use the server address and port as your uri, e.g.http://localhost:19530. If you enable the authentication feature on Milvus, set the token as "<your_username>:<your_password>", otherwise there is no need to set the token.
• You can also use fully managed Milvus on Zilliz Cloud. Simply set the uri and token to the Public Endpoint and API key of your Zilliz Cloud instance.

Step 5: Create Milvus Collection

We'll create a collection in the Milvus client if it doesn't already exist:

Step 6: Embed and Insert Text into Milvus

Let's embed our text and insert it into the Milvus collection:

Step 7: Perform RAG Query

Now we can ask a question and search for relevant data within our Milvus database:

Step 8: Create Prompts for RAG

Let's create the system and user prompts for our RAG query:

Step 9: Set Up Friendli Token

Obtain your FRIENDLI_TOKEN from the Friendli Suite and set it as an environment variable:

Step 10: Execute RAG Query

Now we can execute our RAG query using the Friendli Serverless Endpoints:

This produces the answer based on the provided documents:

Step 11: Multi-Modal Queries

For multi-modal queries, we'll use the Llama-3.2-11b-vision model:

From its response, we can infer that the model correctly understands the image:

Step 12: Combine RAG and Multi-Modal Capabilities

Finally, let's combine the RAG and multi-modal capabilities:

The model correctly generates a correct response based on the image and the documents:

Conclusion

This tutorial has demonstrated how to leverage Milvus and Friendli Serverless Endpoints to implement advanced RAG and multi-modal queries. By combining these powerful technologies, you can create more sophisticated AI applications that can understand and process diverse types of information, leading to more accurate and context-aware responses.