LangChain Integration with Friendli Dedicated Endpoints

In this article, we will demonstrate how to use Friendli Dedicated Endpoints with LangChain. Friendli Dedicated Endpoints is our SaaS service for deploying generative AI models that run Friendli Inference, our flagship LLM serving engine, on various cloud platforms. LangChain is a popular framework for building language model applications. It offers developers a convenient way of combining multiple components into a language model application. Using Friendli Dedicated Endpoints with LangChain allows developers to not only write language model applications easily, but also leverages the capabilities of Friendli Inference, our flagship LLM serving engine, to enhance the performance and cost-efficiency of serving the LLM model.

Building a Friendli LLM interface for LangChain

LangChain provides various LLM model interfaces and also allows defining a custom interface with ease by inheriting LangChain’s base LLM model. First, to get started, you'll need a running Friendli Inference deployment and an API key. Please refer to our docs for running a deployment on Friendli Dedicated Endpoints. Then, Friendli Inference provides a Python SDK for running language completion tasks, so we’ll use its completion API to implement our custom interface.

Here is our Friendli Inference LLM interface for LangChain:

Now we can simply create an instance and use it like any other LLMs in the LangChain framework:

Streaming

Friendli Inference also supports streaming a response, so that instead of waiting for the full response, you can receive intermediate results during generation. The LangChain framework also supports the streaming interface as _stream and _astream method, so we’ll also implement them using Friendli Inference's stream option.

With the streaming interface, you can display the response to the user as it’s being generated in real-time:

In summary, we’ve implemented a custom Friendli Inference LLM interface for LangChain and looked at how it can be used with basic examples. In our next blog post, we will see how to build more complex LLM applications using the Friendli Inference and LangChain. Get started today with Friendli Inference!