October 27, 2023
2 min read

LangChain Integration with Friendli Dedicated Endpoints

Q: What is FriendliAI?

FriendliAI is the Frontier Inference Cloud for Agents, delivering high throughput, low latency, and reliability at scale for agentic workloads. Through vertically optimized inference infrastructure, it delivers 2–5× faster output token speed and a 99.99% uptime SLA for high-volume production traffic.

LangChain Integration with Friendli Dedicated Endpoints thumbnail

In this article, we will demonstrate how to use Friendli Dedicated Endpoints with LangChain. Friendli Dedicated Endpoints is our SaaS service for deploying generative AI models that run Friendli Inference, our flagship LLM serving engine, on various cloud platforms. LangChain is a popular framework for building language model applications. It offers developers a convenient way of combining multiple components into a language model application. Using Friendli Dedicated Endpoints with LangChain allows developers to not only write language model applications easily, but also leverages the capabilities of Friendli Inference, our flagship LLM serving engine, to enhance the performance and cost-efficiency of serving the LLM model.

Building a Friendli LLM interface for LangChain

LangChain provides various LLM model interfaces and also allows defining a custom interface with ease by inheriting LangChain’s base LLM model. First, to get started, you'll need a running Friendli Inference deployment and an API key. Please refer to our docs for running a deployment on Friendli Dedicated Endpoints. Then, Friendli Inference provides a Python SDK for running language completion tasks, so we’ll use its completion API to implement our custom interface.

Here is our Friendli Inference LLM interface for LangChain:

python

from langchain.llms.base import LLM
from langchain.schema import LLMResult
from friendli import Completion, V1CompletionOptions

class FriendliEndpoint(LLM):
    """Friendli LLM interface

    api_key:   Friendli Dedicated Endpoints API Key
    endpoint:  Friendli Dedicated Endpoints deployment endpoint
    option:    Text completion options.
               Please check out https://friendli.ai/docs/openapi/serverless/completions for full options
    """
    api_key: str | None = None
    endpoint: str = ""
    options: dict = dict(
      max_tokens=200,
      top_p=0.8,
      temperature=0.5,
      no_repeat_ngram=3,
    )

    @property
    def _llm_type(self) -> str:
        """Return type of llm."""
        return "friendli"

    def _call(
        self,
        prompt: str,
        stop: list[str] | None = None,
        run_manager: CallbackManagerForLLMRun | None = None,
        **kwargs: Any,
    ) -> str:
    """LLM inference method."""
    options = V1CompletionOptions(
        prompt=prompt,
        stop=stop,
        **self.options,
    )
    # Define an API endpoint instance
    api = Completion(endpoint=self.endpoint, deployment_security_level="public")
    # Requests text generation to Friendli Dedicated Endpoints deployment
    completion = api.create(options=options, stream=False)
    return completion.choices[0].text  # Returns generated text

Now we can simply create an instance and use it like any other LLMs in the LangChain framework:

python

friendli_llm = FriendliEndpoint(
  api_key="<YOUR_FRIENDLI_TOKEN>",
  endpoint="https://friendli-deployment-endpoint",
)
friendli_llm.predict("Python is a popular")
# >> "general-purpose programming language that supports..."

Streaming

Friendli Inference also supports streaming a response, so that instead of waiting for the full response, you can receive intermediate results during generation. The LangChain framework also supports the streaming interface as _stream and _astream method, so we’ll also implement them using Friendli Inference's stream option.

python

from langchain.schema.output import GenerationChunk

class FriendliDeployment(LLM):
    ...
    def _stream(
        self,
        prompt: str,
        stop: list[str] | None = None,
        run_manager: CallbackManagerForLLMRun | None = None,
        **kwargs: Any,
    ) -> Iterator[GenerationChunk]:
        options = V1CompletionOptions(
            prompt=prompt,
            stop=stop,
            **self.options,
        )
        """LLM inference method with streaming option."""
        api = Completion(endpoint=self.endpoint, deployment_security_level="public")
        stream = api.create(options=options, stream=True) # Requests generation with streaming option
        for line in stream:
            # Receives and returns generated tokens in streaming fashion
            chunk = GenerationChunk(text=json.dumps(line.model_dump()))
            yield chunk
            if run_manager:
                # If the callback manager is given, invokes its token handler
                run_manager.on_llm_new_token(line.text, chunk=chunk)

With the streaming interface, you can display the response to the user as it’s being generated in real-time:

python

from friendli.schema.api.v1.completion import V1CompletionLine

async for resp in friendli_llm.astream("Tell me a story"):
    line = V1CompletionLine.model_validate_json(resp)
    print(line, end="")   # Asynchronously prints generated tokens

In summary, we’ve implemented a custom Friendli Inference LLM interface for LangChain and looked at how it can be used with basic examples. In our next blog post, we will see how to build more complex LLM applications using the Friendli Inference and LangChain. Get started today with Friendli Inference!

Written by

FriendliAI Tech & Research

General FAQ

What is FriendliAI?

FriendliAI is the Frontier Inference Cloud for Agents, delivering high throughput, low latency, and reliability at scale for agentic workloads. Through vertically optimized inference infrastructure, it delivers 2–5× faster output token speed and a 99.99% uptime SLA for high-volume production traffic.

How does FriendliAI reduce inference costs?

FriendliAI reduces inference costs through higher GPU utilization and optimized inference performance. FriendliAI's patented continuous batching technique, along with quantization, speculative decoding, KV cache offloading, multi-LoRA serving, and autoscaling, helps you serve more tokens with fewer GPUs, lowering your infrastructure costs without sacrificing performance.

Why should I choose FriendliAI over other inference providers?

FriendliAI is built for production AI agents, combining speed, reliability, and efficiency at scale. It delivers low-latency streaming, reliable long-context inference, and robust tool calling without compromising stability. According to independent OpenRouter benchmarks, FriendliAI consistently ranks among the top providers for throughput, latency, and reliability across leading open-weight models. See why customers choose FriendliAI

Which open-weight models does FriendliAI support?

Run today’s frontier open-weight models—including GLM, MiniMax, Kimi, DeepSeek, Qwen, Gemma, and more—with a simple API call. FriendliAI Model API gives you instant access to the latest models with optimized inference performance for production workloads. Explore models and pricing

How do I get started?

Getting started takes just a few minutes. [1] Sign up for FriendliAI, [2] Generate your API key, and [3] Make your first inference request with frontier open-weight models.

Still have questions?

If you want a customized solution for that key issue that is slowing your growth, support@friendli.ai or click Talk to an engineer — our engineers (not a bot) will reply within one business day.