Skip to main content
You can use LangChain Python SDK to interact with FriendliAI. This makes migration of existing applications already using LangChain particularly easy.

​
How to use

Before you start, ensure you’ve already obtained the FRIENDLI_TOKEN from the Friendli Suite. Our products are entirely compatible with OpenAI, so we use the langchain-openai package by referring to the FriendliAI baseURL. 
pip install -qU langchain-openai langchain

​
Instantiation

Now we can instantiate our model object and generate chat completions. We provide usage examples for each type of endpoint. Choose the one that best suits your needs: 
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="meta-llama-3.1-8b-instruct",
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLI_TOKEN"],
)

​
Runnable interface

We support both synchronous and asynchronous runnable methods to generate a response.

​
Synchronous methods:

result = llm.invoke("Tell me a joke.")
print(result.content)

​
Asynchronous methods:

result = await llm.ainvoke("Tell me a joke.")
print(result.content)

​
Chaining

We can chain our model with a prompt template. Prompt templates convert raw user input to better input to the LLM. 
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a world class technical documentation writer."),
    ("user", "{input}")
])

chain = prompt | llm

print(chain.invoke({"input": "how can langsmith help with testing?"}))
To get the string value instead of the message, we can add an output parser to the chain. 
from langchain_core.output_parsers import StrOutputParser

output_parser = StrOutputParser()

chain = prompt | llm | output_parser

print(chain.invoke({"input": "how can langsmith help with testing?"}))

​
Tool calling

Describe tools and their parameters, and let the model return a tool to invoke with the input arguments. Tool calling is extremely useful for enhancing the model’s capability to provide more comprehensive and actionable responses.

​
Define tools to use

The @tool decorator is used to define a tool. If you set parse_docstring=True, the tool will parse the docstring to extract the information of arguments. 
from langchain_core.tools import tool

@tool
def add(a: int, b: int) -> int:
    """Adds a and b."""
    return a + b

@tool
def multiply(a: int, b: int) -> int:
    """Multiplies a and b."""
    return a * b

tools = [add, multiply]

​
Bind tools to the model

Now models can generate a tool calling response. 
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="meta-llama-3.1-8b-instruct",
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLI_TOKEN"],
)

llm_with_tools = llm.bind_tools(tools)

query = "What is 3 * 12? Also, what is 11 + 49?"

print(llm_with_tools.invoke(query).tool_calls)

​
Generate a tool assisted message

Use the tool call results to generate a message. 
from langchain_core.messages import HumanMessage, ToolMessage

messages = [HumanMessage(query)]
ai_msg = llm_with_tools.invoke(messages)
messages.append(ai_msg)

for tool_call in ai_msg.tool_calls:
    selected_tool = {"add": add, "multiply": multiply}[tool_call["name"].lower()]
    tool_output = selected_tool.invoke(tool_call["args"])
    messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))

print(llm_with_tools.invoke(messages))
For more information on how to use tools, check out the LangChain documentation.