You can import [the dashboard templates](https://github.com/friendliai/container-resource/tree/main/grafana) to your Grafana instance.
The Grafana instance must be connected to a Prometheus instance (or a Prometheus-compatible data source) which is configured to scrape metrics from Friendli Container processes.
1. **Create a Model**: Within SageMaker Inference, define a new model by specifying the model artifacts in your S3 bucket and the Friendli container image from ECR.
2. **Configure the Endpoint**: Create a SageMaker Inference endpoint configuration by selecting the instance type and the number of instances required.
3. **Create the Endpoint**: Utilize the configured settings to launch a SageMaker Inference endpoint.
4. **Invoke the Endpoint**: Once deployed, send requests to your endpoint to receive inference responses.
## Prerequisite
Before beginning, you need to push the Friendli Container image to an ECR repository on AWS.
First, prepare the Friendli Container image by following the instructions in [**Pulling Friendli Container Image**](/guides/container/running_friendli_container/#pulling-friendli-container-image).
Then, tag and push the image to the Amazon ECR repository as guided in [**Pushing a Docker image to an Amazon ECR private repository**](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html).
## Using the AWS Console
Let's delve into the step-by-step instructions for creating an inference endpoint using the AWS Console.
### Step 1: Creating a Model
You can start creating a model by clicking on the **'Create model'** button under **SageMaker > Inference > Models**.
Then, configure the model with the following fields:
* **Model settings**:
* **Model name**: A model name.
* **IAM role**: An IAM role that includes the `AmazonSageMakerFullAccess` policy.
* **Container definition 1**:
* **Container input option**: Select the "Provide model artifacts and inference image location".
* **Model Compression Type**:
* To use a model in the S3 bucket:
* When the model is compressed, select "CompressedModel".
* Otherwise, select "UncompressedModel".
* When using a model from the Hugging Face hub, any option would work fine.
* **Location of inference code image**: Specify the ARN of the ECR repo for the Friendli Container.
* **Location of model artifacts** (optional):
* To use a model in the S3 bucket: Specify the S3 URI where your model is stored. Ensure the file structure matches the directory format compatible with the `--hf-model-name` option of the Friendli Container.
* When using a model from the Hugging Face hub, you can leave this field empty.
* **Environment variables**:
* Always required:
* `FRIENDLI_CONTAINER_SECRET`: Your Friendli Container Secret. Refer to [**Preparing Container Secret**](/guides/container/running_friendli_container/#preparing-container-secret) to learn how to get the container secret.
* `SAGEMAKER_MODE`: This should be set to `True`.
* `SAGEMAKER_NUM_DEVICES`: Number of devices to use for tensor parallelism degree.
* Required when using a model in the S3 bucket:
* `SAGEMAKER_USE_S3`: This should be set to `True`.
* Required when using a model from the Hugging Face hub:
* `SAGEMAKER_HF_MODEL_NAME`: The Hugging Face model name (e.g., `mistralai/Mistral-7B-Instruct-v0.2`)
* For private or gated model repos:
* `HF_TOKEN`: The Hugging Face secret access token.
### Step 2: Creating an Endpoint Configuration
You can start by clicking on the **'Create endpoint configuration'** button under **SageMaker > Inference > Endpoint configurations**.
* **Endpoint configuration**:
* **Endpoint configuration name**: The name of this endpoint configuration.
* **Type of endpoint**: For real-time inference, select "Provisioned".
* **Variants**:
* To create a "Production" variant, click 'Create production variant'.
* Select the model that you have created in [**Step 1**](#step-1-creating-a-model).
* Configure the instance type and count by clicking on 'Edit' in the Actions column.
* Create the endpoint configuration by clicking 'Create endpoint configuration'.
### Step 3: Creating SageMaker Inference Endpoint
You can start by clicking the **'Create endpoint'** button under **SageMaker > Inference > Endpoints**.
* Select "Use an existing endpoint configuration".
* Select the endpoint configuration created in [**Step 2**](#step-2-creating-an-endpoint-configuration).
* Finish by clicking on the 'Create endpoint' button.
### Step 4: Invoking Endpoint
When the endpoint status becomes "In Service", you can invoke the endpoint with the following script, after filling in the endpoint name and the region name:
```python
import boto3
import json
endpoint_name = "FILL OUT ENDPOINT NAME"
region_name = "FILL OUT AWS REGION"
sagemaker_runtime = boto3.client("sagemaker-runtime", region_name=region_name)
prompt = "Story title: 3 llamas go for a walk\nSummary: The 3 llamas crossed a bridge and something unexpected happened\n\nOnce upon a time"
payload = {
"prompt": prompt,
"max_tokens": 512,
"temperature": 0.8,
}
response = sagemaker_runtime.invoke_endpoint(
EndpointName=endpoint_name,
Body=json.dumps(payload),
ContentType="application/json",
)
print(response['Body'].read().decode('utf-8'))
```
## Using the boto3 SDK
Next, let's discover the process for creating a SageMaker endpoint using the boto3 Python SDK.
You can achieve this by using the code snippet below. Be sure to fill in the custom fields, customized for your specific use case:
```python
import boto3
from sagemaker import get_execution_role
sm_client = boto3.client(service_name='sagemaker')
runtime_sm_client = boto3.client(service_name='sagemaker-runtime')
account_id = boto3.client('sts').get_caller_identity()['Account']
region = boto3.Session().region_name
role = get_execution_role()
endpoint_name="FILL OUT ENDPOINT NAME"
model_name="FILL OUT MODEL NAME"
container = "FILL OUT ECR IMAGE NAME" # 
## Prerequisite
huggingface-cli should be installed in your local environment.
```sh
pip install "huggingface_hub[cli]"
```
## Downloading Adapter Checkpoints
For each adapter model that you want to server, you have to download in your local storage.
```sh
# Hugging Face model name of the adapters
export ADAPTER_MODEL1=""
export ADAPTER_MODEL2=""
export ADAPTER_MODEL3=""
export ADAPTER_DIR=/tmp/adapter
huggingface-cli download $ADAPTER_MODEL1 \
--include "adapter_model.safetensors" "adapter_config.json" \
--local-dir $ADAPTER_DIR/model1
huggingface-cli download $ADAPTER_MODEL2 \
--include "adapter_model.safetensors" "adapter_config.json" \
--local-dir $ADAPTER_DIR/model2
huggingface-cli download $ADAPTER_MODEL3 \
--include "adapter_model.safetensors" "adapter_config.json" \
--local-dir $ADAPTER_DIR/model3
...
```
This will result in directory structure like:
```
/tmp/adapter/model1
- adapter_model.safetensors
- adapter_config.json
/tmp/adapter/model2
- adapter_model.safetensors
- adapter_config.json
/tmp/adapter/model3
- adapter_model.safetensors
- adapter_config.json
```
{children}
{caption &&
;
### Uploading Datasets
This document explains how to upload datasets. On Friendli, you can upload datasets via the web interface or the SDK.
{caption}
}
{children}
{caption &&
;
#### Hands-on Tutorial
Deploying `meta-llama-3-8b-instruct` LLM from Hugging Face using Friendli Dedicated Endpoints
## Introduction
With Friendli Dedicated Endpoints, you can easily spin up scalable, secure, and highly available inference deployments, without the need for extensive infrastructure expertise or significant capital expenditures.
This tutorial is designed to guide you through the process of launching and deploying LLMs using Friendli Dedicated Endpoints. Through a series of step-by-step instructions and hands-on examples, you'll learn how to:
* Select and deploy pre-trained LLMs from Hugging Face repositories
* Deploy and manage your models using the Friendli Engine
* Monitor and optimize your inference deployments
By the end of this tutorial, you'll be equipped with the knowledge and skills necessary to unlock the full potential of LLMs in your applications, products, and services. So, let's get started and explore the possibilities of Friendli Dedicated Endpoints!
## Prerequisites:
* A Friendli Suite account with access to [Friendli Dedicated Endpoints](https://friendli.ai/suite)
* A Hugging Face account with access to the [meta-llama-3-8b-instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) model
## Step 1: Create a new endpoint
1. Log in to your Friendli Suite account and navigate to the Friendli Dedicated Endpoints dashboard.
2. If not done already, start the free trial for Dedicated Endpoints.
3. Create a new project, then click on the 'New Endpoint' button.
4. Fill in the basic information:
* Endpoint name: Choose a unique name for your endpoint (e.g., "My New Endpoint").
5. Select the model:
{caption}
}
{children}
{caption &&
;
#### Hands-on Tutorial
Deploying `meta-llama-3-8b-instruct` LLM from W\&B using Friendli Dedicated Endpoints
## Introduction
With Friendli Dedicated Endpoints, you can easily spin up scalable, secure, and highly available inference deployments, without the need for infrastructure expertise or significant capital expenditures.
This tutorial is designed to guide you through the process of launching and deploying LLMs using Friendli Dedicated Endpoints. Through a series of step-by-step instructions and hands-on examples, you'll learn how to:
* Select and deploy pre-trained LLMs from W\&B artifacts
* Deploy and manage your models using the Friendli Engine
* Monitor and optimize your inference deployments
By the end of this tutorial, you'll be equipped with the knowledge and skills necessary to unlock the full potential of LLMs in your applications, products, and services. So, let's get started and explore the possibilities of Friendli Dedicated Endpoints!
## Prerequisites:
* A Friendli Suite account with access to [Friendli Dedicated Endpoints](https://friendli.ai/suite)
* A W\&B account with an api key (as an access token)
## Step 1: Create a new endpoint
1. Log in to your Friendli Suite account and navigate to the Friendli Dedicated Endpoints dashboard.
2. If not done already, start the free trial for Dedicated Endpoints.
3. Create a new project, then click on the 'New Endpoint' button.
4. [Integrate your W\&B account with an api key.](https://wandb.ai/authorize)
5. Fill in the basic information:
* Endpoint name: Choose a unique name for your endpoint (e.g., "My New Endpoint").
6. Select the model:
{caption}
}
{children}
{caption &&
;
## What are Endpoints?
Endpoints are the actual deployments of your models on a dedicated GPU resource.
They provide a stable and efficient interface to serve your models in real-world applications, ensuring high availability and optimized performance.
With endpoints, you can manage model versions, scale resources, and seamlessly integrate your model into production environments.
### Key Capabilities of Endpoints:
* **Efficient Model Serving**: Deploy models on powerful GPU instances optimized for your use case.
* **Flexibility with Multi-LoRA Models**: Serve multiple fine-tuned adapters alongside base models.
* **Autoscaling**: Automatically adjust resources to handle varying workloads, ensuring optimal performance and cost efficiency.
* **Monitoring and Management**: Check endpoint health, adjust configurations, and view logs directly from the platform.
* **Interactive Testing**: Use the integrated playground to test your models before integrating them into applications.
* **API Integration**: Access your models via robust OpenAI-compatible APIs, enabling easy integration into any system.
## Creating Endpoints
You can create your endpoint by specifying the name, the model, and the instance configuration, consisting of your desired GPU specification.
{caption}
}
{children}
{caption &&
;
## Integrations
{caption}
}
* Make sure to use the full name of the artifact.
* The *artifact name* must be in the format of `org/project/artifact_id:version`
* Use the repository id of the model. You may select the entry from the list of autocompleted model repositories.
* You may choose specific branch, or manually enter a commit hash.
The artifact might be nonexistent, or hidden so that you cannot access it.
The repository is gated. Please follow the steps and gain approval from the owner using Hugging Face Hub.
The model does not meet the requirements. Please check if the model follows a correct safetensors format. See the [format requirements](#format-requirements) for details.
The model architecture is not supported. Please refer to the [Supported Models](https://friendli.ai/models/search?products=DEDICATED) page.
Clicking the link above will display a screen like the one shown. Click the 'Deploy now' button here to deploy the LoRA model to Friendli Dedicated Endpoints.
Once the deployment is complete, a screen like the one below will appear. Click the 'Go to Suite' button to navigate to the playground where you can use the LoRA model.
On the [Hugging Face model page](https://huggingface.co/predibase/tldr_content_gen) for this adapter, you can find the Model tree on the right side. This shows the base model used.
In this case, the adapter is based on the `mistralai/Mistral-7B-v0.1` model.
Enter the identified base model name into the model input field on the Endpoint Create page.
Once the base model is selected, the 'Add LoRA adapter' button will become active. Click it to open the modal window for adding LoRA adapters.
In this modal, you can choose between "Project adapters" (adapters fine-tuned within Friendli Suite) and "Hugging Face adapters".
Select "Hugging Face adapters" and enter the Hugging Face Model ID of the adapter. For this tutorial, it's `predibase/tldr_content_gen`.
After adding the adapter, your screen should look like this. Now, select the instance type, configure the autoscaling options appropriately, and click the 'Create' button.
Once the endpoint is deployed, you'll see a screen like this. Navigate to the Playground page to quickly compare the adapter model and the base model.
In the Playground, use the highlighted dropdown menu to switch between the adapter model and the base model for experimentation and comparison.
That's it! You have successfully deployed a LoRA adapter on Friendli Dedicated Endpoints and experimented with it in the Playground.
Now you can explore deploying multiple adapters on a single endpoint (Multi-LoRA Endpoints) or use the API to send requests to the model and integrate it into your applications.
{children}
{caption &&
;
## 1. Log In or Sign Up
* If you have an account, log in using your preferred SSO or email/password combination.
* If you're new to FriendliAI, create an account for free.
{caption}
}
## 4. Deploy Your Endpoint
* Deploy your endpoint, using the model of your choice prepared from step 3, and the instance equipped with your desired GPU specification.
* You can also configure your replicas and the max-batch-size for your endpoint.
{children}
{caption &&
;
## Rollout and Rollback Endpoints Without Downtime
The versioning feature in Friendli Dedicated Endpoints helps you manage all changes to your deployed endpoints safely and transparently. When you update the configurationβlike changing the model, engine settings, or autoscalingβa new version is created instead of replacing the current one.
Each version captures a full snapshot of the deployment, including:
* Model name and artifact source
* Accelerator type and count
* Autoscaling and engine settings
* Metadata (creator, timestamps, comments)
{caption}
}
Use the comprehensive guides below to get started with each modality. ## Quick Navigation * [Image Generation](#image-generation) - Generate images from text prompts * [Vision (Image Understanding)](#vision-image-understanding) - Analyze and understand images * [Video Understanding](#video-understanding) - Process and analyze video content * [Audio and Speech](#audio-and-speech) - Convert audio to text and analyze audio ### Image Generation Transform text prompts into high-quality visuals with Friendli's image generation capabilities. #### Representative Models We support various trending image generation models including: * [FLUX.1-dev](https://friendli.ai/models/search?baseModel=black-forest-labs/FLUX.1-dev) * [FLUX.1-schnell](https://friendli.ai/models/search?baseModel=black-forest-labs/FLUX.1-schnell) * [See all image generation models](https://friendli.ai/models/search?input=TEXT\&output=IMAGE) #### API Usage
Existing applications can migrate with minimal effort, still using the official OpenAI SDKs. ### Specify the base URL and API key Initialize the OpenAI client using Friendliβs base URL and your Friendli token (API key). * **Serverless Endpoints**: `https://api.friendli.ai/serverless/v1`. * **Dedicated Endpoints**: `https://api.friendli.ai/dedicated/v1`. * **Container**: your own container's URL (e.g., `http://HOST:PORT/v1`). Get your Friendli token in [Friendli Suite β Settings β Tokens](https://friendli.ai/suite/setting/tokens).
FriendliAI will make sure your AI runs fast, affordable, and reliable at scale. ### Start building
Predictable, efficient scaling with full observability at scale.
Fast, affordable inference with simple pay-as-you-go pricing.
Kubernetes-native, designed for enhanced privacy, security, and governance.
#### What makes reasoning parsing tedious?
Different models handle reasoning in different ways.
Some models always generate reasoning, while others expose it as an optional feature.
The format also varies. The reasoning content may be wrapped in `Setting it to `true` enables reasoning, while `false` returns empty `
{children}
{caption &&
;
## 1. Log In or Sign Up
* If you have an account, log in using your preferred SSO or email/password combination.
* If you're new to FriendliAI, create an account for free.
{caption}
}
| Feature | **Chat Completion** | **Tool Assisted Chat Completion** | | ----------------- | ------------------------------------------------ | -------------------------------------------------------------------- | | **Response Type** | Based on internal knowledge | Uses external tools for enhanced, real-time responses | | **Capabilities** | Limited to pre-learned knowledge | Can interact with tools for data retrieval and task execution | | **Example** | "What is the weather today?" (general knowledge) | "What is the weather today?" (live API result) | | **Use Cases** | General conversation and Q\&A | Complex tasks like real-time updates, data analysis, file processing | *** ## Integrated Tools Tool Assisted API can also make use of integrated tools, which operate in the same way as built-in tools but require a connection to external services. Because they leverage specialized APIs or platforms, integrated tools are more likely to be production-ready compared to built-in tools, offering higher performance and broader functionality. ### `linkup:search`
**Description:**
[Linkup](https://www.linkup.so) seamlessly integrates with models served by FriendliAI to provide real-time web search capabilities. This enables AI applications to retrieve up-to-date facts, events, and information beyond the modelβs training data, complete with accurate citations.
Grounding responses in real-time data, Linkup improves precision, accuracy and factuality while delivering production-ready state-of-the-art web information retrieval.
**When Used:**
Automatically called when you need to retrieve current information from the web, such as recent news, real-time data, or up-to-date facts.
This tool is particularly useful for tasks requiring accurate and reliable web search results.
If you didnβt claim your free credits during onboarding, you can still do so later. Go to the **Team Settings > Billing** page and start product plans to receive them.
Once you successfully enter the coupon code, the additional credits will be applied immediately.
**Notes**
* If the coupon has expired: Check the expiration date. The free coupon may no longer be valid.
* If the coupon has already been used: It may have already been redeemed by someone on your team.
* If the coupon does not exist: Double-check the code to ensure it is correct.
* If the coupon limit has been reached: The coupon may no longer be available if all available uses have already been claimed.
Enter your coupon code to apply the credits to your team account. Credits are team-based, meaning all members in your team can use them.
{children}
{caption &&
;
Effective management of credentials is crucial when using Friendli Suite and its endpoints for authentication and authorization purposes.
This guide outlines when the credentials are required and provides instructions on how to manage them.
A Friendli Token serves as an alternative method of authorization to signing in with an email and a password.
You can generate a new Friendli Token through the [Friendli Suite](https://friendli.ai/suite), at your **'Personal settings'** page.
1. Go to the [Friendli Suite](https://friendli.ai/suite) and sign in with your account.
2. Click the profile icon at the top-right corner of the page.
3. Click **'Personal settings'** menu.
{caption}
}
{children}
{caption &&
;
Collaborate securely with your teammates.
## Using Projects
Projects are scoped to a team. Only members added to a project can access and manage its resources.
* Control who can manage and access resources.
* Group endpoints and related resources in one place.
* Ensure only the right people can create, update, or terminate endpoints.
{caption}
}
## Broad Model Coverage
Friendli supports tool calling for a wide range of openβsource and commercial models. You can browse available models on our [Models page](https://friendli.ai/models) and try them out with the Playground. ## Tool Calling with Friendli Friendli supports tool calling for a wide range of openβsource and commercial models. #### Tool Calling Parameters To enable tool calling, use the `tools`, `tool_choice`, and `parallel_tool_calls` parameters. | Parameter | Description | default | | --------------------- | ---------------------------------------------------------------------- | ------- | | `tools` | The list of tool objects that define the functions the model can call. | - | | `tool_choice` | Determines the tool calling behavior of the model. | `auto` | | `parallel_tool_calls` | Whether to let the model issue tool calls in parallel. | `True` | By default, the model decides whether to call a function and which one to use. With the `tool_choice` parameter, you can explicitly instruct the model to use a specific function. * `none`: Disable the use of tools. * `auto`: Enable the model to decide whether to use tools and which ones to use. * `required`: Force the model to use a tool, but the model chooses which one. * Named tool choice: Force the model to use a specific tool. It must be in the following format: ```json { "type": "function", "function": { "name": "get_current_weather" // The function name you want to specify } } ``` #### Response Schema Friendli follows the OpenAI function calling schema. Tool calls are returned in `choices[].message.tool_calls[]` with each item containing a `function.name` and JSONβstringified `function.arguments`. After executing a tool, append a new message with role: `tool`, the matching `tool_call_id`, and the tool result in content. ## Simple Example The example below walks through five steps: 1. Define a tool (`get_weather`) that retrieves weather information. 2. Ask a question that triggers tool use. 3. Let the model select the tool. 4. Execute the tool. 5. Generate the final answer using the tool result.
{children}
{caption &&
;
## Goals
* Use tool calling to build your own AI agent with [**Friendli Serverless Endpoints**](https://friendli.ai/products/serverless-endpoints)
* Check out the examples below to see how you can interact with state-of-the-art language models while letting them search the web, run Python code, etc.
* Feel free to make your own custom tools!
## Getting Started
1. Head to [**https://friendli.ai**](https://friendli.ai/get-started/serverless-endpoints), and create an account.
2. Grab a [Friendli Token](https://friendli.ai/suite/setting/tokens) to use Friendli Serverless Endpoints within an agent.
## Step 1. Playground UI
Experience tool calling on the Playground!
{caption}
}
{children}
{caption &&
;
## Introduction
This tutorial is designed to guide you through the process of easily deploying your models from the [W\&B Registry](https://docs.wandb.ai/guides/core/registry/) to Friendli Dedicated Endpoints in the W\&B UI. Through a series of step-by-step instructions and hands-on examples, youβll learn how to:
* **Configure a webhook** in W\&B to trigger deployments to Friendli Dedicated Endpoints.
* **Create a [webhook automation](https://docs.wandb.ai/guides/core/automations/create-automations/webhook/)** to automatically deploy model artifacts when adding new versions.
* **Deploy a model artifact** to Friendli Dedicated Endpoints by adding an alias in the W\&B Registry.
* **Understand how adding and removing aliases** affects deployments on Friendli Dedicated Endpoints.
### Why use W\&B webhook automation with Friendli Dedicated Endpoints?
W\&B users often rely on W\&B Registry to manage the lifecycle of models β from tracking experiment artifacts to promoting the best-performing models for production use. As a W\&B user, integrating Friendli Dedicated Endpoints directly into this workflow allows you to:
* **Streamline deployment**: Transition your models from experimentation to production with minimal effort. By leveraging W\&Bβs aliasing system and FriendliAIβs automated infrastructure, you eliminate the need for custom scripts or manual configurations.
* **Ensure deployment consistency**: Friendli Dedicated Endpoints include support for `idempotencyKey` to ensure the reliability of automated workflows. Each deployment trigger via webhook automation is tracked with a unique `idempotencyKey`, ensuring that operations like endpoint creation or updates are processed exactly once. It prevents duplicate or conflicting operations, giving you confidence in the consistency of your deployment.
By the end of this tutorial, youβll be equipped with the knowledge and skills necessary to seamlessly transfer your models from W\&B Registry to Friendli Dedicated Endpoints for efficient deployment. So, letβs get started and explore the possibilities of Friendli Dedicated Endpoints!
## Prerequisites
* A Friendli Suite account with access to [Friendli Dedicated Endpoints](/guides/dedicated_endpoints/introduction).
* A [personal access token](/guides/suite/personal_access_tokens) generated through Friendli Suite.
## Step 1: Create a secret
{caption}
}
{children}
{caption &&
;
OpenAPI reference of Friendli Suite API. You can interact with the API through HTTP requests from any language.
To send inference requests, send to the URI with the prefix: `https://api.friendli.ai`.
For more information, visit [FriendliAI](https://friendli.ai).
## Authentication
When using Friendli Suite API for inference requests, you need to provide a **Friendli Token** for authentication and authorization purposes.
A Friendli Token serves as an alternative method of authorization to signing in with an email and a password.
You can generate a new Friendli Token through the [Friendli Suite](https://friendli.ai/suite), at your **'Personal settings'** page by following the steps below.
1. Go to the [Friendli Suite](https://friendli.ai/suite) and sign in with your account.
2. Click the profile icon at the top-right corner of the page.
3. Click **'Personal settings'** menu.
{caption}
}
{children}
{caption &&
;
**Linkup** provides real-time web search capabilities. With Linkup integration in Friendli, you can easily enhance your AI applications with up-to-date facts, recent events, and current information that goes beyond what your model was trained on.
You can use Linkup's real-time web search through Friendli Serverless Endpoints with just a few simple steps.
## How to use
### For Playground Testing
1. Create an account at [**https://friendli.ai**](https://friendli.ai).
2. Subscribe to the free trial of the Friendli Serverless Endpoints product. ([guide](/guides/suite/free_credits#receiving-when-you-start))
3. Go to **Serverless Endpoints** from your Project and click **'Try'** button to open the playground.
4. In the playground, open the **Tools** panel and select **Search the web (Linkup)** to test the integration.
{caption}
}
*/}
You can use [**LlamaIndex**](https://github.com/run-llama/llama_index) to interact with FriendliAI.
This makes migration of existing applications already using LlamaIndex particularly easy.
## How to use
Before you start, ensure you've already obtained the `FRIENDLI_TOKEN` from the [Friendli Suite](https://friendli.ai/suite/setting/tokens).
```python
pip install llama-index llama-index-llms-friendli
```
### Instantiation
Now we can instantiate our model object and generate chat completions.
The default model (i.e. `meta-llama-3.3-70b-instruct`) will be used if no model is specified.
```python
import os
from llama_index.llms.friendli import Friendli
os.environ['FRIENDLI_TOKEN'] = "YOUR_FRIENDLI_TOKEN"
llm = Friendli(model="meta-llama-3.3-70b-instruct")
```
### Chat completion
Generate a response from a given conversation.
