{children}
{caption &&
;
## What are Endpoints?
Endpoints are the actual deployments of your models on a dedicated GPU resource.
They provide a stable and efficient interface to serve your models in real-world applications, ensuring high availability and optimized performance.
With endpoints, you can manage model versions, scale resources, and seamlessly integrate your model into production environments.
### Key Capabilities of Endpoints:
* **Efficient Model Serving**: Deploy models on powerful GPU instances optimized for your use case.
* **Flexibility with Multi-LoRA Models**: Serve multiple fine-tuned adapters alongside base models.
* **Autoscaling**: Automatically adjust resources to handle varying workloads, ensuring optimal performance and cost efficiency.
* **Monitoring and Management**: Check endpoint health, adjust configurations, and view logs directly from the platform.
* **Interactive Testing**: Use the integrated playground to test your models before integrating them into applications.
* **API Integration**: Access your models via robust OpenAI-compatible APIs, enabling easy integration into any system.
## Creating Endpoints
You can create your endpoint by specifying the name, the model, and the instance configuration, consisting of your desired GPU specification.
{caption}
}
{children}
{caption &&
;
While following through our tutorials, you might have had questions regarding the details of the requirements and specifications.
We have listed out the frequently asked questions and as a separate document. Please refer to the relevant information below:
## Format Requirements
### General requirements for a model
* A model should be in safetensors format.
* The model should NOT be nested inside another directory.
* Including other arbitrary files (that are not in the list) is totally fine. However, those files will not be downloaded nor used.
| Required | Filename | Description |
| -------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Yes | *safetensors* | Model weight, e.g. model.safetensors. Use model.safetensors.index.json for split safetensors files |
| Yes | config.json | Model config that includes the architecture. ([Supported Models on Friendli](https://friendli.ai/models)) |
| Yes | tokenizer.json | Tokenizer for the model |
| No | tokenizer\_config.json | Tokenizer config. This should be present & have a `chat_template` field for the Friendli Engine to provide chat APIs |
| No | special\_tokens\_map.json | |
### General requirements for a dataset
* Read our documentation on the [fine-tuning dataset format](/guides/dedicated_endpoints/fine-tuning#dataset-format) for information on the dataset requirements.
## 3rd-party account integration
{caption}
}
{children}
{caption &&
;
### In order to fine-tune large generic models for your specific purpose, you may fine-tune models on Friendli Dedicated Endpoints.
Effortlessly fine-tune your model with [Friendli Dedicated Endpoints](https://friendli.ai/products/dedicated-endpoints),
which leverages the Parameter-Efficient Fine-Tuning (PEFT) method to reduce training costs while preserving model quality, similar to full-parameter fine-tuning.
This can make your model become an expert on a specific topic, and prevent hallucinations from your model.
## Table of Contents
1. **[How to Select Your Base Model](#how-to-select-your-base-model)**
2. **[How to Upload Your Dataset](#how-to-upload-your-dataset)**
3. **[How to Create Your Fine-tuning Job](#how-to-create-your-fine-tuning-job)**
4. **[How to Monitor Progress](#how-to-monitor-progress)**
5. **[How to Deploy the Fine-tuned Model](#how-to-deploy-the-fine-tuned-model)**
6. **[Resources](#resources)**
By the end of this guide, you will understand how you can effectively fine-tune your generative AI models by using Friendli Dedicated Endpoints.
## How to Select Your Base Model
Through our (1) Hugging Face Integration and (2) Weights & Biases (W\&B) Integration, you can select the base model to fine-tune.
Explore and find open-source models that are supported on Friendli Dedicated Endpoints [here](https://friendli.ai/models).
For guidance on the necessary format and file requirements, especially when using your own models, review the FAQ section on [general requirements for a model](/guides/dedicated_endpoints/faq#general-requirements-for-a-model).
* **Hugging Face Model**
* **Weights & Biases Model**
### Hugging Face Integration
Integrate your [Hugging Face account](https://huggingface.co) to access your private repo or a gated repo.
Go to [**Personal settings > Account > Hugging Face integration**](https://suite.friendli.ai/default-team/settings/account) and save your [Hugging Face access token](https://huggingface.co/docs/hub/security-tokens).
This access token will be used upon creating your fine-tuning jobs.
{caption}
}
{children}
{caption &&
;
#### Hands-on Tutorial
Deploying `meta-llama-3-8b-instruct` LLM from Hugging Face using Friendli Dedicated Endpoints
## Introduction
With Friendli Dedicated Endpoints, you can easily spin up scalable, secure, and highly available inference deployments, without the need for extensive infrastructure expertise or significant capital expenditures.
This tutorial is designed to guide you through the process of launching and deploying LLMs using Friendli Dedicated Endpoints. Through a series of step-by-step instructions and hands-on examples, you'll learn how to:
* Select and deploy pre-trained LLMs from Hugging Face repositories
* Deploy and manage your models using the Friendli Engine
* Monitor and optimize your inference deployments
By the end of this tutorial, you'll be equipped with the knowledge and skills necessary to unlock the full potential of LLMs in your applications, products, and services. So, let's get started and explore the possibilities of Friendli Dedicated Endpoints!
## Prerequisites:
* A Friendli Suite account with access to [Friendli Dedicated Endpoints](https://suite.friendli.ai)
* A Hugging Face account with access to the [meta-llama-3-8b-instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) model
## Step 1: Create a new endpoint
1. Log in to your Friendli Suite account and navigate to the Friendli Dedicated Endpoints dashboard.
2. If not done already, start the free trial for Dedicated Endpoints.
3. Create a new project, then click on the "New Endpoint" button.
4. Fill in the basic information:
* Endpoint name: Choose a unique name for your endpoint (e.g., "My New Endpoint").
5. Select the model:
{caption}
}
{children}
{caption &&
;
## 1. Log In or Sign Up
* If you have an account, log in using your preferred SSO or email/password combination.
* If you're new to FriendliAI, create an account for free.
## 2. Access Friendli Dedicated Endpoints
* On your dashboard, find the "Friendli Dedicated Endpoints" section.
* If unauthorized, ask your team admin to provide access to the Friendli Dedicated Endpoints at the team settings.
## 3. Select Your Project
* Either create a new project, or choose from your existing projects for your workload.
## 4. Prepare Your Model
* Choose a model that you wish to serve from HuggingFace, Weights & Biases, or upload your custom model on our cloud.
## 5. Deploy Your Endpoint
* Deploy your endpoint, using the model of your choice prepared from step 3, and the instance equipped with your desired GPU specification.
* You can also configure your replicas and the max-batch-size for your endpoint.
{caption}
}
{children}
{caption &&
;
#### Hands-on Tutorial
Deploying `meta-llama-3-8b-instruct` LLM from W\&B using Friendli Dedicated Endpoints
## Introduction
With Friendli Dedicated Endpoints, you can easily spin up scalable, secure, and highly available inference deployments, without the need for infrastructure expertise or significant capital expenditures.
This tutorial is designed to guide you through the process of launching and deploying LLMs using Friendli Dedicated Endpoints. Through a series of step-by-step instructions and hands-on examples, you'll learn how to:
* Select and deploy pre-trained LLMs from W\&B artifacts
* Deploy and manage your models using the Friendli Engine
* Monitor and optimize your inference deployments
By the end of this tutorial, you'll be equipped with the knowledge and skills necessary to unlock the full potential of LLMs in your applications, products, and services. So, let's get started and explore the possibilities of Friendli Dedicated Endpoints!
## Prerequisites:
* A Friendli Suite account with access to [Friendli Dedicated Endpoints](https://suite.friendli.ai)
* A W\&B account with an api key (as an access token)
## Step 1: Create a new endpoint
1. Log in to your Friendli Suite account and navigate to the Friendli Dedicated Endpoints dashboard.
2. If not done already, start the free trial for Dedicated Endpoints.
3. Create a new project, then click on the "New Endpoint" button.
4. [Integrate your W\&B account with an api key.](https://wandb.ai/settings#api)
5. Fill in the basic information:
* Endpoint name: Choose a unique name for your endpoint (e.g., "My New Endpoint").
6. Select the model:
{caption}
}
{children}
{caption &&
;
Effective management of credentials is crucial when using Friendli Suite and its endpoints for authentication and authorization purposes.
This guide outlines when the credentials are required and provides instructions on how to manage them.
A Friendli Token serves as an alternative method of authorization to signing in with an email and a password.
You can generate a new Friendli Token through the [Friendli Suite](https://suite.friendli.ai), at your **"Personal settings"** page.
1. Go to the [Friendli Suite](https://suite.friendli.ai) and sign in with your account.
2. Click the profile icon at the top-right corner of the page.
3. Click **"Personal settings"** menu.
{caption}
}
## Simple Example
In the example below, which consists of 1 to 5 steps, we define a `get_weather` function that retrieves weather information, ask a question that prompts the model to use the tool, and execute the tool to execute the final response.
{children}
{caption &&
;
## 1. Log In or Sign Up
* If you have an account, log in using your preferred SSO or email/password combination.
* If you're new to FriendliAI, create an account for free.
## 2. Access Friendli Serverless Endpoints
* On your dashboard, find the "Friendli Serverless Endpoints" section.
* Click the "Go to playground" button to start generating text.
## 3. Select a Model
* Browse the available generative models.
* Choose the model that best aligns with your desired use case.
* First-time users receives a free trial to explore Friendli Serverless Endpoints without any financial commitment.
{caption}
}
{children}
{caption &&
;
## What is Tool Assisted API?
**Tool Assisted API** enhances a model's capabilities by integrating **tools** that extend its functionality beyond simple conversational interactions.
By using this API, the model becomes more dynamic, providing more comprehensive and actionable responses.
Currently, **[Friendli Serverless Endpoints](/guides/serverless_endpoints/introduction)** supports a variety of built-in tools specifically designed for **Chat Completion** tasks.
***
### What is Chat Completion?
**[Chat completion](/openapi/serverless/chat-completions)** refers to a model's ability to generate responses in a conversation. Given a sequence of messages or conversation turns, the model processes the input and generates a response based on its internal knowledge and training data.
* **Example**:
* **User**: "What is the capital of France?"
* **Model**: "The capital of France is Paris."
However, chat completion has its limitations—it is restricted to the knowledge the model has learned during its training and cannot access real-time or external data.
***
### Is Chat Completion Different from Tool Assisted Chat Completion?
Yes, **[Tool Assisted Chat Completion](/openapi/serverless/tool-assisted-chat-completions)** goes beyond basic chat completion by integrating external tools to enhance the conversation. This allows the model to access real-time data, perform specific tasks, and interact with external systems in ways that chat completion alone cannot achieve.
* **Example**:
* **User**: "What is the weather today?"
* **Model without Tool Access**: Relies on pre-learned information, potentially giving outdated or generalized answers.
* **Model with Tool Access**: Calls a weather API to retrieve live data and responds: "The weather today in New York is 72°F with clear skies."
With tool access, the model provides a more accurate and up-to-date response.
Additionally, some tasks—such as file processing or complex calculations—cannot be performed by the model alone but can be handled with the help of tools.
* **Example**:
* **User**: "Can you extract the text from this document?" (provides a file)
* **Model without Tool Access**: "I cannot extract data from files directly."
* **Model with Tool Access**: Extracts the text from the provided file and responds: "Using the `file:text` tool, I've extracted the following text: \[Text from the file]."
When no tools are specified, the model will respond using only its internal knowledge.
***
### Benefits of Tool Assisted Chat Completion
Tool Assisted Chat Completion offers several advantages over basic chat completion:
* **Real-Time Data Access**: The model can pull live information.
* **Extended Capabilities**: The model can perform complex tasks like running calculations, executing code, extracting text from files, and interacting with databases and APIs.
***
### Comparison: Chat Completion vs. Tool Assisted Chat Completion
{caption}
}| Feature | **Chat Completion** | **Tool Assisted Chat Completion** | | ----------------- | ------------------------------------------------ | -------------------------------------------------------------------- | | **Response Type** | Based on internal knowledge | Uses external tools for enhanced, real-time responses | | **Capabilities** | Limited to pre-learned knowledge | Can interact with tools for data retrieval and task execution | | **Example** | "What is the weather today?" (general knowledge) | "What is the weather today?" (live API result) | | **Use Cases** | General conversation and Q\&A | Complex tasks like real-time updates, data analysis, file processing | *** ## Built-In Tools Tool Assisted API automatically selects the best tool to perform an action based on user input when a specific tool is enabled. These tools can handle various operations, such as calculations, statistical analysis, web search, file content extraction, and code execution. Below is a more detailed description of the available tools in Tool Assisted API and when they are typically used: ### `math:calculator` **Description:** Performs basic arithmetic operations like addition, subtraction, multiplication, division, and more complex calculations like and square roots or exponents. It is useful for any tasks requiring mathematical computation. **When Used:** Automatically called when mathematical expressions or calculations are required. Whether you're solving equations, calculating percentages, or handling financial calculations, this tool performs the task for you. ### `math:statistics` **Description:** Performs statistical analysis, including calculating mean, median, mode, standard deviation, and correlations. It is tailored for situations where you need to analyze or interpret numeric datasets to understand trends or patterns. **When Used:** Automatically called when analyzing numeric data or generating insights from datasets, like summarizing survey results, or calculating probabilities. ### `math:calendar` **Description:** Handles date-related data, such as calculating date differences or finding specific days in the past or future. It is effective in managing and manipulating calendar-based information. **When Used:** Automatically called when operations involving dates or time spans are required, like finding how many days remain until an event. For example, figuring out how many days are left until an event, determining the day of the week for a specific date, or calculating the duration between two dates. ### `web:search` **Description:** Retrieves information from the web based on search queries. It fetches information based on keywords and helps gather knowledge or insights from online sources. **When Used:** Automatically called when you ask questions or seek information that requires external research or the latest data from the web. Whether it is looking up definitions, recent news, or general web searches, this tool handles such tasks effectively. ### `web:url` **Description:** Extracts specific data from a given website. You can provide a URL, and the tool will fetch the relevant content, including text, metadata, or other embedded information, from that web page. **When Used:** Automatically called when extracting content from a provided URL, such as fetching text from articles or blog posts. ### `code:python-interpreter` **Description:** Executes Python code directly within the platform for custom scripts, data processing, or automation. You can run Python scripts, test snippets of code, or automate tasks through coding logic. **When Used:** Automatically called when tasks involve writing or running Python scripts, such as custom data manipulations or logic-based automation. ### `file:text` **Description:** Reads and extracts text from files, supporting only `.txt` and `.pdf` formats. To use this tool, you must provide the file IDs. (For now, only one file is supported.) After uploading a file in the playground, you can copy the file ID by clicking on the files icon in the left sidebar and selecting the option from the dropdown menu next to the uploaded file.
{children}
{caption &&
;
To send inference requests, send to the URI with the prefix: `https://api.friendli.ai`.\
For more information, visit [FriendliAI](https://friendli.ai).
## Authentication
When using Friendli Suite API for inference requests, you need to provide a **Friendli Token** for authentication and authorization purposes.
A Friendli Token serves as an alternative method of authorization to signing in with an email and a password.
You can generate a new Friendli Token through the [Friendli Suite](https://suite.friendli.ai), at your **"Personal settings"** page by following the steps below.
1. Go to the [Friendli Suite](https://suite.friendli.ai) and sign in with your account.
2. Click the profile icon at the top-right corner of the page.
3. Click **"Personal settings"** menu.
{caption}
}
*/}
You can use [**LlamaIndex**](https://github.com/run-llama/llama_index) to interact with FriendliAI.
This makes migration of existing applications already using LlamaIndex particularly easy.
## How to use
Before you start, ensure you've already obtained the `FRIENDLI_TOKEN` from the [Friendli Suite](https://suite.friendli.ai/default-team/settings/tokens).
```python
pip install llama-index llama-index-llms-friendli
```
### Instantiation
Now we can instantiate our model object and generate chat completions.
The default model (i.e. `mixtral-8x7b-instruct-v0-1`) will be used if no model is specified.
```python
from llama_index.llms.friendli import Friendli
import os
os.environ['FRIENDLI_TOKEN'] = "YOUR_FRIENDLI_TOKEN"
llm = Friendli(model="meta-llama-3.3-70b-instruct")
```
### Chat completion
Generate a response from a given conversation.
