Deploy with Hugging Face Models

​

Prerequisites

• A Friendli Suite account with access to Friendli Suite > Dedicated Endpoints
• A Hugging Face account with access to the meta-llama-3-8b-instruct model

​

Step 1: Create a New Endpoint

1. Sign in to your Friendli Suite account and navigate to the Dedicated Endpoints.
2. If your team doesn’t have any credits, purchase some before deploying.
3. Create a new project, then click the New Endpoint button.
4. Fill in the basic information:

• Endpoint name: Enter a unique name for your endpoint (e.g., “My New Endpoint”).

5. Select the model:

• Model Repository: Select Hugging Face as the model provider.
• Model ID: Enter meta-llama/Meta-Llama-3-8B-Instruct as the model id. As the search bar loads the list, click the top result that exactly matches the repository id.

6. Select the instance:

• Instance configuration: Select a suitable instance type based on your performance requirements. FriendliAI suggests 1x A100 80G for most models.

7. Edit the configurations:

• Autoscaling: By default, the autoscaling ranges from 0 to 2 replicas. This means that the deployment will sleep when it’s not being used, which reduces cost.
• Advanced configuration: Some LLM options including the batch size and token configurations are mutable. For this tutorial, leave them as-is.

8. Click Deploy to create a new endpoint.

​

Step 2: Test the Endpoint

1. Wait for Friendli Suite to create and initialize the deployment. This may take a few minutes.

2. In the Playground section, you can enter a sample input prompt (e.g., “What is the capital of France?”).
3. Click the right arrow button to send the inference request.

4. Use the Metrics and Logs sections to monitor the endpoint.

​

Step 3: Send Requests by Using cURL or Python

1. As described in the API docs, you can send instructions with the following code:

import os
from openai import OpenAI
client = OpenAI(
    api_key=os.getenv("API_KEY"),
    base_url="https://api.friendli.ai/dedicated/v1",
)

chat_completion = client.chat.completions.create(
    model="YOUR_ENDPOINT_ID",
    messages=[
        {
            "role": "user",
            "content": "Tell me how to make a delicious pancake"
        }
    ]
)
print(chat_completion.choices[0].message.content)

curl -X POST https://api.friendli.ai/dedicated/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "model": "(endpoint-id)",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "max_tokens": 200,
    "top_k": 1
  }'

2. You can update the model and change almost everything by clicking the update button.