Running Friendli Container on SageMaker

​

Introduction

​

General Workflow

1. Create a Model: Within SageMaker Inference, define a new model by specifying the model artifacts in your S3 bucket and the Friendli Container image from ECR.
2. Configure the Endpoint: Create a SageMaker Inference endpoint configuration by selecting the instance type and the number of instances required.
3. Create the Endpoint: Use the configured settings to launch a SageMaker Inference endpoint.
4. Invoke the Endpoint: Once deployed, send requests to your endpoint to receive inference responses.

​

Prerequisites

​

Using the AWS Console

​

Step 1: Creating a Model

• Model settings:
  • Model name: A model name.
  • IAM role: An IAM role that includes the AmazonSageMakerFullAccess policy.
• Container definition 1:
  • Container input option: Select the “Provide model artifacts and inference image location”.
  • Model Compression Type:
    • To use a model in the S3 bucket:
      • When the model is compressed, select CompressedModel.
      • Otherwise, select UncompressedModel.
    • When using a model from the Hugging Face hub, any option would work fine.
  • Location of inference code image: Specify the ARN of the ECR repo for the Friendli Container.
  • Location of model artifacts (optional):
    • To use a model in the S3 bucket: Specify the S3 URI where your model is stored. Ensure the file structure matches the directory format compatible with the --hf-model-name option of the Friendli Container.
    • When using a model from the Hugging Face hub, you can leave this field empty.
• Environment variables:
  • Always required:
    • FRIENDLI_CONTAINER_SECRET: Your Friendli Container Secret. Refer to Prepare Your Container Secret to learn how to get the container secret.
    • SAGEMAKER_MODE: Set this to True.
    • SAGEMAKER_NUM_DEVICES: Number of devices to use for tensor parallelism degree.
  • Required when using a model in the S3 bucket:
    • SAGEMAKER_USE_S3: Set this to True.
  • Required when using a model from the Hugging Face hub:
    • SAGEMAKER_HF_MODEL_NAME: The Hugging Face model name (e.g., mistralai/Mistral-7B-Instruct-v0.2).
    • For private or gated model repos:
      • HF_TOKEN: The Hugging Face secret access token.

​

Step 2: Creating an Endpoint Configuration

• Endpoint configuration:
  • Endpoint configuration name: The name of this endpoint configuration.
  • Type of endpoint: For real-time inference, select Provisioned.
• Variants:
  • To create a “Production” variant, click Create production variant.
  • Select the model that you have created in Step 1.
  • Configure the instance type and count by clicking Edit in the Actions column.
  • Create the endpoint configuration by clicking Create endpoint configuration.

​

Step 3: Creating a SageMaker Inference Endpoint

• Select Use an existing endpoint configuration.
• Select the endpoint configuration created in Step 2.
• Finish by clicking the Create endpoint button.

​

Step 4: Invoking the Endpoint

import boto3
import json

endpoint_name = "FILL OUT ENDPOINT NAME"
region_name = "FILL OUT AWS REGION"

sagemaker_runtime = boto3.client("sagemaker-runtime", region_name=region_name)

prompt = "Story title: 3 llamas go for a walk\nSummary: The 3 llamas crossed a bridge and something unexpected happened\n\nOnce upon a time"
payload = {
    "prompt": prompt,
    "max_tokens": 512,
    "temperature": 0.8,
}

response = sagemaker_runtime.invoke_endpoint(
    EndpointName=endpoint_name,
    Body=json.dumps(payload),
    ContentType="application/json",
)

print(response['Body'].read().decode('utf-8'))

​

Using the boto3 SDK

import boto3
from sagemaker import get_execution_role

sm_client = boto3.client(service_name='sagemaker')
runtime_sm_client = boto3.client(service_name='sagemaker-runtime')

account_id = boto3.client('sts').get_caller_identity()['Account']
region = boto3.Session().region_name
role = get_execution_role()

endpoint_name="FILL OUT ENDPOINT NAME"
model_name="FILL OUT MODEL NAME"
container = "FILL OUT ECR IMAGE NAME"  # <ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com/IMAGE

instance_type = "ml.g5.12xlarge" # instance type

container = {
    'Image': container,
    'Environment': {
        "HF_TOKEN": "<HUGGINGFACE ACCESS TOKEN>",
        "FRIENDLI_CONTAINER_SECRET": "<FRIENDLI CONTAINER SECRET>",
        "SAGEMAKER_HF_MODEL_NAME": "<HUGGINGFACE MODEL NAME>",  # e.g., meta-llama/Meta-Llama-3-8B
        "SAGEMAKER_MODE": "True",  # Should be true
        "SAGEMAKER_NUM_DEVICES": "4",  # Number of GPUs in `instance_type`
    }
}

endpoint_config_name = 'FILL OUT ENDPOINT CONFIG NAME'

# Create a model
create_model_response = sm_client.create_model(
    ModelName=model_name,
    ExecutionRoleArn=role,
    Containers=[container],
)

# Create an endpoint configuration
create_endpoint_config_response = sm_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[
        {
            'InstanceType': instance_type,
            'InitialInstanceCount': 1,
            'InitialVariantWeight': 1,
            'ModelName': model_name,
            'VariantName': 'AllTraffic',
        },
    ],
)

endpoint_name = "FILL OUT ENDPOINT NAME"

# Create an endpoint
sm_client.create_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=endpoint_config_name,
)

sm_client.describe_endpoint(EndpointName=endpoint_name)