Friendli Dedicated Endpoints
Frequently Asked Questions and Troubleshooting
While following through our tutorials, you might have had questions regarding the details of the requirements and specifications. We have listed out the frequently asked questions as a separate document.
While following through our tutorials, you might have had questions regarding the details of the requirements and specifications.
We have listed out the frequently asked questions as a separate document. Please refer to the relevant information below:
Format Requirements
General requirements for a model
- A model should be in safetensors format.
- The model should NOT be nested inside another directory.
- Including other arbitrary files (that are not in the list) is totally fine. However, those files will not be downloaded nor used.
| Required | Filename | Description |
|---|---|---|
| Yes | safetensors | Model weight, e.g. model.safetensors. Use model.safetensors.index.json for split safetensors files |
| Yes | config.json | Model config that includes the architecture. (Supported Models on Friendli) |
| No | tokenizer.json | Tokenizer for the model |
| No | tokenizer_config.json | Tokenizer config. This should be present & have a chat_template field for the Friendli Engine to provide chat APIs |
| No | special_tokens_map.json |
General requirements for a dataset
The dataset should satisfy the following conditions:- The dataset must contain a column named “messages”.
- Each row in the “messages” column should be compatible with the chat template of the base model.
For example,
tokenizer_config.jsonofmistralai/Mistral-7B-Instruct-v0.2is a template that repeats the messages of a user and an assistant. Concretely, each row in the “messages” field should follow a format like:[{"role": "user", "content": "The 1st user's message"}, {"role": "assistant", "content": "The 1st assistant's message"}]. In this case,HuggingFaceH4/ultrachat_200kis a dataset that is compatible with the chat template.
3rd-party account integration
How to integrate a Hugging Face account
- Log in to Hugging Face, then navigate to user settings → access tokens → User Access Tokens. Acquire a token.
- You may use a fine-grained token. In this case, please make sure the token has view permission for the repository you’d like to use.
- Integrate the key in Friendli Suite → Personal settings → Integrations
If you revoke / invalidate the key, you will have to update the key in order to not disrupt ongoing deployments, or to launch a new inference deployment.
How to integrate a W&B account
- Log in to W&B, then navigate to user settings → danger zone → API keys. Acquire a token.
- Integrate the key in Friendli Suite → Personal settings → Integrations
If you revoke / invalidate the key, you will have to update the key in order to not disrupt ongoing deployments, or to launch a new inference deployment.
Extra: How to upload a safetensors format model to W&B using W&B CLI
- Install the CLI and log in using the API key → Command Line Interface | Weights & Biases Documentation
- Upload the model as a W&B artifact using the command below:
- With all this, the W&B artifact will look like this:
Using 3rd-party model
How to use a W&B artifact as a model
- Use the full name of the artifact
- The artifact name must be in the format of:
org/project/artifact_id:version
How to use a Hugging Face repository as a model
- Use the repository id of the model. You may select the entry from the list of autocompleted model repositories.
- You may choose specific branch, or manually enter a commit hash.
Troubleshooting
Can’t access the artifact
- The artifact might be nonexistent, or hidden so that you cannot access it.
You don’t have access to this gated model
- The repository is gated. Please follow the steps and gain approval from the owner using Hugging Face Hub.
The repository / artifact is invalid
- The model does not meet the requirements. Please check if the model follows a correct safetensors format.
The architecture is not supported
- The model architecture is not supported. Please refer to Supported Models on Friendli.