# Configuration
Source: https://friendli.ai/docs/archive/container/guides/configuration

Configuration reference for Friendli Container: how to pass launch options, serve across multiple GPUs, enable quantization, and run MoE models.

This page is the configuration reference for Friendli Container—how to pass launch options, serve across multiple GPUs, and tune serving for your model. If you haven't run a container yet, start with the [Quickstart](/docs/archive/container/guides/quickstart).

Friendli Container supports direct loading of [`safetensors`](https://huggingface.co/docs/safetensors/index) checkpoints—compatible with [Hugging Face transformers](https://huggingface.co/docs/transformers)—for many model types. You can find the complete list of supported models on the [Supported Models page](https://friendli.ai/models?products=CONTAINER). If your model is not on the list, [contact support](mailto:support@friendli.ai).

## Passing Launch Options

Launch options are passed as arguments after the image name in your `docker run` command:

```sh theme={null}
# Fill the values of following variables.
export HF_MODEL_NAME=""  # Hugging Face model name (e.g., "meta-llama/Meta-Llama-3-8B-Instruct")
export FRIENDLI_CONTAINER_SECRET=""  # Friendli container secret

docker run --gpus '"device=0"' -p 8000:8000 \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  registry.friendli.ai/trial \
    --hf-model-name $HF_MODEL_NAME \
    [LAUNCH_OPTIONS]
```

Replace `[LAUNCH_OPTIONS]` with the options described in [Launch Options](#launch-options). Running the command above starts a Docker container that exposes an HTTP endpoint for handling inference requests.

## Multi-GPU Serving

Friendli Container supports ***tensor parallelism*** and ***pipeline parallelism*** for multi-GPU inference.

### Tensor Parallelism

Use tensor parallelism when serving large models that exceed the memory capacity of a single GPU. It distributes parts of the model's weights across multiple GPUs. To use tensor parallelism with Friendli Container:

1. Specify multiple GPUs for `$GPU_ENUMERATION` (e.g., '"device=0,1,2,3"').
2. Use the `--num-devices` (or `-d`) option to specify the tensor parallelism degree (e.g., `--num-devices 4`).

### Examples

<Tabs>
  <Tab title="Deploying Models on a Single GPU">
    This is an example running [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) with a single GPU.

    ```sh theme={null}
    export FRIENDLI_CONTAINER_SECRET=""  # Friendli container secret (leave it if it's already set in your environment)
    export HF_TOKEN=""  # Access token from Hugging Face (see the caution below)

    docker run -p 8000:8000 --gpus '"device=0"' \
      -e HF_TOKEN=$HF_TOKEN \
      -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
      -v ~/.cache/huggingface:/root/.cache/huggingface \
      registry.friendli.ai/trial \
        --hf-model-name meta-llama/Llama-3.1-8B-Instruct
    ```

    <Warning>
      Since downloading `meta-llama/Llama-3.1-8B-Instruct` is allowed only for authorized users, you need to provide your [Hugging Face User Access Token](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hftoken) through the `HF_TOKEN` environment variable. It works the same for all private repositories.
    </Warning>
  </Tab>

  <Tab title="Deploying Models on Multi-GPU">
    This is an example running [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) with a multi-GPU setup.

    ```sh {5,11} theme={null}
    export FRIENDLI_CONTAINER_SECRET=""  # Friendli container secret (leave it if it's already set in your environment)
    export HF_TOKEN=""  # Access token from Hugging Face (see the caution below)

    docker run -p 8000:8000 \
      --ipc=host --gpus '"device=0,1"' \
      -e HF_TOKEN=$HF_TOKEN \
      -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
      -v ~/.cache/huggingface:/root/.cache/huggingface \
      registry.friendli.ai/trial \
        --hf-model-name meta-llama/Llama-3.1-70B-Instruct \
        --num-devices 2
    ```

    <Warning>
      Since downloading `meta-llama/Llama-3.1-70B-Instruct` is allowed only for authorized users, you need to provide your [Hugging Face User Access Token](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hftoken) through the `HF_TOKEN` environment variable. It works the same for all private repositories.
    </Warning>
  </Tab>
</Tabs>

## Quantization

Friendli Container supports **online quantization**, which quantizes a model instantly when you launch it, as well as serving pre-quantized models. If your model is already quantized or needs to be quantized, check [Quantization](/docs/archive/container/guides/quantization) for more details.

## Serving MoE Models

Running MoE (Mixture of Experts) models requires an additional step to search the execution policy. See [Serving MoE Models](/docs/archive/container/guides/serving-moe-models) to learn how to launch Friendli Container for the MoE model.

## Options for Running Friendli Container

### General Options

| Options     | Type | Summary                                | Default | Required |
| ----------- | ---- | -------------------------------------- | ------- | -------- |
| `--version` | -    | Print Friendli Container version.      | -       | ❌        |
| `--help`    | -    | Print Friendli Container help message. | -       | ❌        |

### Launch Options

| Options                           | Type                        | Summary                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Default             | Required |
| --------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | -------- |
| `--web-server-port`               | INT                         | Web server port.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | 8000                | ❌        |
| `--metrics-port`                  | INT                         | Prometheus metrics export port.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | 8281                | ❌        |
| `--hf-model-name`                 | TEXT                        | Model name hosted on the Hugging Face Models Hub or a path to a local directory containing a model. When a model name is provided, Friendli Container first checks if the model is already cached at \~/.cache/huggingface/hub and uses it if available. If not, it will download the model from the Hugging Face Models Hub before launching the container. When a local path is provided, it will load the model from the location without downloading. This option is only available for models in a safetensors format. | -                   | ❌        |
| `--tokenizer-file-path`           | TEXT                        | Absolute path of tokenizer file. This option is not needed when `tokenizer.json` is located under the path specified at `--ckpt-path`.                                                                                                                                                                                                                                                                                                                                                                                      | -                   | ❌        |
| `--tokenizer-add-special-tokens`  | BOOLEAN                     | Whether or not to add special tokens in tokenization. Equivalent to Hugging Face Tokenizer's `add_special_tokens` argument. The default value is **false** for versions \< v1.6.0.                                                                                                                                                                                                                                                                                                                                          | `true`              | ❌        |
| `--tokenizer-skip-special-tokens` | BOOLEAN                     | Whether or not to remove special tokens in detokenization. Equivalent to Hugging Face Tokenizer's `skip_special_tokens` argument.                                                                                                                                                                                                                                                                                                                                                                                           | `true`              | ❌        |
| `--dtype`                         | CHOICE: \[bf16, fp16, fp32] | Data type of weights and activations. Choose one of \<fp16\|bf16\|fp32>. This argument applies to non-quantized weights and activations. If not specified, Friendli Container follows the value of `torch_dtype` in `config.json` file or assumes fp16.                                                                                                                                                                                                                                                                     | fp16                | ❌        |
| `--bad-stop-file-path`            | TEXT                        | JSON file path that contains stop sequences or bad words/tokens.                                                                                                                                                                                                                                                                                                                                                                                                                                                            | -                   | ❌        |
| `--num-request-threads`           | INT                         | Thread pool size for handling HTTP requests.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | 4                   | ❌        |
| `--timeout-microseconds`          | INT                         | Server-side timeout for client requests, in microseconds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | 0 (no timeout)      | ❌        |
| `--ignore-nan-error`              | BOOLEAN                     | If set to True, ignore NaN error. Otherwise, respond with a 400 status code if NaN values are detected while processing a request.                                                                                                                                                                                                                                                                                                                                                                                          | -                   | ❌        |
| `--max-batch-size`                | INT                         | Max number of sequences that can be processed in a batch.                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | 384                 | ❌        |
| `--num-devices`, `-d`             | INT                         | Number of devices to use as the tensor parallelism degree.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | 1                   | ❌        |
| `--search-policy`                 | BOOLEAN                     | Searches for the best engine policy for the given combination of model, hardware, and parallelism degree. Learn more about policy search at [Optimizing Inference with Policy Search](/docs/archive/container/guides/optimizing-inference-with-policy-search).                                                                                                                                                                                                                                                              | false               | ❌        |
| `--terminate-after-search`        | BOOLEAN                     | Terminates engine container after the policy search.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | false               | ❌        |
| `--algo-policy-dir`               | TEXT                        | Path to directory containing the policy file. The default value is the current working directory. Learn more about policy search at [Optimizing Inference with Policy Search](/docs/archive/container/guides/optimizing-inference-with-policy-search).                                                                                                                                                                                                                                                                      | current working dir | ❌        |
| `--adapter-model`                 | TEXT                        | Add an adapter model with adapter name and path; \<adapter\_name>:\<adapter\_ckpt\_path>. The path can be a name from a Hugging Face model hub.                                                                                                                                                                                                                                                                                                                                                                             | -                   | ❌        |

### Model Specific Options

#### T5

| Options               | Type | Summary                | Default | Required |
| --------------------- | ---- | ---------------------- | ------- | -------- |
| `--max-input-length`  | INT  | Maximum input length.  | -       | ✅        |
| `--max-output-length` | INT  | Maximum output length. | -       | ✅        |


# CUDA Compatibility
Source: https://friendli.ai/docs/archive/container/guides/cuda-compatibility

The Friendli Engine supports CUDA-enabled NVIDIA GPUs, which means it relies on a specific version of CUDA and necessitates proper CUDA compute compatibilities.

The Friendli Engine supports CUDA-enabled NVIDIA GPUs, which means it relies on a specific version of CUDA and necessitates proper [CUDA compute compatibilities](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability).
To use the Friendli Container effectively, ensure that you have the appropriate NVIDIA GPUs and an NVIDIA driver in place.

Currently, we publicly offer a single Friendli Container image (`registry.friendli.ai/trial:latest`) equipped with CUDA 12.4, targeting CUDA compute compatibility versions `8.0`, `8.6`, `8.9`, and `9.0`.
To make the right choices regarding GPUs and driver versions, consult the [required driver versions](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id4) and [GPUs](https://developer.nvidia.com/cuda-gpus) for the CUDA toolkit and compute compatibility.


# Deploy Friendli Container as Amazon EKS Add-On
Source: https://friendli.ai/docs/archive/container/guides/eks-quickstart

Deploy Friendli Container on Amazon EKS using the official AWS EKS Add-On. Set up GPU nodes, install the add-on, and run model inference.

## Introduction

This guide walks you through deploying Friendli Container as an Amazon EKS Add-on to enable real-time inference on your Kubernetes cluster. By using Friendli Container in your EKS environment, you benefit from the Friendli Engine's speed and resource efficiency. You'll learn how to configure GPU nodes, install the add-on, and create inference deployments using Kubernetes manifests.

<Note>
  Walking through this tutorial is easier with `eksctl` and AWS CLI tools. Visit the [`eksctl` documentation](https://docs.aws.amazon.com/en_us/eks/latest/userguide/getting-started-eksctl.html) and [AWS CLI homepage](https://aws.amazon.com/cli/) for the installation guides.
</Note>

## General Workflow

1. **Add GPU Node Group**: Create a GPU-enabled node group in your EKS cluster with instances like g6.xlarge or g5.2xlarge.
2. **Configure Friendli Container EKS add-on**: Subscribe to the Friendli Container add-on from the AWS Marketplace and configure IRSA for license validation.
3. **Create Friendli Deployment**: Deploy your model using Friendli Deployment custom resource.
4. **Run Inference**: Send inference requests to your deployed model.

## Prerequisites

* **AWS account** with permissions for EKS, IAM, EC2 operations
* `eksctl` and **AWS CLI** tools installed and configured
* `kubectl` configured to access your EKS cluster
* *(Optional)* **Hugging Face token** if deploying gated/private models. [Hugging Face token docs](https://huggingface.co/docs/hub/security-tokens)

## 1. Add GPU Node Group to Your EKS Cluster

You need an active Amazon EKS cluster. To create a cluster, consult [the Amazon EKS documentation on creating an EKS cluster](https://docs.aws.amazon.com/en_us/eks/latest/userguide/create-cluster-auto.html).

<Note>
  Friendli Container EKS add-on requires Kubernetes version 1.29 or later.
</Note>

When selecting the AWS region for your new EKS cluster, availability of GPU instances is one of the key factors to consider. You can check [instance availability by region](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html) in the AWS documentation.

| Supported NVIDIA Device | AWS EC2 Instance Type                                           |
| ----------------------- | --------------------------------------------------------------- |
| B200                    | [P6 instances](https://aws.amazon.com/ec2/instance-types/p6/)   |
| H200                    | [P5 instances](https://aws.amazon.com/ec2/instance-types/p5/)   |
| H100                    | [P5 instances](https://aws.amazon.com/ec2/instance-types/p5/)   |
| A100                    | [P4 instances](https://aws.amazon.com/ec2/instance-types/p4/)   |
| L40S                    | [G6e instances](https://aws.amazon.com/ec2/instance-types/g6e/) |
| A10G                    | [G5 instances](https://aws.amazon.com/ec2/instance-types/g5/)   |
| L4                      | [G6 instances](https://aws.amazon.com/ec2/instance-types/g6/)   |

If you're going to use multi-GPU VM instance types, we highly recommend installing the NVIDIA GPU Operator for proper resource management. You can consult [the guide from NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/amazon-eks.html). You can find an example of installing a GPU operator using Helm in [this AWS blog post](https://aws.amazon.com/blogs/containers/maximizing-gpu-utilization-with-nvidias-multi-instance-gpu-mig-on-amazon-eks-running-more-pods-per-gpu-for-enhanced-performance/).

The tutorial assumes the following EKS Add-ons are installed in your cluster. You can click the **Get more add-ons** button in the **AWS add-ons** section to install them.

* Amazon VPC CNI
* CoreDNS
* kube-proxy
* Amazon EKS Pod Identity Agent

Next, add a GPU Node Group to your EKS cluster.

* Open [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters) and select the cluster that you want to create a node group in.
* Select the **Compute** tab and click **Add node group**.
* Configure the new node group by entering the name, Node IAM role, and other information. Click **Create recommended role** to create an IAM role. Click **Next**.
* On the next page, select **Amazon Linux 2023 (x86\_64) Nvidia** for AMI type.
* Select the appropriate instance type for the GPU device of your choice.
  * Suggested instance type for this tutorial is `g6.2xlarge`.
* Configure the disk size. It should be large enough to download the model you want to deploy.
  * Suggested disk size for this tutorial is 100GB.
* Configure the desired node group size.
* Go through the rest of the steps, review the changes and click **Create**.

## 2. Configure Friendli Container EKS Add-On

* Open [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters) and select the cluster that you want to configure.
* Select the **Add-ons** tab and click **Get more add-ons**.
* Scroll down and under the **AWS Marketplace add-ons** section, search and check **Friendli Container**, and click **Next**.
* Click **Next**, review your settings, and click **Create**.

<Note>
  - For the details of the pricing, check [Friendli Container on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-ubylhkhrotpli).
  - For trials, custom offers, and inquiries, visit the [contact page](https://friendli.ai/contact).
</Note>

Now you need to allow the Kubernetes ServiceAccount to contact AWS Marketplace for license validation. Execute the following commands, replacing `<REGION>` with the AWS region where you created the cluster and `<CLUSTER>` with the EKS cluster name.

```sh theme={null}
eksctl utils associate-iam-oidc-provider --region <REGION> --cluster <CLUSTER> --approve

eksctl create iamserviceaccount --region <REGION> --cluster <CLUSTER> \
  --namespace default --name default \
  --role-name AWSMarketplaceMeteringAccessForFriendliContainer \
  --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess \
  --approve --override-existing-serviceaccounts
```

The commands above configure IAM roles for service accounts (IRSA) for the Kubernetes ServiceAccount `default` in the `default` namespace to exercise AWSMarketplaceMeteringFullAccess policy on your behalf. To learn more, see the [IRSA documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).

## 3. Create Friendli Deployment

<Note>
  You need to be able to use the `kubectl` CLI tool to access your EKS cluster. Consult [this guide from AWS](https://docs.aws.amazon.com/en_us/eks/latest/userguide/create-kubeconfig.html) for more details.
</Note>

<Note>
  To deploy a private or gated model in the Hugging Face model hub, you need to [create a Hugging Face access token](https://huggingface.co/settings/tokens) with "read" permission.

  Then create a Kubernetes secret.

  `kubectl create secret generic hf-secret --from-literal token=YOUR_TOKEN_HERE`
</Note>

Friendli Deployment is a Kubernetes custom resource that lets you easily create Friendli Inference Deployments without configuring Kubernetes low-level resources like pods, services, and deployments.

Below is a sample FriendliDeployment to deploy [Meta Llama 3.1 8B](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) on one g6.2xlarge instance.

```yaml theme={null}
apiVersion: friendli.ai/v1alpha1
kind: FriendliDeployment
metadata:
  namespace: default
  name: friendlideployment-sample
spec:
  model:
    huggingFace:
      repository: meta-llama/Llama-3.1-8B-Instruct

      # "token:" section is not needed if the model is
      # a public one.
      token:
        name: hf-secret
        key: token

  resources:

    nodeSelector:
      # Use the name of the node group you want to use.
      eks.amazonaws.com/nodegroup: <NODE GROUP NAME>

    numGPUs: 1
    requests:
      cpu: "6"
      ephemeral-storage: 30Gi
      memory: 25Gi
    limits:
      cpu: "6"
      ephemeral-storage: 30Gi
      memory: 25Gi
  deploymentStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
  service:
    inferencePort: 6000
```

You can modify this YAML file for your use case.

* The "token:" section under spec.model.huggingFace refers to the Kubernetes secret you created for storing the Hugging Face access token. If accessing your model does not require an access token, you can omit the "token:" section entirely.
* In the example above, the node selector is `eks.amazonaws.com/nodegroup: <NODE GROUP NAME>`. Replace the node selector key to match the name of your node group.
* CPU and memory resource requirements are adjusted to the g6.2xlarge instance and you may need to edit those values if you used a different instance type.

<Note>
  If your cluster has NVIDIA GPU Operator installed, put the `nvidia.com/gpu` resource in the `requests:` and `limits:` sections. GPU nodes advertise `nvidia.com/gpu` alongside ordinary resources like `cpu` and `memory`. You can omit `numGPUs` from your FriendliDeployment.

  Below is the equivalent example as above for the GPU Operator-enabled cluster.
</Note>

```yaml theme={null}
  resources:
    nodeSelector:
      # Use the name of the node group you want to use.
      eks.amazonaws.com/nodegroup: <NODE GROUP NAME>
    requests:
      cpu: "6"
      ephemeral-storage: 30Gi
      memory: 25Gi
      nvidia.com/gpu: "1"
    limits:
      cpu: "6"
      ephemeral-storage: 30Gi
      memory: 25Gi
      nvidia.com/gpu: "1"
```

Save your YAML file as "friendlideployment.yaml", and execute `kubectl apply -f friendlideployment.yaml`.

```sh theme={null}
$ kubectl apply -f friendlideployment.yaml
friendlideployment.friendli.ai/friendlideployment-sample created

$ kubectl get pods -n default
NAME                                         READY   STATUS    RESTARTS   AGE
friendlideployment-sample-7d7b877c77-zjgqq   2/2     Running   0          3m18s

$ kubectl get services -n default
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
friendlideployment-sample   ClusterIP   172.20.95.224   <none>        6000/TCP   18m
kubernetes                  ClusterIP   172.20.0.1      <none>        443/TCP    28h
```

Now you can [port-forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to the service to connect to the service from your PC.

```sh theme={null}
$ kubectl port-forward -n default svc/friendlideployment-sample 6000
Forwarding from 127.0.0.1:6000 -> 6000
Forwarding from [::1]:6000 -> 6000
```

In another terminal, use the curl tool to send an inference request.

```sh theme={null}
$ curl http://localhost:6000/v1/completions -H 'Content-Type: application/json' --data-raw '{"prompt": "Hi!", "max_tokens": 10, "stream": false}'
{"choices":[{"finish_reason":"length","index":0,"seed":15349211611234757311,"text":" I'm Alex, and I'm excited to share","tokens":[358,2846,8683,11,323,358,2846,12304,311,4430]}],"id":"cmpl-b2e4b4cba711448c847ab89d763588da","object":"text_completion","usage":{"completion_tokens":10,"prompt_tokens":3,"total_tokens":13}}
```

For more information about Friendli Container usage, check [our documentation](/docs/archive/container/guides/introduction) and [contact us](http://friendli.ai/contact) for inquiries.

## Cleaning Up

You can remove the FriendliDeployment using the `kubectl` CLI tool.

```sh theme={null}
$ kubectl delete friendlideployment -n default friendlideployment-sample
friendlideployment.friendli.ai "friendlideployment-sample" deleted
```

You may also want to scale down or delete your GPU node group to avoid being charged for unused GPU instances.

You can now deploy your models with Friendli Container as an EKS add-on. Use it for real-time inference on your Kubernetes cluster.


# Inference with gRPC
Source: https://friendli.ai/docs/archive/container/guides/inference-with-grpc

Run a gRPC inference server with Friendli Container and send requests using the Friendli Python SDK. Includes setup, configuration, and code examples.

This guide walks you through running a gRPC inference server with Friendli Container and interacting with it through the `friendli` SDK.

## Prerequisites

Install `friendli` to use gRPC client SDK:

```sh theme={null}
pip install friendli
```

<Note>
  Ensure you have the `friendli` SDK version `1.4.1` or higher installed.
</Note>

## Starting Friendli Container with gRPC

Running the Friendli Container with a gRPC server for completions is available by adding the `--grpc true` option to the command argument.
This supports response-streaming gRPC, and you can send requests using our `friendli` SDK.
To start the Friendli Container with gRPC support, use the following command:

```sh theme={null}
export FRIENDLI_CONTAINER_SECRET="YOUR_FRIENDLI_CONTAINER_SECRET_flc_XXX"

# e.g. Running `NousResearch/Hermes-3-Llama-3.1-8B` on GPU 0 with a trial image.
docker run --gpus '"device=0"' -p 8000:8000 \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  registry.friendli.ai/trial:latest  \
  --hf-model-name NousResearch/Hermes-3-Llama-3.1-8B \
  --grpc true
```

<Info>
  You can change the port of the server with the `--web-server-port` argument.
</Info>

## Sending Requests with the Client SDK

Here is how to use the `friendli` SDK to interact with the gRPC server.
This example assumes that the gRPC server is running on `0.0.0.0:8000`.

<CodeGroup>
  ```python Default theme={null}
  from friendli import SyncFriendli

  client = SyncFriendli()

  stream = client.container.chat.complete(
      messages=[
          {"content": "You are a helpful assistant.", "role": "system"},
          {"content": "Hello!", "role": "user"},
      ],
      stream=True,  # Should be True
      top_k=1,
  )

  for chunk in stream:
      print(chunk.text, end="", flush=True)
  ```

  ```python Async theme={null}
  # For asynchronous operations, use the following code snippet:

  import asyncio
  from friendli import AsyncFriendli

  client = AsyncFriendli()

  async def run():
      stream = await client.container.chat.complete(
          messages=[
              {"content": "You are a helpful assistant.", "role": "system"},
              {"content": "Hello!", "role": "user"},
          ],
          stream=True,  # Should be True
          top_k=1,
      )

      async for chunk in stream:
          print(chunk.text, end="", flush=True)

  asyncio.run(run())
  ```
</CodeGroup>

## Properly Closing the Client

By default, the library closes underlying HTTP and gRPC connections when the `client` is garbage-collected.
You can manually close the `Friendli` or `AsyncFriendli` client using the `.close()` method or use a context manager to ensure proper closure when exiting a `with` block.

<CodeGroup>
  ```python Default theme={null}
  from friendli import SyncFriendli

  client = SyncFriendli()

  with client:
      stream = client.container.chat.complete(
          messages=[
              {"content": "You are a helpful assistant.", "role": "system"},
              {"content": "Hello!", "role": "user"},
          ],
          stream=True,  # Should be True
          top_k=1,
          min_tokens=10,
      )

      for chunk in stream:
          print(chunk.text, end="", flush=True)
  ```

  ```python Async theme={null}
  import asyncio
  from friendli import AsyncFriendli

  client = AsyncFriendli()

  async def run():
      async with client:
          stream = await client.container.chat.complete(
              messages=[
                  {"content": "You are a helpful assistant.", "role": "system"},
                  {"content": "Hello!", "role": "user"},
              ],
              stream=True,  # Should be True
              top_k=1,
          )

          async for chunk in stream:
              print(chunk.text, end="", flush=True)

  asyncio.run(run())
  ```
</CodeGroup>


# Introducing Friendli Container
Source: https://friendli.ai/docs/archive/container/guides/introduction

Deploy generative AI models on your own infrastructure with Friendli Container. Full control over GPU resources, networking, and scaling.

While Friendli Model APIs and Dedicated Endpoints offer convenient cloud-based solutions, you may want even more control and flexibility. Friendli Container is the answer.

## What Is Friendli Container

Friendli Container packages the Friendli Engine, our cutting-edge serving technology, as a Docker container you run on your own infrastructure. With it, you can:

* **Run on your own infrastructure**: Deploy on your existing GPU machines or your preferred cloud provider, keeping data within your own environment.
* **Keep full control**: Customize the container configuration to match your workflows, and manage your own GPU resources for potential cost savings.
* **Serve securely and privately**: Run models entirely in your environment—ideal for sensitive data and compliance requirements.

Friendli Container is a good fit if you handle sensitive data, want full control over your serving environment, or already own a GPU cluster.

## Next Steps

<CardGroup>
  <Card title="QuickStart" icon="rocket" href="/docs/archive/container/guides/quickstart">
    Run your first container, from trial access to your first inference request.
  </Card>

  <Card title="Configuration" icon="gear" href="/docs/archive/container/guides/configuration">
    Configure launch options, multi-GPU serving, and more in detail.
  </Card>

  <Card title="Browse Models" icon="layer-group" href="https://friendli.ai/models?products=CONTAINER">
    Explore models you can serve with Friendli Container.
  </Card>
</CardGroup>


# Observability for Friendli Container
Source: https://friendli.ai/docs/archive/container/guides/monitoring

Observability is an integral part of DevOps. To support this, Friendli Container exports internal metrics in a Prometheus text format.

Observability is an integral part of DevOps. To support this, Friendli Container exports internal metrics in a [Prometheus](https://prometheus.io) text format.

By default, metrics are served at `http://localhost:8281/metrics`. You can configure the port number using the command line option `--metrics-port`.

## Supported Metrics

### Counters

Counters are cumulative metrics whose values monotonically increase.
They are often used in combination with Prometheus function [rate()](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) for calculating the throughput.

| Metric Name                       | Description                                              |
| --------------------------------- | -------------------------------------------------------- |
| friendli\_requests\_total         | Cumulative number of requests received                   |
| friendli\_responses\_total        | Cumulative number of responses sent                      |
| friendli\_items\_total            | Cumulative number of items requested                     |
| friendli\_failure\_by\_cancel     | Cumulative number of failed requests due to cancellation |
| friendli\_failure\_by\_timeout    | Cumulative number of failed requests due to timeout      |
| friendli\_failure\_by\_nan\_error | Cumulative number of failed requests due to NaN error    |
| friendli\_failure\_by\_reject     | Cumulative number of failed requests due to rejection    |

<Note>
  One inference request may generate multiple results with the `n` field in the request body.
  Upon receiving such a request, `friendli_requests_total` is increased by 1 and `friendli_items_total` is increased by `n`.
</Note>

### Gauges

Gauges are numerical values that can go up and down to represent the current value.

| Metric Name                        | Description                                                           |
| ---------------------------------- | --------------------------------------------------------------------- |
| friendli\_current\_requests        | Current number of requests in the engine (either assigned or waiting) |
| friendli\_current\_items           | Current number of items in the engine (either assigned or waiting)    |
| friendli\_current\_assigned\_items | Current number of items actively processed by the engine              |
| friendli\_current\_waiting\_items  | Current number of items waiting in the internal queue                 |

### Histograms

[Histograms](https://prometheus.io/docs/practices/histograms) are used to track the distribution of variables over time.

<table>
  <thead>
    <tr>
      <th>Histogram</th>
      <th>Metric Name</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><a href="https://friendli.ai/blog/friendli-tcache/">Friendli TCache</a> hit ratio (0≤value≤1)</td>
      <td>friendli\_tcache\_hit\_ratio\_bucket</td>
      <td>Bucketized number of histogram samples for TCache hit ratio, with <code>le</code> label</td>
    </tr>

    <tr>
      <td>friendli\_tcache\_hit\_ratio\_count</td>
      <td>Total number of histogram samples for TCache hit ratio</td>
    </tr>

    <tr>
      <td>friendli\_tcache\_hit\_ratio\_sum</td>
      <td>Sum of histogram sample values for TCache hit ratio</td>
    </tr>

    <tr>
      <td>The length of input tokens (Experimental metric)</td>
      <td>friendli\_input\_lengths\_bucket</td>
      <td>Bucketized number of histogram samples for length of input tokens, with <code>le</code> label</td>
    </tr>

    <tr>
      <td>friendli\_input\_lengths\_count</td>
      <td>Total number of histogram samples for length of input tokens</td>
    </tr>

    <tr>
      <td>friendli\_input\_lengths\_sum</td>
      <td>Sum of histogram sample values for length of input tokens</td>
    </tr>

    <tr>
      <td>The length of output tokens (Experimental metric)</td>
      <td>friendli\_output\_lengths\_bucket</td>
      <td>Bucketized number of histogram samples for length of output tokens, with <code>le</code> label</td>
    </tr>

    <tr>
      <td>friendli\_output\_lengths\_count</td>
      <td>Total number of histogram samples for length of output tokens</td>
    </tr>

    <tr>
      <td>friendli\_output\_lengths\_sum</td>
      <td>Sum of histogram sample values for length of output tokens</td>
    </tr>
  </tbody>
</table>

<Note>
  For visualizing histograms using Grafana, [How to visualize Prometheus histograms in Grafana](https://grafana.com/blog/2020/06/23/how-to-visualize-prometheus-histograms-in-grafana) provides useful tips.
</Note>

### Quantiles

Quantiles are used to show the current p50 (median), p90, and p99 percentiles of variables.

<table>
  <thead>
    <tr>
      <th>Quantiles</th>
      <th>Metric Name</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Request completion latency (in nanoseconds)</td>
      <td>friendli\_requests\_latencies</td>
      <td>Percentile value for request completion latency (<code>quantile</code> label is either <code>0.5</code>, <code>0.9</code>, or <code>0.99</code>)</td>
    </tr>

    <tr>
      <td>friendli\_requests\_latencies\_count</td>
      <td>Total number of samples for request completion latency</td>
    </tr>

    <tr>
      <td>friendli\_requests\_latencies\_sum</td>
      <td>Sum of sample values for request completion latency</td>
    </tr>

    <tr>
      <td>Time to first token (TTFT) (in nanoseconds)</td>
      <td>friendli\_requests\_ttft</td>
      <td>Percentile value for time to first token (TTFT) (<code>quantile</code> label is either <code>0.5</code>, <code>0.9</code>, or <code>0.99</code>)</td>
    </tr>

    <tr>
      <td>friendli\_requests\_ttft\_count</td>
      <td>Total number of samples for time to first token (TTFT)</td>
    </tr>

    <tr>
      <td>friendli\_requests\_ttft\_sum</td>
      <td>Sum of sample values for time to first token (TTFT)</td>
    </tr>

    <tr>
      <td>Request queueing delay (in nanoseconds)</td>
      <td>friendli\_requests\_queueing\_delays</td>
      <td>Percentile value for queueing delay (<code>quantile</code> label is either <code>0.5</code>, <code>0.9</code>, or <code>0.99</code>)</td>
    </tr>

    <tr>
      <td>friendli\_requests\_queueing\_delays\_count</td>
      <td>Total number of samples for queueing delay</td>
    </tr>

    <tr>
      <td>friendli\_requests\_queueing\_delays\_sum</td>
      <td>Sum of sample values for queueing delay</td>
    </tr>
  </tbody>
</table>

### Info

The following information metric always has a value of 1. The metric labels contain useful information in text.

| Metric Name               | Label     | Description    |
| ------------------------- | --------- | -------------- |
| friendli\_engine\_version | `version` | Engine version |

## Grafana Dashboard Template

<RoundedBorderBox>
  <img alt="Grafana Dashboard" />
</RoundedBorderBox>

You can import [the dashboard templates](https://github.com/friendliai/container-resource/tree/main/grafana) to your Grafana instance.
The Grafana instance must be connected to a Prometheus instance (or a Prometheus-compatible data source) that scrapes metrics from Friendli Container processes.

<Note>
  The dashboard template works with Grafana v8.0.0 or later versions. We recommend using Grafana v10.0.0 or later for the best experience.
</Note>


# Optimize with Policy Search
Source: https://friendli.ai/docs/archive/container/guides/optimizing-inference-with-policy-search

Boost inference throughput by up to 2x for MoE and quantized models by running execution policy search in Friendli Container for production.

## Introduction

For specialized cases, like **serving MoE models (e.g., Mixtral)** or **quantized models**, you can further optimize inference performance through an execution policy search.
You can skip this process, but it is necessary to get the optimized speed of Friendli Engine.
When the Friendli Engine runs with the optimal policy, the performance can increase by 1.5x to 2x (i.e., throughput and latency).
Therefore, we recommend skipping policy search for simple model testing, and performing policy search for cost analysis or latency analysis in a production service.

<Note>
  Policy search is effective only when serving (1) MoE models or (2) AWQ, FP8, or INT8 quantized models. Otherwise, it is useless.
</Note>

## Running Policy Search

You can run policy search by adding the following options to the launch command of Friendli Container.

| Options                    | Type    | Summary                                                                                                                                                             | Default             |
| -------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- |
| `--algo-policy-dir`        | TEXT    | Path to the directory to save the searched optimal policy file. The default value is the current working directory.                                                 | current working dir |
| `--search-policy`          | BOOLEAN | Runs policy search to find the best Friendli execution policy for the given configuration such as model type, GPU, NVIDIA driver version, quantization scheme, etc. | false               |
| `--terminate-after-search` | BOOLEAN | Terminates engine container after policy search.                                                                                                                    | false               |

### Example: `FriendliAI/Llama-3.1-8B-Instruct-fp8`

For example, you can start the policy search for [FriendliAI/Llama-3.1-8B-Instruct-fp8](https://huggingface.co/FriendliAI/Llama-3.1-8B-Instruct-fp8) model as follows:

```sh theme={null}
export HF_MODEL_NAME="FriendliAI/Llama-3.1-8B-Instruct-fp8"
export FRIENDLI_CONTAINER_SECRET="YOUR CONTAINER SECRET"
export FRIENDLI_CONTAINER_IMAGE="registry.friendli.ai/trial"
export GPU_ENUMERATION='"device=0"'
export POLICY_DIR=$PWD/policy

mkdir -p $POLICY_DIR

docker run \
  --gpus $GPU_ENUMERATION \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $POLICY_DIR:/policy \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  $FRIENDLI_CONTAINER_IMAGE \
    --hf-model-name $HF_MODEL_NAME \
    --algo-policy-dir /policy \
    --search-policy true
```

### Example: `mistralai/Mixtral-8x7B-Instruct-v0.1` (TP=4)

```sh theme={null}
export HF_MODEL_NAME="mistralai/Mixtral-8x7B-Instruct-v0.1"
export FRIENDLI_CONTAINER_SECRET="YOUR CONTAINER SECRET"
export FRIENDLI_CONTAINER_IMAGE="registry.friendli.ai/trial"
export GPU_ENUMERATION='"device=0,1,2,3"'
export POLICY_DIR=$PWD/policy

mkdir -p $POLICY_DIR

docker run -p 8000:8000 \
  --ipc=host --gpus $GPU_ENUMERATION \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $POLICY_DIR:/policy \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  $FRIENDLI_CONTAINER_IMAGE \
    --hf-model-name $HF_MODEL_NAME \
    --num-devices 4 \
    --algo-policy-dir /policy \
    --search-policy true
```

Once the policy search is complete, a policy file will be created in `$POLICY_DIR`.
If the policy file already exists, the engine will search only the necessary spaces and update the policy file accordingly.
After the policy search, the engine starts to serve the endpoint using the policy file.

<Note>
  It takes up to several minutes to find the optimal policy for Llama 2 13B model with NVIDIA A100 80GB GPU.
  The estimated time and remaining time will be displayed in the stderr when you run the policy search.
</Note>

## Running Policy Search Without Starting Serving Endpoint

To search for the best policy without starting the serving endpoint, launch the engine with the Friendli Container command and include the `--terminate-after-search true` option.

### Example: `FriendliAI/Llama-3.1-8B-Instruct-fp8`

```sh theme={null}
docker run \
  --gpus $GPU_ENUMERATION \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $POLICY_DIR:/policy \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  $FRIENDLI_CONTAINER_IMAGE \
    --hf-model-name FriendliAI/Llama-3.1-8B-Instruct-fp8 \
    --algo-policy-dir /policy \
    --search-policy true \
    --terminate-after-search true
```

### Example: `mistralai/Mixtral-8x7B-Instruct-v0.1` (TP=4)

```sh theme={null}
docker run -p 8000:8000 \
  --ipc=host --gpus $GPU_ENUMERATION \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $POLICY_DIR:/policy \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  $FRIENDLI_CONTAINER_IMAGE \
    --hf-model-name mistralai/Mixtral-8x7B-Instruct-v0.1 \
    --num-devices 4 \
    --algo-policy-dir /policy \
    --search-policy true \
    --terminate-after-search true
```

## FAQ: When to Run Policy Search Again

The execution policy depends on the following factors:

* Model
* GPU
* GPU count and parallelism degree (The value for `--num-devices` option)
* NVIDIA Driver major version
* Friendli Container version

You should run policy search again when any of these are changed from your serving setup.


# Quantization
Source: https://friendli.ai/docs/archive/container/guides/quantization

Learn how to serve pre-quantized models or perform online quantization with Friendli Container to reduce memory and speed up inference.

## What Is Quantization

**Quantization** is a technique that reduces the precision of a generative AI model's parameters, optimizing memory usage and inference speed while maintaining response quality.

### Friendli Container Supports

* **Online quantization**: Quantize your model *<u>on the fly at serving time</u>*. You don't need to prepare pre-quantized weights in advance. Launch the model with the `--quantization` option, and the system dynamically quantizes it as the container starts.
* **Serving a pre-quantized model**: Serve a model that has been *<u>already quantized beforehand</u>*. In this mode, you use model weights that were already quantized and load them during serving.

## Serving a Model with Online Quantization

If you want to serve your own model but need to quantize it or adjust its precision, Friendli Container offers **online quantization**. You don't need to prepare a quantized model in advance.

Once your model is ready, you can serve it with online quantization by adding the `--quantization` argument when [running Friendli Container](/docs/archive/container/guides/configuration).

* `--quantization` `(8bit|4bit|16bit)`: Applies online quantization with the specified precision. It automatically detects your hardware and selects a suitable quantization scheme.

<Note>
  - Use `--quantization 8bit` for **NVIDIA Ada, Hopper, and Blackwell** GPUs.
  - Use `--quantization 4bit` for **NVIDIA Hopper and Blackwell** GPUs.
</Note>

<Tip>
  To dequantize a model to 16-bit precision, use `--quantization 16bit`.
</Tip>

### Example: `deepseek-ai/DeepSeek-R1` with 4-Bit Online Quantization on NVIDIA H200 GPUs

```sh theme={null}
# GPU Info: NVIDIA H200 * 4
# Fill the values of following variables.
export FRIENDLI_CONTAINER_SECRET=""  # Friendli container secret
export FRIENDLI_CONTAINER_IMAGE=""  # Friendli container image (e.g., "registry.friendli.ai/trial")
export GPU_ENUMERATION=""  # GPUs (e.g., '"device=0,1,2,3"')

docker run \
  --gpus $GPU_ENUMERATION \
  -p 8000:8000 \
  -v $HF_HOME:/root/.cache/huggingface \
  -v $POLICY_DIR:/policy \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  $FRIENDLI_CONTAINER_IMAGE \
    --hf-model-name deepseek-ai/DeepSeek-R1 \
    --quantization 4bit \
    --algo-policy-dir /policy \
    --search-policy true
```

<Tip>
  To serve online quantized models efficiently, you must run a policy search to explore the optimal execution policy. Learn how to run the policy search at [Running Policy Search](/docs/archive/container/guides/optimizing-inference-with-policy-search#running-policy-search).
</Tip>

## Serving a Pre-Quantized Model

If you have already quantized and uploaded a model to the Hugging Face Hub, Friendli Container supports the model with the following options:

* **Quantized model with well-known quantizations:**
  * [MXFP4](https://huggingface.co/docs/transformers/en/quantization/mxfp4)
  * [**Fine-grained FP8**](https://huggingface.co/docs/transformers/quantization/finegrained_fp8) (including DeepSeek-V3 style FP8 Quantization)
  * a subset of models created by:
    * [AWQ](https://huggingface.co/docs/transformers/en/quantization/awq)
    * [**AutoFP8**](https://github.com/neuralmagic/AutoFP8)
    * [**compressed-tensors**](https://huggingface.co/docs/transformers/en/quantization/compressed_tensors)
* [**Quantized model checkpoints by FriendliAI**](https://huggingface.co/FriendliAI)

### Example: `openai/gpt-oss-120b` on NVIDIA B200 GPU

```sh theme={null}
# GPU Info: NVIDIA B200
# Fill the values of following variables.
export FRIENDLI_CONTAINER_SECRET=""  # Friendli container secret
export FRIENDLI_CONTAINER_IMAGE=""  # Friendli container image (e.g., "registry.friendli.ai/trial")
export GPU_ENUMERATION=""  # GPUs (e.g., '"device=0"')

docker run \
  --gpus $GPU_ENUMERATION \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $POLICY_DIR:/policy \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  $FRIENDLI_CONTAINER_IMAGE \
    --hf-model-name openai/gpt-oss-120b \
    --algo-policy-dir /policy \
    --search-policy true
```

<Tip>
  To serve pre-quantized models efficiently, you must run a policy search to explore the optimal execution policy. Learn how to run the policy search at [Running Policy Search](/docs/archive/container/guides/optimizing-inference-with-policy-search#running-policy-search).
</Tip>


# QuickStart: Friendli Container Trial
Source: https://friendli.ai/docs/archive/container/guides/quickstart

Get started with Friendli Container trial. Access the registry, configure your secret, launch the container, and monitor with Grafana.

Get started with [Friendli Container](/docs/archive/container/guides/introduction). This quickstart walks you through running your first container—from trial access to your first inference request—and serving an LLM in a secure, private environment.

For detailed launch options, multi-GPU serving, and the full option reference, see [Configuration](/docs/archive/container/guides/configuration).

## Prerequisites

* **Hardware Requirements**: Friendli Container targets x86\_64 architecture and supports NVIDIA GPUs. Prepare compatible GPUs and drivers by referring to [our required CUDA compatibility guide](/docs/archive/container/guides/cuda-compatibility).
* **Software Requirements**: Your machine should be able to run containers with the [NVIDIA container toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html). This tutorial uses Docker as the container runtime, along with [Docker Compose](https://docs.docker.com/compose).
* **Model Compatibility**: If your model is in a [safetensors](https://huggingface.co/docs/safetensors/index) format, which is compatible with [Hugging Face transformers](https://huggingface.co/docs/transformers), you can serve the model directly with the Friendli Container. Check our [Model library](https://friendli.ai/models) for the non-exhaustive list of supported models.

This tutorial assumes that your model of choice is uploaded to [Hugging Face](https://huggingface.co) and you have access to it. If the model is gated or private, you need to prepare a [Hugging Face Access Token](https://huggingface.co/settings/tokens).

## Getting Access to Friendli Container

### Activate Your Free Trial

[Contact sales](https://friendli.ai/contact) to activate your free trial.

### Get Access to the Container Registry

You need a Personal API key to sign in to the container registry.

1. Go to [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys) and click **Create API Key**.
2. Save the API key you just created.

### Prepare Your Container Secret

A container secret is a code that activates Friendli Container. You pass the container secret as an environment variable when running the container image.

1. Go to [Friendli Suite > Container > Container Secrets](https://friendli.ai/suite/~/container/secrets) and click **Create secret**.
2. Save the secret you just created.

<Note>
  **🔑 Secret Rotation**

  You can rotate the container secret for security reasons. If you rotate the container secret, a new secret will be created and the previous secret will be automatically revoked in **30** minutes.
</Note>

## Running Friendli Container

### Pull the Friendli Container Image

1. Sign in to the container registry using the email address for your Friendli Suite account and the Personal API key.

```sh theme={null}
export FRIENDLI_EMAIL="YOUR ACCOUNT EMAIL ADDRESS"
export API_KEY="YOUR_API_KEY"
docker login registry.friendli.ai -u $FRIENDLI_EMAIL -p $API_KEY
```

2. Pull the image.

```sh theme={null}
docker pull registry.friendli.ai/trial
```

### Run Friendli Container with a Hugging Face Model

1. Clone our [container resource](https://github.com/friendliai/container-resource) git repository.

```sh theme={null}
git clone https://github.com/friendliai/container-resource
cd container-resource/quickstart/docker-compose
```

2. Set up environment variables.

   ```sh theme={null}
    export HF_MODEL_NAME="<...>"  # Hugging Face model name (e.g., "meta-llama/Meta-Llama-3-8B-Instruct")
    export FRIENDLI_CONTAINER_SECRET="<...>"  # Friendli container secret
   ```

   If your model is a private or gated one, you also need to provide a [Hugging Face Access Token](https://huggingface.co/settings/tokens).

   ```sh theme={null}
    export HF_TOKEN="<...>"  # Hugging Face Access Token
   ```

3. Launch the Friendli Container.

   ```sh theme={null}
   docker compose up -d
   ```

<Note>
  By default, the container will listen for inference requests at TCP port 8000 and a Grafana service will be available at TCP port 3000. You can change the designated ports using the following environment variables. For example, if you want to use TCP port 8001 and port 3001 for Grafana, execute the command below.

  ```sh theme={null}
  export FRIENDLI_PORT="8001"
  export FRIENDLI_GRAFANA_PORT="3001"
  ```
</Note>

<Note>
  Even though the machine has multiple GPUs, the container will make use of only one GPU, specifically the first GPU (`device_ids: ['0']`). You can edit `docker-compose.yaml` to change what GPU device the container will use.
</Note>

<Note>
  The downloaded Hugging Face model will be cached in the `$HOME/.cache/huggingface` directory. You may want to clean up this directory after completing this tutorial.
</Note>

### Send Inference Requests

You can now send inference requests to the running container. For information on all available parameters, refer to the [API reference](/docs/openapi/container/overview).

<CodeGroup>
  ```sh curl theme={null}
  curl -X POST http://0.0.0.0:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "messages": [
        {"role": "user", "content": "What makes a good leader?"}
      ],
      "max_tokens": 30
    }'
  ```

  ```python OpenAI Python SDK theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="http://0.0.0.0:8000/v1"
  )

  completion = client.chat.completions.create(
      model="",
      messages=[
          {"role": "user", "content": "What makes a good leader?"}
      ],
      max_tokens=30,
      stream=True
  )
  for chunk in completion:
      print(chunk.choices[0].delta.content, end="", flush=True)
  ```

  ```python Friendli Python SDK theme={null}
  from friendli import SyncFriendli

  client = SyncFriendli()

  stream = client.container.chat.complete(
      messages=[{"role": "user", "content": "Python is a popular"}],
      max_tokens=30,
      stream=True,
  )
  for chunk in stream:
      print(chunk.text, end="", flush=True)
  ```

  ```sh Completion theme={null}
  curl -X POST http://0.0.0.0:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
      "prompt": "What makes a good leader?",
      "max_tokens": 30
    }'
  ```

  ```sh Tokenization theme={null}
  curl -X POST http://0.0.0.0:8000/v1/tokenize \
    -H "Content-Type: application/json" \
    -d '{
      "prompt": "What is generative AI?"
    }'
  ```

  ```sh Detokenization theme={null}
  curl -X POST http://0.0.0.0:8000/v1/detokenize \
    -H "Content-Type: application/json" \
    -d '{
      "tokens": [
        128000,
        3923,
        374,
        1803,
        1413,
        15592,
        30
      ]
    }'
  ```
</CodeGroup>

<Note>
  Chat completion requests work only if the model's tokenizer config contains a `chat_template`.
</Note>

### Monitor with Grafana

Using your browser, open `http://0.0.0.0:3000/d/friendli-engine`, and sign in with username `admin` and password `admin`. You can now access the dashboards showing useful engine metrics.

<RoundedBorderBox>
  <img alt="Grafana Dashboard" />
</RoundedBorderBox>

<Note>
  If you cannot open a browser directly on the GPU machine, use SSH to forward requests from the browser on your PC to the GPU machine.

  ```sh theme={null}
  # Change these variables to match your environment.
  LOCAL_GRAFANA_PORT=3000  # The number of the port in your PC.
  FRIENDLI_GRAFANA_PORT=3000  # The number of the port in the GPU machine.

  ssh "$GPU_MACHINE_ADDRESS" -L "$LOCAL_GRAFANA_PORT:0.0.0.0:$FRIENDLI_GRAFANA_PORT"
  ```

  You should replace `$GPU_MACHINE_ADDRESS` with the address of the GPU machine. You may also use the `-l login_name` or `-p port` options to connect to the GPU machine using SSH.

  Then using your browser on the PC, open `http://0.0.0.0:$LOCAL_GRAFANA_PORT/d/friendli-engine`.
</Note>

## Going Further

Congratulations! You can now serve your LLM of choice using your hardware, with the power of the most efficient LLM serving engine on the planet. The following topics will help you go further through your AI endeavors.

* **Multi-GPU Serving**: Although this tutorial is limited to using only one GPU, Friendli Container supports tensor parallelism and pipeline parallelism for multi-GPU inference. Check [Multi-GPU Serving](/docs/archive/container/guides/configuration#multi-gpu-serving) for more information.
* **Serving Multi-LoRA Models**: You can deploy multiple customized LLMs without additional GPU resources. See [Serving Multi-LoRA Models](/docs/archive/container/guides/serving-multi-lora-models) to learn how to launch the container with your adapters.
* **Quantization**: Friendli Container supports **online quantization**, which quantizes a model instantly when you launch it. You can also serve a pre-quantized model. Check [Quantization](/docs/archive/container/guides/quantization) for more information.
* **Serving MoE Models**: Running MoE (Mixture of Experts) models requires an additional step of [execution policy search](/docs/archive/container/guides/optimizing-inference-with-policy-search). See [Serving MoE Models](/docs/archive/container/guides/serving-moe-models) to learn how to launch the container with MoE models.

If you are stuck or need help going through this tutorial, ask for support by sending an email to [Support](mailto:support@friendli.ai).


# Running Friendli Container on SageMaker
Source: https://friendli.ai/docs/archive/container/guides/sagemaker-integration

Create a real-time inference endpoint in Amazon SageMaker with Friendli Container. Use Friendli Engine for faster, cost-efficient serving.

## Introduction

This guide walks you through creating a [real-time inference endpoint in Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html) with Friendli Container backend.
By using Friendli Container in your SageMaker pipeline, you benefit from the Friendli Engine's speed and resource efficiency.
You'll learn how to create inference endpoints using both the AWS console and the boto3 Python SDK.

## General Workflow

<img alt="SageMaker Workflow" />

1. **Create a Model**: Within SageMaker Inference, define a new model by specifying the model artifacts in your S3 bucket and the Friendli Container image from ECR.
2. **Configure the Endpoint**: Create a SageMaker Inference endpoint configuration by selecting the instance type and the number of instances required.
3. **Create the Endpoint**: Use the configured settings to launch a SageMaker Inference endpoint.
4. **Invoke the Endpoint**: Once deployed, send requests to your endpoint to receive inference responses.

## Prerequisites

Before beginning, you need to push the Friendli Container image to an ECR repository on AWS.
First, prepare the Friendli Container image by following the instructions in [**Pull the Friendli Container image**](/docs/archive/container/guides/quickstart#pull-the-friendli-container-image).
Then, tag and push the image to the Amazon ECR repository as guided in [**Pushing a Docker image to an Amazon ECR private repository**](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html).

## Using the AWS Console

Follow these step-by-step instructions to create an inference endpoint using the AWS Console.

### Step 1: Creating a Model

You can start creating a model by clicking the **Create model** button under **SageMaker > Inference > Models**.
Then, configure the model with the following fields:

* **Model settings**:
  * **Model name**: A model name.
  * **IAM role**: An IAM role that includes the `AmazonSageMakerFullAccess` policy.
* **Container definition 1**:
  * **Container input option**: Select the "Provide model artifacts and inference image location".
  * **Model Compression Type**:
    * To use a model in the S3 bucket:
      * When the model is compressed, select **CompressedModel**.
      * Otherwise, select **UncompressedModel**.
    * When using a model from the Hugging Face hub, any option would work fine.
  * **Location of inference code image**: Specify the ARN of the ECR repo for the Friendli Container.
  * **Location of model artifacts** (optional):
    * To use a model in the S3 bucket: Specify the S3 URI where your model is stored. Ensure the file structure matches the directory format compatible with the `--hf-model-name` option of the Friendli Container.
    * When using a model from the Hugging Face hub, you can leave this field empty.
* **Environment variables**:
  * Always required:
    * `FRIENDLI_CONTAINER_SECRET`: Your Friendli Container Secret. Refer to [**Prepare your container secret**](/docs/archive/container/guides/quickstart#prepare-your-container-secret) to learn how to get the container secret.
    * `SAGEMAKER_MODE`: This should be set to `True`.
    * `SAGEMAKER_NUM_DEVICES`: Number of devices to use for tensor parallelism degree.
  * Required when using a model in the S3 bucket:
    * `SAGEMAKER_USE_S3`: This should be set to `True`.
  * Required when using a model from the Hugging Face hub:
    * `SAGEMAKER_HF_MODEL_NAME`: The Hugging Face model name (e.g., `mistralai/Mistral-7B-Instruct-v0.2`).
    * For private or gated model repos:
      * `HF_TOKEN`: The Hugging Face secret access token.

### Step 2: Creating an Endpoint Configuration

You can start by clicking the **Create endpoint configuration** button under **SageMaker > Inference > Endpoint configurations**.

* **Endpoint configuration**:
  * **Endpoint configuration name**: The name of this endpoint configuration.
  * **Type of endpoint**: For real-time inference, select **Provisioned**.
* **Variants**:
  * To create a "Production" variant, click **Create production variant**.
  * Select the model that you have created in [**Step 1**](#step-1-creating-a-model).
  * Configure the instance type and count by clicking **Edit** in the Actions column.
  * Create the endpoint configuration by clicking **Create endpoint configuration**.

### Step 3: Creating a SageMaker Inference Endpoint

You can start by clicking the **Create endpoint** button under **SageMaker > Inference > Endpoints**.

* Select **Use an existing endpoint configuration**.
* Select the endpoint configuration created in [**Step 2**](#step-2-creating-an-endpoint-configuration).
* Finish by clicking the **Create endpoint** button.

### Step 4: Invoking the Endpoint

When the endpoint status becomes "In Service", you can invoke the endpoint with the following script, after filling in the endpoint name and the region name:

```python theme={null}
import boto3
import json

endpoint_name = "FILL OUT ENDPOINT NAME"
region_name = "FILL OUT AWS REGION"

sagemaker_runtime = boto3.client("sagemaker-runtime", region_name=region_name)

prompt = "Story title: 3 llamas go for a walk\nSummary: The 3 llamas crossed a bridge and something unexpected happened\n\nOnce upon a time"
payload = {
    "prompt": prompt,
    "max_tokens": 512,
    "temperature": 0.8,
}

response = sagemaker_runtime.invoke_endpoint(
    EndpointName=endpoint_name,
    Body=json.dumps(payload),
    ContentType="application/json",
)

print(response['Body'].read().decode('utf-8'))
```

## Using the boto3 SDK

Next, create a SageMaker endpoint using the boto3 Python SDK.
You can achieve this by using the code snippet below. Be sure to fill in the custom fields, customized for your specific use case:

```python theme={null}
import boto3
from sagemaker import get_execution_role

sm_client = boto3.client(service_name='sagemaker')
runtime_sm_client = boto3.client(service_name='sagemaker-runtime')

account_id = boto3.client('sts').get_caller_identity()['Account']
region = boto3.Session().region_name
role = get_execution_role()

endpoint_name="FILL OUT ENDPOINT NAME"
model_name="FILL OUT MODEL NAME"
container = "FILL OUT ECR IMAGE NAME"  # <ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com/IMAGE

instance_type = "ml.g5.12xlarge" # instance type

container = {
    'Image': container,
    'Environment': {
        "HF_TOKEN": "<HUGGINGFACE ACCESS TOKEN>",
        "FRIENDLI_CONTAINER_SECRET": "<FRIENDLI CONTAINER SECRET>",
        "SAGEMAKER_HF_MODEL_NAME": "<HUGGINGFACE MODEL NAME>",  # e.g., meta-llama/Meta-Llama-3-8B
        "SAGEMAKER_MODE": "True",  # Should be true
        "SAGEMAKER_NUM_DEVICES": "4",  # Number of GPUs in `instance_type`
    }
}

endpoint_config_name = 'FILL OUT ENDPOINT CONFIG NAME'

# Create a model
create_model_response = sm_client.create_model(
    ModelName=model_name,
    ExecutionRoleArn=role,
    Containers=[container],
)

# Create an endpoint configuration
create_endpoint_config_response = sm_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[
        {
            'InstanceType': instance_type,
            'InitialInstanceCount': 1,
            'InitialVariantWeight': 1,
            'ModelName': model_name,
            'VariantName': 'AllTraffic',
        },
    ],
)

endpoint_name = "FILL OUT ENDPOINT NAME"

# Create an endpoint
sm_client.create_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=endpoint_config_name,
)

sm_client.describe_endpoint(EndpointName=endpoint_name)
```

You can invoke this endpoint by following [**Step 4**](#step-4-invoking-the-endpoint).

You can now deploy your models with Friendli Container on SageMaker endpoints. Use them for real-time inference.


# Serving MoE Models
Source: https://friendli.ai/docs/archive/container/guides/serving-moe-models

Serve Mixture of Experts (MoE) models like Mixtral 8x7B with Friendli Container. Covers policy search setup and multi-GPU Docker configuration.

## Introduction

This guide explores the steps to serve Mixture of Experts (MoE) models such as Mixtral 8x7B using Friendli Container.

## Search Optimal Policy and Run Friendli Container

To serve MoE models efficiently, you need to run a policy search to find the optimal execution policy.
Learn how to run the policy search at [Running Policy Search](/docs/archive/container/guides/optimizing-inference-with-policy-search#running-policy-search).
Once the search finds an optimal policy, it compiles the policy into a file that you can use to create serving endpoints.
The engine then serves the endpoint using the optimal policy.


# Serving Multi-LoRA Models
Source: https://friendli.ai/docs/archive/container/guides/serving-multi-lora-models

Serve multiple LoRA-adapted LLMs simultaneously with Friendli Container without additional GPU resources. No retraining needed for task-specific models.

## Introduction

As the demand for highly specialized AI capabilities surges, deploying multiple customized large language models (LLMs) without additional GPU resources represents a significant leap forward.
The Friendli Engine addresses this challenge through Multi-LoRA (Low-Rank Adaptation) serving. This method lets you simultaneously serve multiple LLMs optimized for specific tasks, without extensive retraining.
This advancement opens new avenues for AI efficiency and adaptability, promising to revolutionize the deployment of AI solutions on constrained hardware.
This article provides an overview of efficiently serving Multi-LoRA models with the Friendli Engine.

<img alt="LoRA Serving" />

## Prerequisites

Install `huggingface-cli` in your local environment.

```sh theme={null}
pip install "huggingface_hub[cli]"
```

## Downloading Adapter Checkpoints

Download each adapter model you want to serve to your local storage.

```sh theme={null}
# Hugging Face model name of the adapters
export ADAPTER_MODEL1=""
export ADAPTER_MODEL2=""
export ADAPTER_MODEL3=""
export ADAPTER_DIR=/tmp/adapter

huggingface-cli download $ADAPTER_MODEL1 \
  --include "adapter_model.safetensors" "adapter_config.json" \
  --local-dir $ADAPTER_DIR/model1
huggingface-cli download $ADAPTER_MODEL2 \
  --include "adapter_model.safetensors" "adapter_config.json" \
  --local-dir $ADAPTER_DIR/model2
huggingface-cli download $ADAPTER_MODEL3 \
  --include "adapter_model.safetensors" "adapter_config.json" \
  --local-dir $ADAPTER_DIR/model3
...
```

This will result in a directory structure like:

```text theme={null}
/tmp/adapter/model1
  - adapter_model.safetensors
  - adapter_config.json
/tmp/adapter/model2
  - adapter_model.safetensors
  - adapter_config.json
/tmp/adapter/model3
  - adapter_model.safetensors
  - adapter_config.json
```

<Note>
  If an adapter's Hugging Face repo does not contain an `adapter_model.safetensors` checkpoint file, you have to manually convert `adapter_model.bin` into `adapter_model.safetensors`.
  You can use the [official app](https://huggingface.co/spaces/safetensors/convert) or the [Python script](https://github.com/huggingface/safetensors/tree/main/bindings/python) for conversion.
</Note>

## Launching Friendli Engine in a Container

When you have prepared adapter model checkpoints, now you can serve the Multi-LoRA model with Friendli Container.
In addition to the command for running the base model, you have to add the `--adapter-model` argument.

* `--adapter-model`: Add an adapter model with adapter name and path. The path can be a Hugging Face hub name.

```sh theme={null}
# Fill the values of following variables.
export HF_BASE_MODEL_NAME=""  # Hugging Face base model name (e.g., "meta-llama/Llama-2-7b-chat-hf")
export FRIENDLI_CONTAINER_SECRET=""  # Friendli container secret
export FRIENDLI_CONTAINER_IMAGE=""  # Friendli container image (e.g., "registry.friendli.ai/trial")
export GPU_ENUMERATION=""  # GPUs (e.g., '"device=0,1"')
export ADAPTER_NAME=""  # Specify the adapter's name (a user-defined alias).
export ADAPTER_DIR=/tmp/adapter

docker run \
  --gpus $GPU_ENUMERATION \
  -p 8000:8000 \
  -v $ADAPTER_DIR:/adapter \
  -e FRIENDLI_CONTAINER_SECRET=$FRIENDLI_CONTAINER_SECRET \
  $FRIENDLI_CONTAINER_IMAGE \
    --hf-model-name $HF_BASE_MODEL_NAME \
    --adapter-model $ADAPTER_NAME:/adapter/model1 \
    [LAUNCH_OPTIONS]
```

You can find available options for `[LAUNCH_OPTIONS]` at [Configuration: Launch Options](/docs/archive/container/guides/configuration#launch-options).

<Note>
  If you want to launch with multiple adapters, you can use `--adapter-model` with a comma-separated string.

  (e.g. `--adapter-model "adapter_name_0:/adapter/model1,adapter_name_1:/adapter/model2"`)
</Note>

<Warning>
  If a `tokenizer_config.json` file is in an adapter checkpoint path, the engine uses a different chat template in `tokenizer_config.json`.
</Warning>

### Example: Llama 2 7B Chat + LoRA Adapter

This is an example that runs [`meta-llama/Llama-2-7b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) with [`FinGPT/fingpt-forecaster_dow30_llama2-7b_lora`](https://huggingface.co/FinGPT/fingpt-forecaster_dow30_llama2-7b_lora) adapter model.

```sh theme={null}
export ADAPTER_DIR=/tmp/adapter

huggingface-cli download FinGPT/fingpt-forecaster_dow30_llama2-7b_lora \
  --include "adapter_model.safetensors" "adapter_config.json" \
  --local-dir $ADAPTER_DIR/model1

docker run \
  --gpus '"device=0"' \
  -p 8000:8000 \
  -v $ADAPTER_DIR:/adapter \
  -e FRIENDLI_CONTAINER_SECRET="YOUR CONTAINER SECRET" \
  registry.friendli.ai/trial \
    --hf-model-name meta-llama/Llama-2-7b-chat-hf \
    --adapter-model adapter-model-name:/adapter/model1
```

## Sending a Request to a Specific Adapter

You can generate an inference result from a specific adapter model by specifying `model` in the body of an inference request.
For example, assuming you set the launch option of `--adapter-model` to "\<adapter-model-name>:\<adapter-file-path>", you can send a request to the adapter model as follows.

```sh theme={null}
curl -X POST http://0.0.0.0:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "adapter-model-name",
    "prompt": "Python is a language",
    "max_tokens": 30
  }'
```

## Sending a Request to the Base Model

If you omit the `model` field in your request, the base model will be used for generating an inference response.
You can send a request to the base model as shown below.

```sh theme={null}
curl -X POST http://0.0.0.0:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Python is a language",
    "max_tokens": 30
  }'
```

## Limitations

<Warning>
  We only support models compatible with [`peft`](https://github.com/huggingface/peft).

  Base model checkpoint and adapter model checkpoint should have the same datatype.

  When serving multiple adapters simultaneously, each adapter model should have the same target modules. In Hugging Face, the target modules are listed at `adapter_config.json`.
</Warning>


# Friendli Python SDK
Source: https://friendli.ai/docs/archive/python-sdk/how-to

Install and use the Friendli Python SDK to access Model APIs, Dedicated Endpoints, and Container. Covers chat, completions, streaming, and async usage.

## Introduction

The [Friendli Python SDK](https://github.com/friendliai/friendli-python) provides a powerful and flexible way to interact with FriendliAI services, including Model APIs, Dedicated Endpoints, and Container. You can easily integrate your Python applications with FriendliAI.

## Installation

You can install the SDK with either pip or poetry:

```bash theme={null}
# Using pip
pip install friendli

# Using poetry
poetry add friendli
```

## Authentication

Authenticate using a Personal API key, which you can generate from [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys):

```python theme={null}
import os
from friendli import SyncFriendli

with SyncFriendli(
    token=os.environ["API_KEY"],
) as friendli:
    # Your code here
```

For detailed instructions on generating an API key, see the [Manage Your FriendliAI API Keys](/docs/guides/suite/personal-api-keys) guide.

## Chat Completions

The SDK supports chat completions across all deployment types. Choose the deployment option that best fits your needs.

<CodeGroup>
  ```python Model APIs theme={null}
  import os
  from friendli import SyncFriendli

  with SyncFriendli(
      token=os.environ["API_KEY"],
  ) as friendli:
      res = friendli.serverless.chat.complete(
          messages=[
              {
                  "content": "You are a helpful assistant.",
                  "role": "system",
              },
              {
                  "content": "Hello!",
                  "role": "user",
              },
          ],
          model="zai-org/GLM-5.3",
          max_tokens=200,
      )

      print(res)
  ```

  ```python Dedicated Endpoints theme={null}
  import os
  from friendli import SyncFriendli

  with SyncFriendli(
      token=os.environ["API_KEY"],
  ) as friendli:
      res = friendli.dedicated.chat.complete(
          messages=[
              {
                  "content": "You are a helpful assistant.",
                  "role": "system",
              },
              {
                  "content": "Hello!",
                  "role": "user",
              },
          ],
          model="YOUR_ENDPOINT_ID",
          max_tokens=200,
      )

      print(res)
  ```

  ```python Container Deployment theme={null}
  from friendli import SyncFriendli

  with SyncFriendli() as friendli:
      res = friendli.container.chat.complete(
          messages=[
              {
                  "content": "You are a helpful assistant.",
                  "role": "system",
              },
              {
                  "content": "Hello!",
                  "role": "user",
              },
          ],
          max_tokens=200,
      )

      print(res)
  ```
</CodeGroup>

### Asynchronous Chat Completions

<CodeGroup>
  ```python Model APIs theme={null}
  import asyncio
  import os
  from friendli import AsyncFriendli

  async def main():
      async with AsyncFriendli(
          token=os.environ["API_KEY"],
      ) as friendli:
          res = await friendli.serverless.chat.complete(
              messages=[
                  {
                      "content": "You are a helpful assistant.",
                      "role": "system",
                  },
                  {
                      "content": "Hello!",
                      "role": "user",
                  },
              ],
              model="zai-org/GLM-5.3",
              max_tokens=200,
          )

          print(res)

  asyncio.run(main())
  ```

  ```python Dedicated Endpoints theme={null}
  import asyncio
  import os
  from friendli import AsyncFriendli

  async def main():
      async with AsyncFriendli(
          token=os.environ["API_KEY"],
      ) as friendli:
          res = await friendli.dedicated.chat.complete(
              messages=[
                  {
                      "content": "You are a helpful assistant.",
                      "role": "system",
                  },
                  {
                      "content": "Hello!",
                      "role": "user",
                  },
              ],
              model="YOUR_ENDPOINT_ID",
              max_tokens=200,
          )

          print(res)

  asyncio.run(main())
  ```

  ```python Container Deployment theme={null}
  import asyncio
  from friendli import AsyncFriendli

  async def main():
      async with AsyncFriendli() as friendli:
          res = await friendli.container.chat.complete(
              messages=[
                  {
                      "content": "You are a helpful assistant.",
                      "role": "system",
                  },
                  {
                      "content": "Hello!",
                      "role": "user",
                  },
              ],
              max_tokens=200,
          )

          print(res)

  asyncio.run(main())
  ```
</CodeGroup>

## Advanced Features

### Streaming Responses

The SDK supports streaming responses using server-sent events, which you can consume using a simple `for` loop:

```python theme={null}
import os
from friendli import SyncFriendli

with SyncFriendli(
    token=os.environ["API_KEY"],
) as friendli:
    res = friendli.serverless.chat.stream(
        messages=[
            {
                "content": "You are a helpful assistant.",
                "role": "system",
            },
            {
                "content": "Hello!",
                "role": "user",
            },
        ],
        model="zai-org/GLM-5.3",
        max_tokens=200,
    )

    with res as event_stream:
        for event in event_stream:
            # Process each chunk as it arrives
            print(event, flush=True)
```

### Custom Retry Strategy

You can customize retry behavior for operations that support retries:

```python theme={null}
import os
from friendli import SyncFriendli
from friendli.utils import BackoffStrategy, RetryConfig

with SyncFriendli(
    token=os.environ["API_KEY"],
) as friendli:
    res = friendli.serverless.chat.complete(
        messages=[
            {
                "content": "You are a helpful assistant.",
                "role": "system",
            },
            {
                "content": "Hello!",
                "role": "user",
            },
        ],
        model="zai-org/GLM-5.3",
        max_tokens=200,
        retries=RetryConfig("backoff", BackoffStrategy(1, 50, 1.1, 100), False),
    )

    # Handle response
    print(res)
```

### Error Handling

The SDK provides comprehensive error handling with detailed exception information:

```python theme={null}
import os
from friendli import SyncFriendli, models

with SyncFriendli(
    token=os.environ["API_KEY"],
) as friendli:
    try:
        res = friendli.dedicated.endpoint.create(
            advanced={
                "tokenizer_add_special_tokens": True,
                "tokenizer_skip_special_tokens": False,
            },
            hf_model_repo="<value>",
            instance_option_id="<id>",
            name="<value>",
            project_id="<id>",
        )

        # Handle response
        print(res)

    except models.HTTPValidationError as e:
        # Handle validation errors
        print(f"Validation error: {e.data}")
    except models.SDKError as e:
        # Handle general SDK errors
        print(f"Error {e.status_code}: {e.message}")
```

### Custom Logging

You can pass your own logger to the client class to help troubleshoot and diagnose issues during API interactions. This is especially useful when you encounter unexpected behavior or errors.

```python theme={null}
import logging
import os

from friendli import SyncFriendli

# Configure your custom logger, for example:
logger = logging.getLogger(__name__)
logging.basicConfig(
    format="[%(filename)s:%(lineno)s - %(funcName)s()] %(message)s",
    level=logging.INFO,
    handlers=[logging.StreamHandler()],
)

with SyncFriendli(
    server_url=SERVER_URL,
    token=TOKEN,
    debug_logger=logger,  # Pass your logger here
) as friendli:
    # Your code here
    pass
```

## Further Resources

For complete API documentation, advanced usage examples, and detailed reference information, visit the [Friendli Python SDK GitHub repository](https://github.com/friendliai/friendli-python).


# Latest Updates
Source: https://friendli.ai/docs/changelog

Track the latest FriendliAI product updates: new models, pricing changes, new features, and more.

Stay up to date with the latest from FriendliAI—new models, new features, pricing updates, and more. Use the filters to see updates by product or update type.

<Update label="September 16, 2026">
  ## FriendliLink Added

  FriendliLink is now available for **Model APIs**. You can use the CLI to connect Claude Code, Codex, Cursor, DeepSeek Harness, Hermes Agent, OpenCode, or Pi to FriendliAI with one command.

  To learn more, see [Set Up Your Agent with FriendliLink](/docs/examples/agents/friendlilink).
</Update>

<Update label="September 6, 2026">
  ## LG AI Research Model Deprecated

  The following **Model APIs** model is no longer available:

  * `LGAI-EXAONE/K-EXAONE-2.0-750B-A37B`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="August 28, 2026">
  ## Z.ai Models Added

  The following **Model APIs** models are now available:

  * `zai-org/GLM-5.3`
  * `zai-org/GLM-5.3-Flash`

  To learn more, see [Models > GLM-5.3](https://friendli.ai/models/zai-org/GLM-5.3) and [Models > GLM-5.3-Flash](https://friendli.ai/models/zai-org/GLM-5.3-Flash).
</Update>

<Update label="August 20, 2026">
  ## LG AI Research Model Deprecated

  The following **Model APIs** model is no longer available:

  * `LGAI-EXAONE/K-EXAONE-236B-A23B`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="August 6, 2026">
  ## LG AI Research Model Added

  The following **Model APIs** model is now available:

  * `LGAI-EXAONE/K-EXAONE-2.0-750B-A37B`

  To learn more, see [Models > K-EXAONE-2.0-750B-A37B](https://friendli.ai/models/LGAI-EXAONE/K-EXAONE-2.0-750B-A37B).
</Update>

<Update label="August 5, 2026">
  ## Qwen Model Deprecated

  The following **Model APIs** model is no longer available:

  * `Qwen/Qwen3-235B-A22B-Instruct-2507`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="July 6, 2026">
  ## Tool Assisted API Deprecated

  The **Model APIs** Tool Assisted API is no longer available. Use tool calling to connect models to your own tools instead.

  To learn more, see [Tool Calling](/docs/guides/tool-calling).
</Update>

<Update label="July 3, 2026">
  ## Z.ai Model Deprecated

  The following **Model APIs** model is no longer available:

  * `zai-org/GLM-5`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="June 26, 2026">
  ## Meta Llama Models Deprecated

  The following **Model APIs** models are no longer available:

  * `meta-llama/Llama-3.1-8B-Instruct`
  * `meta-llama/Llama-3.3-70B-Instruct`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="June 16, 2026">
  ## Z.ai Model Added

  The following **Model APIs** model is now available:

  * `zai-org/GLM-5.2`

  To learn more, see [Models > GLM-5.2](https://friendli.ai/models/zai-org/GLM-5.2).
</Update>

<Update label="May 22, 2026">
  ## DeepSeek Model Deprecated

  The following **Model APIs** model is no longer available:

  * `deepseek-ai/DeepSeek-V3.1`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="April 16, 2026">
  ## Qwen Model Deprecated

  The following **Model APIs** model is no longer available:

  * `Qwen/Qwen3-30B-A3B`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="April 15, 2026">
  ## Z.ai Model Deprecated

  The following **Model APIs** model is no longer available:

  * `zai-org/GLM-4.7`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="April 9, 2026">
  ## MiniMax Model Deprecated

  The following **Model APIs** model is no longer available:

  * `MiniMaxAI/MiniMax-M2.1`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="April 7, 2026">
  ## Z.ai Model Added

  The following **Model APIs** model is now available:

  * `zai-org/GLM-5.1`

  To learn more, see [Models > GLM-5.1](https://friendli.ai/models/zai-org/GLM-5.1).
</Update>

<Update label="April 3, 2026">
  ## Cohere Labs Model Family Added

  The **Dedicated Endpoints** `CohereAsrForConditionalGeneration` model family is now available. For example, you can now deploy endpoints for the following model:

  * `CohereLabs/cohere-transcribe-03-2026`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Google Model Family Added

  The **Dedicated Endpoints** `Gemma4ForConditionalGeneration` model family is now available. For example, you can now deploy endpoints for the following model:

  * `google/gemma-4-31B-it`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## DeepSeek and Qwen Model Pricing Changed

  The pricing model for the following **Model APIs** models is now token-based:

  * `deepseek-ai/DeepSeek-V3.1`
  * `Qwen/Qwen3-30B-A3B`

  To learn more, see [Pricing > Model APIs](https://friendli.ai/pricing?product=model-apis).
</Update>

<Update label="April 1, 2026">
  ## OpenAI Model Added

  The following **Model APIs** model is now available:

  * `openai/whisper-large-v3`

  To learn more, see [Models > whisper-large-v3](https://friendli.ai/models/openai/whisper-large-v3).
</Update>

<Update label="March 18, 2026">
  ## LG AI Research Model Added

  The following **Model APIs** model is now available:

  * `LGAI-EXAONE/K-EXAONE-236B-A23B`

  To learn more, see [Models > K-EXAONE-236B-A23B](https://friendli.ai/models/LGAI-EXAONE/K-EXAONE-236B-A23B).

  ***

  ## LG AI Research Model Pricing Changed

  Cached input pricing is now available for the following **Model APIs** model:

  * `LGAI-EXAONE/K-EXAONE-236B-A23B`

  To learn more, see [Pricing > Model APIs](https://friendli.ai/pricing?product=model-apis).
</Update>

<Update label="March 7, 2026">
  ## DeepSeek Model Added

  The following **Model APIs** model is now available:

  * `deepseek-ai/DeepSeek-V3.2`

  To learn more, see [Models > DeepSeek-V3.2](https://friendli.ai/models/deepseek-ai/DeepSeek-V3.2).

  ***

  ## MiniMax and Z.ai Model Pricing Changed

  Cached input pricing is now available for the following **Model APIs** models:

  * `MiniMaxAI/MiniMax-M2.1`
  * `zai-org/GLM-5`

  To learn more, see [Pricing > Model APIs](https://friendli.ai/pricing?product=model-apis).
</Update>

<Update label="March 4, 2026">
  ## Host KV Cache Added

  Host KV Cache is now available for **Dedicated Endpoints**. It offloads KV cache to host memory to extend capacity beyond GPU limits. Your endpoint retains more tokens during inference.

  To learn more, see [Endpoints > What You Can Configure](/docs/guides/dedicated-endpoints/endpoints#what-you-can-configure).

  ***

  ## Draft-Model Speculative Decoding Added

  Draft-model speculative decoding is now available for **Dedicated Endpoints**, for a curated list of target models. You can pair a target model with a draft model that proposes multiple tokens for the target to verify in parallel, which improves throughput and latency.

  To learn more, see [Speculative Decoding > Draft-Model Method](/docs/guides/dedicated-endpoints/speculative-decoding#draft-model-method).
</Update>

<Update label="March 1, 2026">
  ## MiniMax Model Pricing Changed

  Cached input pricing is now available for the following **Model APIs** model:

  * `MiniMaxAI/MiniMax-M2.5`

  To learn more, see [Pricing > Model APIs](https://friendli.ai/pricing?product=model-apis).
</Update>

<Update label="February 28, 2026">
  ## LG AI Research Model Deprecated

  The following **Model APIs** model is no longer available:

  * `LGAI-EXAONE/EXAONE-4.0.1-32B`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="February 20, 2026">
  ## LG AI Research Model Deprecated

  The following **Model APIs** model is no longer available:

  * `LGAI-EXAONE/K-EXAONE-236B-A23B`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="February 19, 2026">
  ## MiniMax Model Added

  The following **Model APIs** model is now available:

  * `MiniMaxAI/MiniMax-M2.5`

  To learn more, see [Models > MiniMax-M2.5](https://friendli.ai/models/MiniMaxAI/MiniMax-M2.5).
</Update>

<Update label="February 11, 2026">
  ## Z.ai Model Added

  The following **Model APIs** model is now available:

  * `zai-org/GLM-5`

  To learn more, see [Models > GLM-5](https://friendli.ai/models/zai-org/GLM-5).
</Update>

<Update label="January 21, 2026">
  ## Z.ai Model Family Added

  The **Dedicated Endpoints** `Glm4MoeLiteForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `zai-org/GLM-4.7-Flash`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>

<Update label="January 20, 2026">
  ## Z.ai Model Added

  The following **Model APIs** model is now available:

  * `zai-org/GLM-4.7`

  To learn more, see [Models > GLM-4.7](https://friendli.ai/models/zai-org/GLM-4.7).

  ***

  ## MiniMax Model Pricing Changed

  The pricing model for the following **Model APIs** model is now token-based:

  * `MiniMaxAI/MiniMax-M2.1`

  To learn more, see [Pricing > Model APIs](https://friendli.ai/pricing?product=model-apis).
</Update>

<Update label="January 16, 2026">
  ## DeepSeek Model Deprecated

  The following **Model APIs** model is no longer available:

  * `deepseek-ai/DeepSeek-R1-0528`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="January 14, 2026">
  ## MiniMax Model Added

  The following **Model APIs** model is now available:

  * `MiniMaxAI/MiniMax-M2.1`

  To learn more, see [Models > MiniMax-M2.1](https://friendli.ai/models/MiniMaxAI/MiniMax-M2.1).
</Update>

<Update label="January 2, 2026">
  ## LG AI Research Model Family Added

  The **Dedicated Endpoints** `ExaoneMoEForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `LGAI-EXAONE/K-EXAONE-236B-A23B`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>

<Update label="December 31, 2025">
  ## LG AI Research Model Added

  The following **Model APIs** model is now available:

  * `LGAI-EXAONE/K-EXAONE-236B-A23B`

  To learn more, see [Models > K-EXAONE-236B-A23B](https://friendli.ai/models/LGAI-EXAONE/K-EXAONE-236B-A23B).
</Update>

<Update label="December 5, 2025">
  ## Tencent Model Family Added

  The **Dedicated Endpoints** `HunYuanVLForConditionalGeneration` model family is now available. For example, you can now deploy endpoints for the following model:

  * `tencent/HunyuanOCR`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## MiniMax Model Family Added

  The **Dedicated Endpoints** `MiniMaxM2ForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `MiniMaxAI/MiniMax-M2`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Google Model Family Added

  The **Dedicated Endpoints** `Gemma3TextModel` model family is now available. For example, you can now deploy endpoints for the following model:

  * `google/embeddinggemma-300m`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Microsoft Model Family Added

  The **Dedicated Endpoints** `Phi4MMForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `microsoft/Phi-4-multimodal-instruct`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>

<Update label="December 1, 2025">
  ## DeepSeek Model Added

  The following **Model APIs** model is now available:

  * `deepseek-ai/DeepSeek-V3.1`

  To learn more, see [Models > DeepSeek-V3.1](https://friendli.ai/models/deepseek-ai/DeepSeek-V3.1).
</Update>

<Update label="November 27, 2025">
  ## Basic Plan Expanded

  The **Dedicated Endpoints** Basic plan now includes the following features:

  * Scale replicas by queued and in-flight requests with a request count scaling policy.
  * Serve multiple LoRA adapters on a single endpoint.
  * Monitor endpoint performance with metrics.
  * Inspect endpoint activity in real time with logs.

  These features were previously only available on the Enterprise plan.
</Update>

<Update label="November 21, 2025">
  ## Black Forest Labs Model Family Added

  The **Dedicated Endpoints** `FluxKontextPipeline` model family is now available. For example, you can now deploy endpoints for the following model:

  * `black-forest-labs/FLUX.1-Kontext-dev`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Ai2 Model Family Added

  The **Dedicated Endpoints** `Olmo3ForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `allenai/Olmo-3-32B-Think`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## LightOn Model Family Added

  The **Dedicated Endpoints** `LightOnOCRForConditionalGeneration` model family is now available. For example, you can now deploy endpoints for the following model:

  * `lightonai/LightOnOCR-1B-1025`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## PaddlePaddle Model Family Added

  The **Dedicated Endpoints** `PaddleOCRVLForConditionalGeneration` model family is now available. For example, you can now deploy endpoints for the following model:

  * `PaddlePaddle/PaddleOCR-VL`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## DeepSeek Model Family Added

  The **Dedicated Endpoints** `DeepseekOCRForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `deepseek-ai/DeepSeek-OCR`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>

<Update label="November 7, 2025">
  ## Qwen Model Families Added

  The **Dedicated Endpoints** `Qwen3VLForConditionalGeneration` and `Qwen3VLMoeForConditionalGeneration` model families are now available. For example, you can now deploy endpoints for the following models:

  * `Qwen/Qwen3-VL-4B-Instruct`
  * `Qwen/Qwen3-VL-30B-A3B-Instruct`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## IBM Granite Model Family Added

  The **Dedicated Endpoints** `GraniteMoeHybridForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `ibm-granite/granite-4.0-h-small`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## rednote hilab Model Family Added

  The **Dedicated Endpoints** `DotsOCRForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `rednote-hilab/dots.ocr`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>

<Update label="November 1, 2025">
  ## Qwen Model Pricing Changed

  The pricing model for the following **Model APIs** model is now token-based:

  * `Qwen/Qwen3-235B-A22B-Instruct-2507`

  To learn more, see [Pricing > Model APIs](https://friendli.ai/pricing?product=model-apis).
</Update>

<Update label="September 15, 2025">
  ## Qwen Model Family Added

  The **Dedicated Endpoints** `Qwen3NextForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `Qwen/Qwen3-Next-80B-A3B-Instruct`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Tencent Model Family Added

  The **Dedicated Endpoints** `HunYuanDenseV1ForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `tencent/Hunyuan-MT-7B`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Swiss AI Initiative Model Family Added

  The **Dedicated Endpoints** `ApertusForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `swiss-ai/Apertus-8B-Instruct-2509`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## ByteDance Seed Model Family Added

  The **Dedicated Endpoints** `SeedOssForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `ByteDance-Seed/Seed-OSS-36B-Instruct`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>

<Update label="September 12, 2025">
  ## Custom Chat Templates Added

  Custom chat templates are now available for **Dedicated Endpoints**. You can paste or upload a [Jinja](https://jinja.palletsprojects.com/en/stable/) template when you create an endpoint.

  To learn more, see [Endpoints > What You Can Configure](/docs/guides/dedicated-endpoints/endpoints#what-you-can-configure).

  ***

  ## 4-Bit Online Quantization Added

  Online quantization with 4-bit precision is now available for **Dedicated Endpoints**. You can run models on smaller instances with negligible quality impact.

  To learn more, see [Online Quantization](/docs/guides/dedicated-endpoints/online-quantization).
</Update>

<Update label="September 10, 2025">
  ## Reasoning Parser Added

  The reasoning parser is now available for **Model APIs** and **Dedicated Endpoints**. When you turn it on, responses return reasoning in its own field, separate from the message content.

  To learn more, see [Reasoning > `parse_reasoning`](/docs/guides/capabilities/reasoning#parse_reasoning).
</Update>

<Update label="September 8, 2025">
  ## K-intelligence Model Deprecated

  The following **Model APIs** model is no longer available:

  * `K-intelligence/Midm-2.0-Base-Instruct`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="September 4, 2025">
  ## K-intelligence Model Deprecated

  The following **Model APIs** model is no longer available:

  * `K-intelligence/Midm-2.0-Mini-Instruct`

  To browse all models, see [Models > Model APIs](https://friendli.ai/models?products=SERVERLESS).
</Update>

<Update label="September 1, 2025">
  ## NVIDIA B200 GPUs Added

  NVIDIA B200 GPUs are now available for **Dedicated Endpoints**, alongside A100, H100, and H200 GPUs. You'll see the new option when you create a new endpoint.

  To learn more, see [Dedicated Endpoints > GPUs and Pricing](/docs/guides/dedicated-endpoints/pricing).
</Update>

<Update label="August 22, 2025">
  ## OpenAI Model Family Added

  The **Dedicated Endpoints** `GptOssForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `openai/gpt-oss-20b`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Linkup Web Search Integration Added

  Linkup web search is now available for **Model APIs** as a built-in tool. You can use it to ground model responses with live web results.

  To learn more, see [Partnering with Linkup: Built‑in AI Web Search in Friendli Serverless Endpoints](https://friendli.ai/blog/linkup-partnership).
</Update>

<Update label="August 19, 2025">
  ## Request Count Scaling Policy Added

  A request count scaling policy is now available for **Dedicated Endpoints** on the Enterprise plan. You can scale replicas according to queued and in-flight requests.

  To learn more, see [Autoscaling > Scaling Policies](/docs/guides/dedicated-endpoints/autoscaling#scaling-policies).
</Update>

<Update label="August 8, 2025">
  ## N-gram Speculative Decoding Added

  N-gram speculative decoding is now available for **Dedicated Endpoints**. You can turn it on for predictable tasks, where it delivers substantial performance gains.

  To learn more, see [Introducing N-gram Speculative Decoding: Faster Inference for Structured Tasks](https://friendli.ai/blog/n-gram-speculative-decoding).

  ***

  ## Reasoning Output Token Limits Raised

  Output token limits are now higher for **Model APIs** reasoning models. You can run demanding tasks without response truncation.

  To learn more, see [Chat Completions API](/docs/openapi/model-apis/chat-completions).
</Update>

<Update label="August 1, 2025">
  ## HyperCLOVA X Model Family Added

  The **Dedicated Endpoints** `HyperCLOVAXForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `naver-hyperclovax/HyperCLOVAX-SEED-Think-14B`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Qwen Model Added

  The following **Model APIs** model is now available:

  * `Qwen/Qwen3-235B-A22B-Instruct-2507`

  To learn more, see [Models > Qwen3-235B-A22B-Instruct-2507](https://friendli.ai/models/Qwen/Qwen3-235B-A22B-Instruct-2507).
</Update>

<Update label="July 25, 2025">
  ## Online Quantization Added

  Online quantization is now available for **Dedicated Endpoints**. You can quantize models with no advance preparation and accelerate inference.

  To learn more, see [Announcing Online Quantization: Faster, Cheaper Inference with Same Accuracy](https://friendli.ai/blog/online-quantization).
</Update>

<Update label="July 14, 2025">
  ## LG AI Research Model Added

  The following **Model APIs** model is now available:

  * `LGAI-EXAONE/EXAONE-4.0.1-32B`

  To learn more, see [LG AI Research Partners with FriendliAI to Launch EXAONE 4.0 for Fast, Scalable API](https://friendli.ai/blog/lg-ai-research-partnership-exaone-4.0).
</Update>

<Update label="July 11, 2025">
  ## DeepSeek Model Added

  The following **Model APIs** model is now available:

  * `deepseek-ai/DeepSeek-R1-0528`

  To learn more, see [Models > DeepSeek-R1-0528](https://friendli.ai/models/deepseek-ai/DeepSeek-R1-0528).
</Update>

<Update label="July 8, 2025">
  ## rednote hilab Model Family Added

  The **Dedicated Endpoints** `Dots1ForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `rednote-hilab/dots.llm1.inst`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Z.ai Model Family Added

  The **Dedicated Endpoints** `Glm4vForConditionalGeneration` model family is now available. For example, you can now deploy endpoints for the following model:

  * `zai-org/GLM-4.1V-9B-Thinking`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Kwai Keye Model Family Added

  The **Dedicated Endpoints** `KeyeForConditionalGeneration` model family is now available. For example, you can now deploy endpoints for the following model:

  * `Kwai-Keye/Keye-VL-8B-Preview`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Tencent Model Family Added

  The **Dedicated Endpoints** `HunYuanMoEV1ForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `tencent/Hunyuan-A13B-Instruct`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Microsoft Model Family Added

  The **Dedicated Endpoints** `PhiMoEForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `microsoft/Phi-mini-MoE-instruct`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## MiniMax Model Family Added

  The **Dedicated Endpoints** `MiniMaxM1ForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `MiniMaxAI/MiniMax-M1-80k`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).

  ***

  ## Baidu Model Families Added

  The **Dedicated Endpoints** `Ernie4_5_MoeForCausalLM` and `Ernie4_5_ForCausalLM` model families are now available. For example, you can now deploy endpoints for the following models:

  * `baidu/ERNIE-4.5-21B-A3B-Thinking`
  * `baidu/ERNIE-4.5-0.3B-PT`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>

<Update label="July 3, 2025">
  ## LG AI Research Model Family Added

  The **Dedicated Endpoints** `Exaone4ForCausalLM` model family is now available. For example, you can now deploy endpoints for the following model:

  * `LGAI-EXAONE/EXAONE-4.0.1-32B`

  To browse all models, see [Models > Dedicated Endpoints](https://friendli.ai/models?products=DEDICATED).
</Update>


# Use Claude Code with FriendliAI
Source: https://friendli.ai/docs/examples/agents/claude-code

Configure Claude Code to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <Tip>
    [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect your agent to FriendliAI with one command.
  </Tip>

  1. Open your `~/.claude/settings.json` file.

  2. Add the following to the file:

     ```json highlight={3,4,5,7} theme={null}
     {
       "env": {
         "ANTHROPIC_AUTH_TOKEN": "<FRIENDLIAI_API_KEY>",
         "ANTHROPIC_BASE_URL": "https://api.friendli.ai/serverless",
         "ANTHROPIC_DEFAULT_HAIKU_MODEL": "<FRIENDLIAI_MODEL_ID>"
       },
       "model": "<FRIENDLIAI_MODEL_ID>"
     }
     ```

     Replace the following placeholders:

     * `<FRIENDLIAI_API_KEY>`: Your FriendliAI API key.
     * `<FRIENDLIAI_MODEL_ID>`: The FriendliAI model ID. For example, `zai-org/GLM-5.3`.

  3. Save your file.

  To learn more, see [Claude Code Docs > Set environment variables](https://code.claude.com/docs/en/env-vars#set-environment-variables).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Cline with FriendliAI
Source: https://friendli.ai/docs/examples/agents/cline

Configure Cline to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <AgentAlternativeTip />

  1. Open Visual Studio Code. Then, open the Cline panel.

  2. In the panel's upper-right corner, click **Settings**.

  3. Connect Cline to FriendliAI's API:
     1. In **API Provider**, select **OpenAI Compatible**.

     2. In **Base URL**, paste the following FriendliAI base URL:

        ```text theme={null}
        https://api.friendli.ai/serverless/v1
        ```

     3. In **OpenAI Compatible API Key**, paste your FriendliAI API key.

     4. In **Model ID**, enter the FriendliAI model ID. For example, enter `zai-org/GLM-5.3`.

     5. Click **Done**.

  To learn more, see [Cline Docs > OpenAI Compatible](https://docs.cline.bot/provider-config/openai-compatible).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Codex with FriendliAI
Source: https://friendli.ai/docs/examples/agents/codex

Configure Codex CLI to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <Tip>
    [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect your agent to FriendliAI with one command.
  </Tip>

  1. Open your `~/.codex/config.toml` file.

  2. Add the following to the start of the file:

     ```toml highlight={1,7} theme={null}
     model = "<FRIENDLIAI_MODEL_ID>"
     model_provider = "friendliai"

     [model_providers.friendliai]
     name = "FriendliAI"
     base_url = "https://api.friendli.ai/serverless/v1"
     experimental_bearer_token = "<FRIENDLIAI_API_KEY>"
     ```

     Replace the following placeholders:

     * `<FRIENDLIAI_API_KEY>`: Your FriendliAI API key.
     * `<FRIENDLIAI_MODEL_ID>`: The FriendliAI model ID. For example, `zai-org/GLM-5.3`.

  3. Save your file.

  To learn more, see [Codex Docs > Advanced configuration](https://learn.chatgpt.com/docs/config-file/config-advanced).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Cursor with FriendliAI
Source: https://friendli.ai/docs/examples/agents/cursor

Configure Cursor to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <Tip>
    [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect your agent to FriendliAI with one command.
  </Tip>

  1. Open Cursor. Then, open **Cursor Settings**.

  2. In the left sidebar, click **Models**.

  3. Expand **API Keys**.

  4. Add your FriendliAI API key:
     1. In **OpenAI API Key**, paste your key.

     2. Turn on **OpenAI API Key**.

     3. When prompted, click **Enable OpenAI API Key**.

  5. Override the base URL:
     1. Turn on **Override OpenAI Base URL**.

     2. Paste the following FriendliAI base URL:

        ```text theme={null}
        https://api.friendli.ai/serverless/v1
        ```

  6. Add the FriendliAI model ID:
     1. In **Models**, enter the model ID. For example, enter `zai-org/GLM-5.3`.

     2. Click **Add Custom Model**.

     3. To confirm, click **Add**.

  To learn more, see [Cursor Docs > Bring your own API Key](https://cursor.com/help/models-and-usage/api-keys).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use DeepSeek Harness with FriendliAI
Source: https://friendli.ai/docs/examples/agents/deepseek-harness

Configure DeepSeek Harness to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <Tip>
    [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect your agent to FriendliAI with one command.
  </Tip>

  1. In your terminal, run the following command:

     ```sh theme={null}
     npx @deepseek-ai/dsh web
     ```

     DeepSeek Harness opens.

  2. Open **Settings**. Then, click **Models**.

  3. Click **Add a custom provider**. Then, connect DeepSeek Harness to FriendliAI:
     1. In **Provider ID**, enter `friendliai`.

     2. *(Optional)* In **Display name**, enter "FriendliAI".

     3. In **Base URL**, paste the following FriendliAI base URL:

        ```text theme={null}
        https://api.friendli.ai/serverless/v1
        ```

     4. In **API protocol**, select `openai-completions`.

     5. In **API key**, paste your FriendliAI API key.

     6. Click **Fetch available models**. Then, click **Add selected**.

     7. Click **Create provider**.

  4. When you're ready to send a prompt, select a model. For example, select **zai-org/GLM-5.3**.

  To learn more, see [DeepSeek Harness Docs > Configure models](https://deepseek-harness.github.io/deepseek-harness/en/guide/providers).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Set Up Your Agent with FriendliLink
Source: https://friendli.ai/docs/examples/agents/friendlilink

Set up your coding agent with the FriendliLink CLI to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

Use [FriendliLink](https://github.com/friendliai/friendlilink) to connect your agent to FriendliAI with one command. Once you complete these steps, your agent will send requests to FriendliAI.

To get started, complete the following steps:

<Steps>
  <Step title="Install Your Agent">
    If you haven't already, install your agent:

    * [Claude Code](https://code.claude.com/docs)
    * [Codex](https://learn.chatgpt.com/docs/codex/cli#getting-started)
    * [Cursor](https://cursor.com/docs/get-started/quickstart)
    * [DeepSeek Harness](https://deepseek-harness.github.io/deepseek-harness/en/guide/quickstart)
    * [Hermes Agent](https://hermes-agent.nousresearch.com/docs/getting-started/installation)
    * [OpenCode](https://opencode.ai/docs#install)
    * [Pi](https://pi.dev/docs/latest/quickstart)
  </Step>

  <Step title="Get a FriendliAI API Key">
    If you need a new API key, [create one](/docs/examples/agents/overview#create-an-api-key).
  </Step>

  <Step title="Install FriendliLink">
    In your terminal, run the following command:

    ```sh theme={null}
    curl -fsSL https://raw.githubusercontent.com/friendliai/friendlilink/main/install.sh | bash
    ```

    The command installs the FriendliLink CLI. You can use `friendlilink` or `frlink`.
  </Step>

  <Step title="Save Your API Key">
    In your terminal, run the following command:

    ```sh theme={null}
    frlink login
    ```

    Then, paste your FriendliAI API key.
  </Step>

  <Step title="Set Up Your Agent">
    In your terminal, run the following command:

    ```sh theme={null}
    frlink <FRIENDLILINK_AGENT_ID> on
    ```

    Replace `<FRIENDLILINK_AGENT_ID>` with your agent's ID:

    | Agent            | FriendliLink Agent ID |
    | ---------------- | --------------------- |
    | Claude Code      | `claude`              |
    | Codex            | `codex`               |
    | Cursor           | `cursor`              |
    | DeepSeek Harness | `dsh`                 |
    | Hermes Agent     | `hermes`              |
    | OpenCode         | `opencode`            |
    | Pi               | `pi`                  |

    Then, select a model. For example, select **zai-org/GLM-5.3**.
  </Step>

  <Step title="Send a Prompt to Your Agent">
    Open your agent and send a prompt. For example, use the following prompt:

    <Prompt description="Explore your codebase." icon="robot">
      What can you tell me about this codebase?
    </Prompt>

    To learn more, see [FriendliLink README](https://github.com/friendliai/friendlilink).
  </Step>
</Steps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Hermes Agent with FriendliAI
Source: https://friendli.ai/docs/examples/agents/hermes-agent

Configure Hermes Agent to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <Tip>
    [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect your agent to FriendliAI with one command.
  </Tip>

  1. In your terminal, run the following command:

     ```sh theme={null}
     hermes model
     ```

     Hermes Agent opens.

  2. Select **Custom endpoint (enter URL manually)**.

  3. Connect Hermes Agent to FriendliAI's API:
     1. In **API base URL**, paste the following FriendliAI base URL:

        ```text theme={null}
        https://api.friendli.ai/serverless/v1
        ```

     2. In **API Key**, paste your FriendliAI API key.

     3. In **Select API compatibility mode**, press <kbd>Enter</kbd> to auto-detect.

     4. In **Available models**, select a model. For example, select **zai-org/GLM-5.3**.

     5. In **Context length in tokens**, press <kbd>Enter</kbd> to auto-detect.

     6. *(Optional)* In **Display name**, enter "FriendliAI".

  To learn more, see [Hermes Agent Docs > AI Providers](https://hermes-agent.nousresearch.com/docs/integrations/providers).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Kilo Code with FriendliAI
Source: https://friendli.ai/docs/examples/agents/kilo-code

Configure Kilo Code to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <AgentAlternativeTip />

  1. Open Visual Studio Code. Then, open the Kilo Code panel.

  2. In the panel's upper-right corner, click **Settings**.

  3. In the left sidebar, click **Providers**.

  4. Click **Show more providers**.

  5. Search for and select **Friendli**.

  6. Paste your FriendliAI API key. Then, click **Submit**.

  7. Open the Kilo Code panel and select a model. For example, select `zai-org/GLM-5.3`.

  To learn more, see [Kilo Code Docs > Model Selection Guide](https://kilo.ai/docs/code-with-ai/agents/model-selection).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use OpenClaw with FriendliAI
Source: https://friendli.ai/docs/examples/agents/openclaw

Configure OpenClaw to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  1. In your terminal, run the following command:

     ```sh theme={null}
     openclaw onboard --install-daemon
     ```

     OpenClaw opens.

  2. Review the security disclaimer. If you understand it, continue.

  3. In **Setup mode**, select **Quickstart**.

  4. In **Model/auth provider**, select **More...**. Then, select **Custom Provider**.

  5. Connect OpenClaw to FriendliAI's API:
     1. In **API Base URL**, paste the following FriendliAI base URL:

        ```text theme={null}
        https://api.friendli.ai/serverless/v1
        ```

     2. When asked **How do you want to provide this API Key?**, select **Paste API Key now**. Then, paste your FriendliAI API key.

     3. In **Endpoint compatibility**, select **OpenAI-compatible**.

     4. In **Model ID**, enter the FriendliAI model ID. For example, enter `zai-org/GLM-5.3`.

     5. *(Optional)* In **Endpoint ID**, enter "FriendliAI".

     6. In **Model alias**, press <kbd>Enter</kbd>.

     7. When asked **Does this model support image input?**, select **Yes** or **No**.

  6. Follow the onscreen instructions.

  To learn more, see [OpenClaw Docs > Onboarding (CLI)](https://docs.openclaw.ai/start/wizard).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use OpenCode with FriendliAI
Source: https://friendli.ai/docs/examples/agents/opencode

Configure OpenCode to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <Tip>
    [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect your agent to FriendliAI with one command.
  </Tip>

  1. In your terminal, run the following command:

     ```sh theme={null}
     opencode
     ```

     OpenCode opens.

  2. Run the following OpenCode slash command:

     ```text theme={null}
     /connect
     ```

  3. Search for and select **Friendli**.

  4. In **API Key**, paste your FriendliAI API key.

  5. Select a model. For example, select **GLM-5.3**.

  To learn more, see [OpenCode Docs > Providers](https://opencode.ai/docs/providers).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use OpenHands with FriendliAI
Source: https://friendli.ai/docs/examples/agents/openhands

Configure OpenHands to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  1. Start Agent Canvas. Then, open `http://localhost:8000`.

  2. On **Choose your agent**, select **OpenHands**. Then, click **Next**.

  3. On **Set up your LLM**, click the **Advanced** tab.

  4. Connect OpenHands to FriendliAI's API:
     1. In **Authentication**, select **API key**.

     2. In **Custom Model**, enter the FriendliAI model ID with the `friendliai/` provider prefix. For example, enter `friendliai/zai-org/GLM-5.3`.

     3. In **Base URL**, paste the following FriendliAI base URL:

        ```text theme={null}
        https://api.friendli.ai/serverless/v1
        ```

     4. In **API Key**, paste your FriendliAI API key.

     5. Click **Next**.

  <Tip>If you already finished the first-time setup, open **Settings > LLM** and add a new LLM profile with the same values instead.</Tip>

  To learn more, see [OpenHands Docs > Manage LLM Profiles](https://docs.openhands.dev/openhands/usage/agent-canvas/llm-profiles).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Your Agent with FriendliAI
Source: https://friendli.ai/docs/examples/agents/overview

Set up your favorite coding agent with FriendliAI to run fast, cost-efficient, and reliable open-source models.

Set up your favorite agent with FriendliAI to connect to fast, cost-efficient, and reliable open-weight models. With FriendliAI, your agent responds with low first-token latency, processes codebase-scale requests at low per-token cost, and sustains throughput across long-running tasks.

## Create an API Key

<Steps>
  <Step title="Create a Friendli Suite Account">
    If you haven't already, [sign up](https://auth.friendli.ai/sign-up) for an account.
  </Step>

  <Step title="Sign In">
    Then, [sign in](https://auth.friendli.ai). Friendli Suite opens your dashboard.
  </Step>

  <Step title="Create a FriendliAI API Key">
    1. In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**.
    2. In the upper-right corner, click <Icon icon="plus" /> **Create API Key**.
    3. (Optional) Name your API key and set an expiration date.
    4. Click **Create Key**.
    5. Click <Icon icon="copy" /> **Copy**.
  </Step>
</Steps>

## Choose Your Agent

<Tip>
  [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect Claude Code, Codex, Cursor, DeepSeek Harness, Hermes Agent, OpenCode, and Pi to FriendliAI -- with one command.
</Tip>

Choose the agent you want to set up:

<CardGroup>
  <Card title="Claude Code" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/claude-code.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=9c4bf33ae4d81e93dec4f185477fa4c8" href="/docs/examples/agents/claude-code" />

  <Card title="Cline" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/cline.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=7fa6e62f59e9a807cb136664ade1d90e" href="/docs/examples/agents/cline" />

  <Card title="Codex" icon="https://mintcdn.com/friendliai/3Y9MxkMqtZLA0bIV/static/images/examples/agents/codex.svg?fit=max&auto=format&n=3Y9MxkMqtZLA0bIV&q=85&s=fee97c30f2879382ef664bf9e6ed37e4" href="/docs/examples/agents/codex" />

  <Card title="Cursor" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/cursor.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=8c890caaac0f6e4cbdfa92a9b2767f0f" href="/docs/examples/agents/cursor" />

  <Card title="DeepSeek Harness" icon="https://mintcdn.com/friendliai/mYmiUJysoER-Vfy3/static/images/examples/agents/deepseek-harness.svg?fit=max&auto=format&n=mYmiUJysoER-Vfy3&q=85&s=451a59945f219886bc0c63d5181a2968" href="/docs/examples/agents/deepseek-harness" />

  <Card title="Hermes Agent" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/hermes-agent.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=ea425107c2c69ac758b12fab7dddeec6" href="/docs/examples/agents/hermes-agent" />

  <Card title="Kilo Code" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/kilo-code.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=b05b7bbaaf3b86fe2e8ad29a5e90d819" href="/docs/examples/agents/kilo-code" />

  <Card title="OpenClaw" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/openclaw.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=11e87f38d21472bd902a324bc21052ec" href="/docs/examples/agents/openclaw" />

  <Card title="OpenCode" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/opencode.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=689cbd6dbead5ce016522a10956faa8f" href="/docs/examples/agents/opencode" />

  <Card title="OpenHands" icon="https://mintcdn.com/friendliai/OqBtpQxRtuA8s0Rk/static/images/examples/agents/openhands.svg?fit=max&auto=format&n=OqBtpQxRtuA8s0Rk&q=85&s=034f90a693915fcdbb728093bdd95737" href="/docs/examples/agents/openhands" />

  <Card title="Pi" icon="https://mintcdn.com/friendliai/3qtUSVRfjaVm2dWP/static/images/examples/agents/pi.svg?fit=max&auto=format&n=3qtUSVRfjaVm2dWP&q=85&s=1d328a7567e92b47306101d9f84e06b9" href="/docs/examples/agents/pi" />

  <Card title="Zoo Code" icon="https://mintcdn.com/friendliai/QQ_244i73sKFTyKo/static/images/examples/agents/zoo-code.svg?fit=max&auto=format&n=QQ_244i73sKFTyKo&q=85&s=df142708ffc2e37acf41df0ebc78a69b" href="/docs/examples/agents/zoo-code" />
</CardGroup>


# Use Pi with FriendliAI
Source: https://friendli.ai/docs/examples/agents/pi

Configure Pi to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  <Tip>
    [Set Up FriendliLink](/docs/examples/agents/friendlilink) and connect your agent to FriendliAI with one command.
  </Tip>

  1. Open your `~/.pi/agent/models.json` file.

  2. Add the following to the file:

     ```json highlight={6,9,10} theme={null}
     {
       "providers": {
         "friendliai-chat-completions": {
           "baseUrl": "https://api.friendli.ai/serverless/v1",
           "api": "openai-completions",
           "apiKey": "<FRIENDLIAI_API_KEY>",
           "models": [
             {
               "id": "<FRIENDLIAI_MODEL_ID>",
               "reasoning": <FRIENDLIAI_MODEL_REASONING>
             }
           ]
         }
       }
     }
     ```

     Replace the following placeholders:

     * `<FRIENDLIAI_API_KEY>`: Your FriendliAI API key.
     * `<FRIENDLIAI_MODEL_ID>`: The FriendliAI model ID. For example, `zai-org/GLM-5.3`.
     * `<FRIENDLIAI_MODEL_REASONING>`: Whether the model is a reasoning model. For example, `true` for `zai-org/GLM-5.3`.

  3. Save your file.

  To learn more, see [Pi Docs > Custom Models](https://pi.dev/docs/latest/models).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Zoo Code with FriendliAI
Source: https://friendli.ai/docs/examples/agents/zoo-code

Configure Zoo Code to use FriendliAI for fast, cost-efficient, and reliable open-source model inference.

<AgentIntro />

<AgentSteps>
  1. Open Visual Studio Code. Then, open the Zoo Code panel.

  2. In the panel's upper-right corner, click **Settings**.

  3. Connect Zoo Code to FriendliAI's API:
     1. In **API Provider**, select **FriendliAI**.

     2. In **FriendliAI API Key**, paste your FriendliAI API key.

     3. In **Model**, select a model. For example, select **zai-org/GLM-5.3**.

  To learn more, see [Zoo Code Docs > FriendliAI](https://docs.zoocode.dev/providers/friendli).
</AgentSteps>

<Card title="Help" icon="circle-question" href="https://learn.friendli.ai/collections/1270196079-integrations-and-compatibility">
  Find answers to common agent setup questions.
</Card>


# Use Popular Models on FriendliAI
Source: https://friendli.ai/docs/examples/models/overview

Choose a popular open-weight model to build with FriendliAI. Create an API key and start sending requests with low latency and low per-token cost.

Connect to a fast, cost-efficient, and reliable open-weight model of your choice. With FriendliAI, you can send requests to models with low first-token latency. These models can process codebase-scale requests at low per-token cost and sustain throughput across long-running tasks.

## Create an API Key

<Steps>
  <Step title="Create a Friendli Suite Account">
    If you haven't already, [sign up](https://auth.friendli.ai/sign-up) for an account.
  </Step>

  <Step title="Sign In">
    Then, [sign in](https://auth.friendli.ai). Friendli Suite opens your dashboard.
  </Step>

  <Step title="Create a FriendliAI API Key">
    1. In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**.
    2. In the upper-right corner, click <Icon icon="plus" /> **Create API Key**.
    3. (Optional) Name your API key and set an expiration date.
    4. Click **Create Key**.
    5. Click <Icon icon="copy" /> **Copy**.
  </Step>
</Steps>

## Choose a Model

You're ready to choose one of FriendliAI's popular models to build your next project.

Choose the model you want to work with:

<CardGroup>
  <Card title="Z.ai" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/zai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=0e59d2daee9e670622fb06169d4e4750" href="/docs/examples/models/zai-glm-5-2/overview">
    GLM-5.2
  </Card>

  <Card title="Z.ai" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/zai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=0e59d2daee9e670622fb06169d4e4750" href="/docs/examples/models/zai-glm-5-3/overview">
    GLM-5.3 <Badge>New</Badge>
  </Card>

  <Card title="Z.ai" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/zai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=0e59d2daee9e670622fb06169d4e4750" href="/docs/examples/models/zai-glm-5-3-flash/overview">
    GLM-5.3-Flash <Badge>New</Badge>
  </Card>
</CardGroup>


# Control GLM-5.2 Reasoning
Source: https://friendli.ai/docs/examples/models/zai-glm-5-2/control-reasoning

Control GLM-5.2 reasoning on FriendliAI. Turn thinking on or off, set the reasoning effort and budget, and parse reasoning from responses.

GLM-5.2 is a [controllable reasoning model](/docs/guides/capabilities/reasoning#reasoning-model-types). You can control its reasoning with the following parameters:

| Parameter           | Data Type            | Default |
| ------------------- | -------------------- | ------- |
| `enable_thinking`   | boolean              | `true`  |
| `parse_reasoning`   | boolean              | `true`  |
| `include_reasoning` | boolean              | `true`  |
| `reasoning_effort`  | enum (`high`, `max`) | `max`   |
| `reasoning_budget`  | integer              | `null`  |
| `clear_thinking`    | boolean              | `true`  |

If you don't set a parameter, the model uses its default. You can edit a parameter to control how the model reasons.

## Control Reasoning

To learn how to control the model's reasoning, see the following sections:

<Steps>
  <Step title="Turn Reasoning On" icon="dot">
    By default, reasoning's on. To turn it off, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-19} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.2",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "chat_template_kwargs": {
            "enable_thinking": False,
          },
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14-16} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.2",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "chat_template_kwargs": {
          "enable_thinking": false,
        },
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10-12} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.2",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "chat_template_kwargs": {
            "enable_thinking": false
          }
      }'
      ```
    </CodeGroup>

    To learn more, see [`enable_thinking`](/docs/guides/capabilities/reasoning#enable_thinking).
  </Step>

  <Step title="Parse Reasoning" icon="dot">
    By default, reasoning's parsed. To turn parsing off, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.2",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "parse_reasoning": False,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.2",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "parse_reasoning": false,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.2",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "parse_reasoning": false
      }'
      ```
    </CodeGroup>

    To learn more, see [`parse_reasoning`](/docs/guides/capabilities/reasoning#parse_reasoning).
  </Step>

  <Step title="Include Reasoning" icon="dot">
    By default, reasoning's included. To exclude it, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.2",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "include_reasoning": False,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.2",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "include_reasoning": false,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.2",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "include_reasoning": false
      }'
      ```
    </CodeGroup>

    To learn more, see [`include_reasoning`](/docs/guides/capabilities/reasoning#include_reasoning).
  </Step>

  <Step title="Set the Reasoning Effort" icon="dot">
    By default, reasoning effort's set to `max`. To set it to `high`, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.2",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "reasoning_effort": "high",
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.2",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "reasoning_effort": "high",
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.2",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "reasoning_effort": "high"
      }'
      ```
    </CodeGroup>

    To learn more, see [`reasoning_effort`](/docs/guides/capabilities/reasoning#reasoning_effort).
  </Step>

  <Step title="Set a Reasoning Budget" icon="dot">
    By default, there's no reasoning budget. To set one (for example, to 10,000 tokens), use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.2",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "reasoning_budget": 10000,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.2",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "reasoning_budget": 10000,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.2",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "reasoning_budget": 10000
      }'
      ```
    </CodeGroup>

    To learn more, see [`reasoning_budget`](/docs/guides/capabilities/reasoning#reasoning_budget).
  </Step>

  <Step title="Clear Reasoning" icon="dot">
    By default, reasoning's cleared. To preserve it, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-19} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.2",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "chat_template_kwargs": {
            "clear_thinking": False,
          },
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14-16} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.2",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "chat_template_kwargs": {
          "clear_thinking": false,
        },
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10-12} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.2",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "chat_template_kwargs": {
            "clear_thinking": false
          }
      }'
      ```
    </CodeGroup>

    To learn more, see [`clear_thinking`](/docs/guides/capabilities/reasoning#clear_thinking).
  </Step>
</Steps>


# Use GLM-5.2 with FriendliAI
Source: https://friendli.ai/docs/examples/models/zai-glm-5-2/overview

Use Z.ai's GLM-5.2 on FriendliAI. Review the model ID, context window, and pricing, then choose a feature to start building.

Before [GLM-5.3](/docs/examples/models/zai-glm-5-3/overview), GLM-5.2 was Z.ai's flagship open-weight model. It's built for coding agents and long-horizon tasks. Its one-million-token context window can process an entire codebase in a single request. And you can balance response quality and cost by editing the model's reasoning parameters.

To use GLM-5.2, see the following model properties:

| Model Property    | Value             |
| ----------------- | ----------------- |
| Model ID          | `zai-org/GLM-5.2` |
| Input Modalities  | Text              |
| Output Modalities | Text              |
| Context Window    | 1,048,576 tokens  |
| Max Output        | 131,072 tokens    |

To estimate costs, see how tokens are priced:

| Pricing      | Cost per 1M tokens |
| ------------ | ------------------ |
| Input        | \$1.40             |
| Cached Input | \$0.26             |
| Output       | \$4.40             |

## Choose a Model Feature

Choose what you'd like to do with the model:

<Card title="Send Requests" icon="arrow-left-arrow-right" href="/docs/examples/models/zai-glm-5-2/send-requests">
  Connect to GLM-5.2 with the OpenAI-compatible Chat Completions API.
</Card>

<Card title="Control Reasoning" icon="brain" href="/docs/examples/models/zai-glm-5-2/control-reasoning">
  Edit GLM-5.2's reasoning parameters.
</Card>

## Other Resources

<CardGroup>
  <Card title="FriendliAI Models" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/friendliai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=466b0a104b441d83d3742ce60c8c2a99" href="https://friendli.ai/models/zai-org/GLM-5.2" />

  <Card title="Z.ai Docs" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/zai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=0e59d2daee9e670622fb06169d4e4750" href="https://docs.z.ai/guides/llm/glm-5.2" />

  <Card title="Hugging Face Models" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/hugging-face.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=bc3d036a5f064b9b7b2087ca2528729a" href="https://huggingface.co/zai-org/GLM-5.2" />
</CardGroup>


# Send Requests to GLM-5.2
Source: https://friendli.ai/docs/examples/models/zai-glm-5-2/send-requests

Send your first request to GLM-5.2 with the OpenAI-compatible Chat Completions API using Python, JavaScript, or cURL.

Use the OpenAI-compatible Chat Completions API to connect to GLM-5.2. You can use the OpenAI Python or JavaScript SDK, or send requests directly with cURL.

If you need a new API key, [create one](/docs/examples/models/overview#create-an-api-key).

## Send a Request

<Steps>
  <Step title="Set Up Your API Key">
    In your terminal, run the following command:

    ```bash theme={null}
    export FRIENDLIAI_API_KEY="<FRIENDLIAI_API_KEY>"
    ```

    Replace `<FRIENDLIAI_API_KEY>` with your API key.
  </Step>

  <Step title="Install the OpenAI SDK">
    In your terminal, run the following command:

    <CodeGroup>
      ```bash OpenAI Python SDK theme={null}
      pip install openai
      ```

      ```bash OpenAI JavaScript SDK theme={null}
      npm install openai
      ```
    </CodeGroup>
  </Step>

  <Step title="Send Your Request to FriendliAI">
    Run the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.2",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.2",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.2",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ]
      }'
      ```
    </CodeGroup>

    <Check>
      FriendliAI returns a response, similar to the following:

      ```json expandable wrap theme={null}
      {
        "id": "chatcmpl-04eb40ba7fc14cf8a51b7869d9cfa76f",
        "object": "chat.completion",
        "choices": [
          {
            "index": 0,
            "message": {
              "role": "assistant",
              "content": "FriendliAI is a generative AI infrastructure company that provides optimized solutions to help businesses deploy large language models faster, more efficiently, and at a lower cost.",
              "reasoning": "...",
              "reasoning_content": "..."
            },
            "logprobs": null,
            "finish_reason": "stop"
          }
        ],
        "created": 1781892285,
        "usage": {
          "completion_tokens": 412,
          "prompt_tokens": 27,
          "prompt_tokens_details": {
            "cached_tokens": 0
          },
          "total_tokens": 439
        },
        "model": "zai-org/GLM-5.2"
      }
      ```
    </Check>
  </Step>
</Steps>


# Control GLM-5.3-Flash Reasoning
Source: https://friendli.ai/docs/examples/models/zai-glm-5-3-flash/control-reasoning

Control GLM-5.3-Flash reasoning on FriendliAI. Set the reasoning effort and budget, parse reasoning from responses, and clear reasoning between requests.

GLM-5.3-Flash is an [always-on reasoning model](/docs/guides/capabilities/reasoning#reasoning-model-types). You can control its reasoning with the following parameters:

| Parameter           | Data Type                   | Default |
| ------------------- | --------------------------- | ------- |
| `parse_reasoning`   | boolean                     | `true`  |
| `include_reasoning` | boolean                     | `true`  |
| `reasoning_effort`  | enum (`low`, `high`, `max`) | `max`   |
| `reasoning_budget`  | integer                     | `null`  |
| `clear_thinking`    | boolean                     | `false` |

If you don't set a parameter, the model uses its default. You can edit a parameter to control how the model reasons.

## Control Reasoning

To learn how to control the model's reasoning, see the following sections:

<Steps>
  <Step title="Parse Reasoning" icon="dot">
    By default, reasoning's parsed. To turn parsing off, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3-Flash",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "parse_reasoning": False,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3-Flash",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "parse_reasoning": false,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3-Flash",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "parse_reasoning": false
      }'
      ```
    </CodeGroup>

    To learn more, see [`parse_reasoning`](/docs/guides/capabilities/reasoning#parse_reasoning).
  </Step>

  <Step title="Include Reasoning" icon="dot">
    By default, reasoning's included. To exclude it, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3-Flash",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "include_reasoning": False,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3-Flash",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "include_reasoning": false,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3-Flash",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "include_reasoning": false
      }'
      ```
    </CodeGroup>

    To learn more, see [`include_reasoning`](/docs/guides/capabilities/reasoning#include_reasoning).
  </Step>

  <Step title="Set the Reasoning Effort" icon="dot">
    By default, reasoning effort's set to `max`. To set it to `low`, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3-Flash",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "reasoning_effort": "low",
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3-Flash",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "reasoning_effort": "low",
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3-Flash",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "reasoning_effort": "low"
      }'
      ```
    </CodeGroup>

    To learn more, see [`reasoning_effort`](/docs/guides/capabilities/reasoning#reasoning_effort).
  </Step>

  <Step title="Set a Reasoning Budget" icon="dot">
    By default, there's no reasoning budget. To set one (for example, to 10,000 tokens), use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3-Flash",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "reasoning_budget": 10000,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3-Flash",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "reasoning_budget": 10000,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3-Flash",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "reasoning_budget": 10000
      }'
      ```
    </CodeGroup>

    To learn more, see [`reasoning_budget`](/docs/guides/capabilities/reasoning#reasoning_budget).
  </Step>

  <Step title="Clear Reasoning" icon="dot">
    By default, reasoning's preserved. To clear it, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-19} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3-Flash",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "chat_template_kwargs": {
            "clear_thinking": True,
          },
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14-16} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3-Flash",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "chat_template_kwargs": {
          "clear_thinking": true,
        },
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10-12} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3-Flash",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "chat_template_kwargs": {
            "clear_thinking": true
          }
      }'
      ```
    </CodeGroup>

    To learn more, see [`clear_thinking`](/docs/guides/capabilities/reasoning#clear_thinking).
  </Step>
</Steps>


# Use GLM-5.3-Flash with FriendliAI
Source: https://friendli.ai/docs/examples/models/zai-glm-5-3-flash/overview

Use Z.ai's GLM-5.3-Flash on FriendliAI. Review the model ID, context window, and pricing, then choose a feature to start building.

GLM-5.3-Flash is Z.ai's efficient open-weight model. It's small, yet outperforms [GLM-5.2](/docs/examples/models/zai-glm-5-2/overview) at about one-tenth the price. If you need better performance for more complex tasks, try Z.ai's flagship: [GLM-5.3](/docs/examples/models/zai-glm-5-3/overview).

To use GLM-5.3-Flash, see the following model properties:

| Model Property    | Value                   |
| ----------------- | ----------------------- |
| Model ID          | `zai-org/GLM-5.3-Flash` |
| Input Modalities  | Text, Images, Video     |
| Output Modalities | Text                    |
| Context Window    | 1,048,576 tokens        |
| Max Output        | 131,072 tokens          |

To estimate costs, see how tokens are priced:

| Pricing      | Cost per 1M tokens |
| ------------ | ------------------ |
| Input        | \$0.15             |
| Cached Input | \$0.03             |
| Output       | \$0.50             |

## Choose a Model Feature

Choose what you'd like to do with the model:

<Card title="Send Requests" icon="arrow-left-arrow-right" href="/docs/examples/models/zai-glm-5-3-flash/send-requests">
  Connect to GLM-5.3-Flash with the OpenAI-compatible Chat Completions API.
</Card>

<Card title="Control Reasoning" icon="brain" href="/docs/examples/models/zai-glm-5-3-flash/control-reasoning">
  Edit GLM-5.3-Flash's reasoning parameters.
</Card>

## Other Resources

<CardGroup>
  <Card title="FriendliAI Models" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/friendliai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=466b0a104b441d83d3742ce60c8c2a99" href="https://friendli.ai/models/zai-org/GLM-5.3-Flash" />

  <Card title="Z.ai Docs" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/zai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=0e59d2daee9e670622fb06169d4e4750" href="https://docs.z.ai/guides/llm/glm-5.3-flash" />

  <Card title="Hugging Face Models" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/hugging-face.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=bc3d036a5f064b9b7b2087ca2528729a" href="https://huggingface.co/zai-org/GLM-5.3-Flash" />
</CardGroup>


# Send Requests to GLM-5.3-Flash
Source: https://friendli.ai/docs/examples/models/zai-glm-5-3-flash/send-requests

Send your first request to GLM-5.3-Flash with the OpenAI-compatible Chat Completions API using Python, JavaScript, or cURL.

Use the OpenAI-compatible Chat Completions API to connect to GLM-5.3-Flash. You can use the OpenAI Python or JavaScript SDK, or send requests directly with cURL.

If you need a new API key, [create one](/docs/examples/models/overview#create-an-api-key).

## Send a Request

<Steps>
  <Step title="Set Up Your API Key">
    In your terminal, run the following command:

    ```bash theme={null}
    export FRIENDLIAI_API_KEY="<FRIENDLIAI_API_KEY>"
    ```

    Replace `<FRIENDLIAI_API_KEY>` with your API key.
  </Step>

  <Step title="Install the OpenAI SDK">
    In your terminal, run the following command:

    <CodeGroup>
      ```bash OpenAI Python SDK theme={null}
      pip install openai
      ```

      ```bash OpenAI JavaScript SDK theme={null}
      npm install openai
      ```
    </CodeGroup>
  </Step>

  <Step title="Send Your Request to FriendliAI">
    Run the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3-Flash",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3-Flash",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3-Flash",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ]
      }'
      ```
    </CodeGroup>

    <Check>
      FriendliAI returns a response, similar to the following:

      ```json expandable wrap theme={null}
      {
        "id": "chatcmpl-04eb40ba7fc14cf8a51b7869d9cfa76f",
        "object": "chat.completion",
        "choices": [
          {
            "index": 0,
            "message": {
              "role": "assistant",
              "content": "FriendliAI is a generative AI infrastructure company that provides optimized solutions to help businesses deploy large language models faster, more efficiently, and at a lower cost.",
              "reasoning": "...",
              "reasoning_content": "..."
            },
            "logprobs": null,
            "finish_reason": "stop"
          }
        ],
        "created": 1781892285,
        "usage": {
          "completion_tokens": 412,
          "prompt_tokens": 27,
          "prompt_tokens_details": {
            "cached_tokens": 0
          },
          "total_tokens": 439
        },
        "model": "zai-org/GLM-5.3-Flash"
      }
      ```
    </Check>
  </Step>
</Steps>


# Control GLM-5.3 Reasoning
Source: https://friendli.ai/docs/examples/models/zai-glm-5-3/control-reasoning

Control GLM-5.3 reasoning on FriendliAI. Set the reasoning effort and budget, parse reasoning from responses, and clear reasoning between requests.

GLM-5.3 is an [always-on reasoning model](/docs/guides/capabilities/reasoning#reasoning-model-types). You can control its reasoning with the following parameters:

| Parameter           | Data Type                   | Default |
| ------------------- | --------------------------- | ------- |
| `parse_reasoning`   | boolean                     | `true`  |
| `include_reasoning` | boolean                     | `true`  |
| `reasoning_effort`  | enum (`low`, `high`, `max`) | `max`   |
| `reasoning_budget`  | integer                     | `null`  |
| `clear_thinking`    | boolean                     | `false` |

If you don't set a parameter, the model uses its default. You can edit a parameter to control how the model reasons.

## Control Reasoning

To learn how to control the model's reasoning, see the following sections:

<Steps>
  <Step title="Parse Reasoning" icon="dot">
    By default, reasoning's parsed. To turn parsing off, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "parse_reasoning": False,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "parse_reasoning": false,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "parse_reasoning": false
      }'
      ```
    </CodeGroup>

    To learn more, see [`parse_reasoning`](/docs/guides/capabilities/reasoning#parse_reasoning).
  </Step>

  <Step title="Include Reasoning" icon="dot">
    By default, reasoning's included. To exclude it, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "include_reasoning": False,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "include_reasoning": false,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "include_reasoning": false
      }'
      ```
    </CodeGroup>

    To learn more, see [`include_reasoning`](/docs/guides/capabilities/reasoning#include_reasoning).
  </Step>

  <Step title="Set the Reasoning Effort" icon="dot">
    By default, reasoning effort's set to `max`. To set it to `low`, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "reasoning_effort": "low",
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "reasoning_effort": "low",
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "reasoning_effort": "low"
      }'
      ```
    </CodeGroup>

    To learn more, see [`reasoning_effort`](/docs/guides/capabilities/reasoning#reasoning_effort).
  </Step>

  <Step title="Set a Reasoning Budget" icon="dot">
    By default, there's no reasoning budget. To set one (for example, to 10,000 tokens), use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "reasoning_budget": 10000,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "reasoning_budget": 10000,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "reasoning_budget": 10000
      }'
      ```
    </CodeGroup>

    To learn more, see [`reasoning_budget`](/docs/guides/capabilities/reasoning#reasoning_budget).
  </Step>

  <Step title="Clear Reasoning" icon="dot">
    By default, reasoning's preserved. To clear it, use the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-19} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "chat_template_kwargs": {
            "clear_thinking": True,
          },
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14-16} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "chat_template_kwargs": {
          "clear_thinking": true,
        },
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10-12} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "chat_template_kwargs": {
            "clear_thinking": true
          }
      }'
      ```
    </CodeGroup>

    To learn more, see [`clear_thinking`](/docs/guides/capabilities/reasoning#clear_thinking).
  </Step>
</Steps>


# Use GLM-5.3 with FriendliAI
Source: https://friendli.ai/docs/examples/models/zai-glm-5-3/overview

Use Z.ai's GLM-5.3 on FriendliAI. Review the model ID, context window, and pricing, then choose a feature to start building.

GLM-5.3 is Z.ai's latest flagship open-weight model. It uses the same base model as [GLM-5.2](/docs/examples/models/zai-glm-5-2/overview), but it's better at coding and discovering security vulnerabilities—all for the same price. For simpler tasks, you can set a lower reasoning effort, though you can't turn reasoning off. For even more efficiency, try [GLM-5.3-Flash](/docs/examples/models/zai-glm-5-3-flash/overview).

To use GLM-5.3, see the following model properties:

| Model Property    | Value             |
| ----------------- | ----------------- |
| Model ID          | `zai-org/GLM-5.3` |
| Input Modalities  | Text              |
| Output Modalities | Text              |
| Context Window    | 1,048,576 tokens  |
| Max Output        | 131,072 tokens    |

To estimate costs, see how tokens are priced:

| Pricing      | Cost per 1M tokens |
| ------------ | ------------------ |
| Input        | \$1.40             |
| Cached Input | \$0.26             |
| Output       | \$4.40             |

## Choose a Model Feature

Choose what you'd like to do with the model:

<Card title="Send Requests" icon="arrow-left-arrow-right" href="/docs/examples/models/zai-glm-5-3/send-requests">
  Connect to GLM-5.3 with the OpenAI-compatible Chat Completions API.
</Card>

<Card title="Control Reasoning" icon="brain" href="/docs/examples/models/zai-glm-5-3/control-reasoning">
  Edit GLM-5.3's reasoning parameters.
</Card>

## Other Resources

<CardGroup>
  <Card title="FriendliAI Models" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/friendliai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=466b0a104b441d83d3742ce60c8c2a99" href="https://friendli.ai/models/zai-org/GLM-5.3" />

  <Card title="Z.ai Docs" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/zai.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=0e59d2daee9e670622fb06169d4e4750" href="https://docs.z.ai/guides/llm/glm-5.3" />

  <Card title="Hugging Face Models" icon="https://mintcdn.com/friendliai/XeTFvlau9QViw0uB/static/images/model-providers/hugging-face.svg?fit=max&auto=format&n=XeTFvlau9QViw0uB&q=85&s=bc3d036a5f064b9b7b2087ca2528729a" href="https://huggingface.co/zai-org/GLM-5.3" />
</CardGroup>


# Send Requests to GLM-5.3
Source: https://friendli.ai/docs/examples/models/zai-glm-5-3/send-requests

Send your first request to GLM-5.3 with the OpenAI-compatible Chat Completions API using Python, JavaScript, or cURL.

Use the OpenAI-compatible Chat Completions API to connect to GLM-5.3. You can use the OpenAI Python or JavaScript SDK, or send requests directly with cURL.

If you need a new API key, [create one](/docs/examples/models/overview#create-an-api-key).

## Send a Request

<Steps>
  <Step title="Set Up Your API Key">
    In your terminal, run the following command:

    ```bash theme={null}
    export FRIENDLIAI_API_KEY="<FRIENDLIAI_API_KEY>"
    ```

    Replace `<FRIENDLIAI_API_KEY>` with your API key.
  </Step>

  <Step title="Install the OpenAI SDK">
    In your terminal, run the following command:

    <CodeGroup>
      ```bash OpenAI Python SDK theme={null}
      pip install openai
      ```

      ```bash OpenAI JavaScript SDK theme={null}
      npm install openai
      ```
    </CodeGroup>
  </Step>

  <Step title="Send Your Request to FriendliAI">
    Run the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ]
      }'
      ```
    </CodeGroup>

    <Check>
      FriendliAI returns a response, similar to the following:

      ```json expandable wrap theme={null}
      {
        "id": "chatcmpl-04eb40ba7fc14cf8a51b7869d9cfa76f",
        "object": "chat.completion",
        "choices": [
          {
            "index": 0,
            "message": {
              "role": "assistant",
              "content": "FriendliAI is a generative AI infrastructure company that provides optimized solutions to help businesses deploy large language models faster, more efficiently, and at a lower cost.",
              "reasoning": "...",
              "reasoning_content": "..."
            },
            "logprobs": null,
            "finish_reason": "stop"
          }
        ],
        "created": 1781892285,
        "usage": {
          "completion_tokens": 412,
          "prompt_tokens": 27,
          "prompt_tokens_details": {
            "cached_tokens": 0
          },
          "total_tokens": 439
        },
        "model": "zai-org/GLM-5.3"
      }
      ```
    </Check>
  </Step>
</Steps>


# LangChain Node.js SDK
Source: https://friendli.ai/docs/examples/sdks/langchain/nodejs

Use the LangChain Node.js SDK with FriendliAI for seamless integration and enhanced tool-calling capabilities in your applications.

You can use [**LangChain Node.js SDK**](https://github.com/langchain-ai/langchainjs) to interact with FriendliAI.
This makes migration of existing applications already using LangChain particularly easy.

## How to Use

Before you start, ensure you've already obtained the `API_KEY` from the [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).
FriendliAI is fully compatible with OpenAI, so you can use the `@langchain/openai` package by pointing it at the FriendliAI `baseURL`.

<CodeGroup>
  ```bash npm theme={null}
  npm i @langchain/core @langchain/openai
  ```

  ```bash yarn theme={null}
  yarn add @langchain/core @langchain/openai
  ```

  ```bash pnpm theme={null}
  pnpm add @langchain/core @langchain/openai
  ```
</CodeGroup>

### Instantiation

Now you can instantiate the model object and generate chat completions.
Choose the example for your endpoint type:

<CodeGroup>
  ```js Model APIs theme={null}
  import { ChatOpenAI } from "@langchain/openai";

  const model = new ChatOpenAI({
    model: "zai-org/GLM-5.3",
    apiKey: process.env.API_KEY,
    configuration: {
      baseURL: "https://api.friendli.ai/serverless/v1",
    },
  });
  ```

  ```js Dedicated Endpoints theme={null}
  import { ChatOpenAI } from "@langchain/openai";

  const model = new ChatOpenAI({
    model: "YOUR_ENDPOINT_ID",
    apiKey: process.env.API_KEY,
    configuration: {
      baseURL: "https://api.friendli.ai/dedicated/v1",
    },
  });
  ```

  ```js Dedicated Endpoints with Adapter Route theme={null}
  import { ChatOpenAI } from "@langchain/openai";

  const model = new ChatOpenAI({
    model: "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE",
    apiKey: process.env.API_KEY,
    configuration: {
      baseURL: "https://api.friendli.ai/dedicated/v1",
    },
  });
  ```
</CodeGroup>

### Runnable Interface

FriendliAI supports both synchronous and asynchronous runnable methods to generate a response.

```js theme={null}
import { HumanMessage, SystemMessage } from "@langchain/core/messages";

const messages = [
  new SystemMessage("Translate the following from English into Italian"),
  new HumanMessage("hi!"),
];

const result = await model.invoke(messages);
console.log(result);
```

### Chaining

You can chain the model with a prompt template.
Prompt templates convert raw user input to better input to the LLM.

```javascript theme={null}
import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  ["system", "You are a world class technical documentation writer."],
  ["user", "{input}"],
]);

const chain = prompt.pipe(model);

console.log(
  await chain.invoke({ input: "how can langsmith help with testing?" })
);
```

To get the string value instead of the message, add an output parser to the chain.

```javascript theme={null}
import { StringOutputParser } from "@langchain/core/output_parsers";

const outputParser = new StringOutputParser();

const chain = prompt.pipe(model).pipe(outputParser);

console.log(
  await chain.invoke({ input: "how can langsmith help with testing?" })
);
```

### Tool Calling

Describe tools and their parameters, and let the model return a tool to invoke with the input arguments.
Tool calling is extremely useful for enhancing the model's capability to provide more comprehensive and actionable responses.

#### Define Tools to Use

You can define tools with Zod schemas and use them to generate tool calls.

<CodeGroup>
  ```bash npm theme={null}
  npm i zod
  ```

  ```bash yarn theme={null}
  yarn add zod
  ```

  ```bash pnpm theme={null}
  pnpm add zod
  ```
</CodeGroup>

```js theme={null}
import { tool } from "@langchain/core/tools";
import { z } from "zod";

/**
 * Note that the descriptions here are crucial, as they will be passed along
 * to the model along with the class name.
 */
const calculatorSchema = z.object({
  operation: z
    .enum(["add", "subtract", "multiply", "divide"])
    .describe("The type of operation to execute."),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const calculatorTool = tool(
  async ({ operation, number1, number2 }) => {
    // Functions must return strings
    if (operation === "add") {
      return `${number1 + number2}`;
    } else if (operation === "subtract") {
      return `${number1 - number2}`;
    } else if (operation === "multiply") {
      return `${number1 * number2}`;
    } else if (operation === "divide") {
      return `${number1 / number2}`;
    } else {
      throw new Error("Invalid operation.");
    }
  },
  {
    name: "calculator",
    description: "Can perform mathematical operations.",
    schema: calculatorSchema,
  }
);

console.log(
  await calculatorTool.invoke({ operation: "add", number1: 3, number2: 4 })
);
```

#### Bind Tools to the Model

Now models can generate a tool calling response.

```js theme={null}
const modelWithTools = model.bindTools([calculatorTool]);

const messages = [new HumanMessage("What is 3 * 12? Also, what is 11 + 49?")];

const aiMessage = await modelWithTools.invoke(messages);

console.log(aiMessage);
```

#### Generate a Tool-Assisted Message

Use the tool call results to generate a message.

```js theme={null}
messages.push(aiMessage);

const toolsByName = {
  calculator: calculatorTool,
};

for (const toolCall of aiMessage.tool_calls) {
  const selectedTool = toolsByName[toolCall.name];
  const toolMessage = await selectedTool.invoke(toolCall);
  messages.push(toolMessage);
}

console.log(await modelWithTools.invoke(messages));
```

For more information on how to use tools, check out the [LangChain documentation](https://js.langchain.com/v0.2/docs/how_to/#tools).


# LangChain Python SDK
Source: https://friendli.ai/docs/examples/sdks/langchain/python

Integrate FriendliAI with LangChain Python SDK. Use ChatOpenAI with tool calling and connect to Model APIs or Dedicated Endpoints.

You can use [**LangChain Python SDK**](https://github.com/langchain-ai/langchain) to interact with FriendliAI.
This makes migration of existing applications already using LangChain particularly easy.

## How to Use

Before you start, ensure you've already obtained the `API_KEY` from the [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).
FriendliAI is fully compatible with OpenAI, so you can use the `langchain-openai` package by pointing it at the FriendliAI `baseURL`.

```bash theme={null}
pip install -qU langchain-openai langchain
```

### Instantiation

Now you can instantiate the model object and generate chat completions.
Choose the example for your endpoint type:

<CodeGroup>
  ```python Model APIs theme={null}
  import os
  from langchain_openai import ChatOpenAI

  llm = ChatOpenAI(
      model="zai-org/GLM-5.3",
      base_url="https://api.friendli.ai/serverless/v1",
      api_key=os.environ["API_KEY"],
  )
  ```

  ```python Dedicated Endpoints theme={null}
  import os
  from langchain_openai import ChatOpenAI

  llm = ChatOpenAI(
      model="YOUR_ENDPOINT_ID",
      base_url="https://api.friendli.ai/dedicated/v1",
      api_key=os.environ["API_KEY"],
  )
  ```

  ```python Dedicated Endpoints with Adapter Route theme={null}
  import os
  from langchain_openai import ChatOpenAI

  llm = ChatOpenAI(
      model="YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE",
      base_url="https://api.friendli.ai/dedicated/v1",
      api_key=os.environ["API_KEY"],
  )
  ```
</CodeGroup>

### Runnable Interface

FriendliAI supports both synchronous and asynchronous runnable methods to generate a response.

#### Synchronous Methods

<CodeGroup>
  ```python invoke theme={null}
  result = llm.invoke("Tell me a joke.")
  print(result.content)
  ```

  ```python stream theme={null}
  for chunk in llm.stream("Tell me a joke."):
      print(chunk.content, end="", flush=True)
  ```

  ```python batch theme={null}
  for r in llm.batch(["Tell me a joke.", "Tell me a useless fact."]):
      print(r.content, "\n\n")
  ```
</CodeGroup>

#### Asynchronous Methods

<CodeGroup>
  ```python ainvoke theme={null}
  result = await llm.ainvoke("Tell me a joke.")
  print(result.content)
  ```

  ```python astream theme={null}
  async for chunk in llm.astream("Tell me a joke."):
      print(chunk.content, end="", flush=True)
  ```

  ```python abatch theme={null}
  for r in await llm.abatch(["Tell me a joke.", "Tell me a useless fact."]):
      print(r.content, "\n\n")
  ```
</CodeGroup>

### Chaining

You can [chain](https://python.langchain.com/v0.2/docs/how_to/sequence) the model with a prompt template.
Prompt templates convert raw user input to better input to the LLM.

```python theme={null}
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a world class technical documentation writer."),
    ("user", "{input}")
])

chain = prompt | llm

print(chain.invoke({"input": "how can langsmith help with testing?"}))
```

To get the string value instead of the message, add an output parser to the chain.

```python theme={null}
from langchain_core.output_parsers import StrOutputParser

output_parser = StrOutputParser()

chain = prompt | llm | output_parser

print(chain.invoke({"input": "how can langsmith help with testing?"}))
```

### Tool Calling

Describe tools and their parameters, and let the model return a tool to invoke with the input arguments.
Tool calling is extremely useful for enhancing the model's capability to provide more comprehensive and actionable responses.

#### Define Tools to Use

Use the `@tool` decorator to define a tool.
If you set `parse_docstring=True`, the tool will parse the docstring to extract the information of arguments.

<CodeGroup>
  ```python Default theme={null}
  from langchain_core.tools import tool

  @tool
  def add(a: int, b: int) -> int:
      """Adds a and b."""
      return a + b

  @tool
  def multiply(a: int, b: int) -> int:
      """Multiplies a and b."""
      return a * b

  tools = [add, multiply]
  ```

  ```python Parse Docstring theme={null}
  from langchain_core.tools import tool

  @tool(parse_docstring=True)
  def add(a: int, b: int) -> int:
      """Adds a and b.

      Args:
          a: The first integer.
          b: The second integer.
      """
      return a + b

  @tool(parse_docstring=True)
  def multiply(a: int, b: int) -> int:
      """Multiplies a and b.

      Args:
          a: The first integer.
          b: The second integer.
      """
      return a * b

  tools = [add, multiply]
  ```
</CodeGroup>

#### Bind Tools to the Model

Now models can generate a tool calling response.

```python theme={null}
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="zai-org/GLM-5.3",
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["API_KEY"],
)

llm_with_tools = llm.bind_tools(tools)

query = "What is 3 * 12? Also, what is 11 + 49?"

print(llm_with_tools.invoke(query).tool_calls)
```

#### Generate a Tool-Assisted Message

Use the tool call results to generate a message.

```python theme={null}
from langchain_core.messages import HumanMessage, ToolMessage

messages = [HumanMessage(query)]
ai_msg = llm_with_tools.invoke(messages)
messages.append(ai_msg)

for tool_call in ai_msg.tool_calls:
    selected_tool = {"add": add, "multiply": multiply}[tool_call["name"].lower()]
    tool_output = selected_tool.invoke(tool_call["args"])
    messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))

print(llm_with_tools.invoke(messages))
```

For more information on how to use tools, check out the [LangChain documentation](https://python.langchain.com/v0.2/docs/how_to/#tools).


# Linkup
Source: https://friendli.ai/docs/examples/sdks/linkup

Find and access high-quality web content using the Linkup API, integrated with Friendli Model APIs for seamless interaction.

**Linkup** provides real-time web search capabilities. With Linkup integration in FriendliAI, you can easily enhance your AI applications with up-to-date facts, recent events, and current information that goes beyond what your model was trained on.

You can use Linkup's real-time web search through Friendli Model APIs with just a few simple steps.

## How to Use

### For Playground Testing

1. Create an account at [**Friendli Suite**](https://friendli.ai/suite).
2. Go to **Model APIs** from your Project and click the **Try** button to open the playground.
3. In the playground, open the **Tools** panel and select **Search the web (Linkup)** to test the integration.

<RoundedBorderBox>
  <img alt="Linkup Integrated Playground" />
</RoundedBorderBox>

## Notes & Caveats

* Linkup and FriendliAI both have rate limits. Handle retries/backoff accordingly.
* Keep API keys and tokens secret (use environment variables or secret managers).


# LiteLLM
Source: https://friendli.ai/docs/examples/sdks/litellm

Use LiteLLM with FriendliAI to call Model APIs and Dedicated Endpoints. Includes setup, model selection, and streaming examples.

You can use [**LiteLLM**](https://github.com/BerriAI/litellm) to interact with FriendliAI. If you already use LiteLLM, you can migrate your existing applications with minimal changes.

## How to Use

Before you start, get your `FRIENDLIAI_API_KEY` from [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys). Then add the `friendliai` prefix to your endpoint name in the `model` parameter.

### Chat Completion

Choose the example for your endpoint type. For Model APIs, you can specify one of the [available models](https://friendli.ai/models?products=SERVERLESS).

<CodeGroup>
  ```python Model APIs theme={null}
  import os
  from litellm import completion

  os.environ['FRIENDLIAI_API_KEY'] = "YOUR_API_KEY"

  response = completion(
      model="friendliai/zai-org/GLM-5.3",
      messages=[
          {"role": "user", "content": "hello from litellm"}
      ],
  )
  print(response)
  ```

  ```python Dedicated Endpoints theme={null}
  import os
  from litellm import completion

  os.environ['FRIENDLIAI_API_KEY'] = "YOUR_API_KEY"
  os.environ['FRIENDLI_API_BASE'] = "https://api.friendli.ai/dedicated/v1"

  response = completion(
      model="friendliai/YOUR_ENDPOINT_ID",
      messages=[
          {"role": "user", "content": "hello from litellm"}
      ],
  )
  print(response)
  ```

  ```python Dedicated Endpoints with Adapter Route theme={null}
  import os
  from litellm import completion

  os.environ['FRIENDLIAI_API_KEY'] = "YOUR_API_KEY"
  os.environ['FRIENDLI_API_BASE'] = "https://api.friendli.ai/dedicated/v1"

  response = completion(
      model="friendliai/YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE",
      messages=[
          {"role": "user", "content": "hello from litellm"}
      ],
  )
  print(response)
  ```
</CodeGroup>

### Chat Completion - Streaming

<CodeGroup>
  ```python Model APIs theme={null}
  import os
  from litellm import completion

  os.environ['FRIENDLIAI_API_KEY'] = "YOUR_API_KEY"

  response = completion(
      model="friendliai/zai-org/GLM-5.3",
      messages=[
          {"role": "user", "content": "hello from litellm"}
      ],
      stream=True
  )

  for chunk in response:
      print(chunk)
  ```

  ```python Dedicated Endpoints theme={null}
  import os
  from litellm import completion

  os.environ['FRIENDLIAI_API_KEY'] = "YOUR_API_KEY"
  os.environ['FRIENDLI_API_BASE'] = "https://api.friendli.ai/dedicated/v1"

  response = completion(
      model="friendliai/YOUR_ENDPOINT_ID",
      messages=[
          {"role": "user", "content": "hello from litellm"}
      ],
      stream=True
  )

  for chunk in response:
      print(chunk)
  ```

  ```python Dedicated Endpoints with Adapter Route theme={null}
  import os
  from litellm import completion

  os.environ['FRIENDLIAI_API_KEY'] = "YOUR_API_KEY"
  os.environ['FRIENDLI_API_BASE'] = "https://api.friendli.ai/dedicated/v1"

  response = completion(
      model="friendliai/YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE",
      messages=[
          {"role": "user", "content": "hello from litellm"}
      ],
      stream=True
  )

  for chunk in response:
      print(chunk)
  ```
</CodeGroup>


# LlamaIndex
Source: https://friendli.ai/docs/examples/sdks/llamaindex

Integrate FriendliAI with LlamaIndex. Use the Friendli LLM class for chat completions and text completions with sync, async, and streaming support.

You can use [**LlamaIndex**](https://github.com/run-llama/llama_index) to interact with FriendliAI.
This makes migration of existing applications already using LlamaIndex particularly easy.

## How to Use

Before you start, ensure you've already obtained the `API_KEY` from the [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).

```bash theme={null}
pip install llama-index llama-index-llms-friendli
```

### Instantiation

Now you can instantiate the model object and generate chat completions.
If you don't specify a model, the default model (i.e. `zai-org/GLM-5.3`) is used.

```python theme={null}
import os
from llama_index.llms.friendli import Friendli

os.environ['API_KEY'] = "YOUR_API_KEY"

llm = Friendli(model="zai-org/GLM-5.3")
```

### Chat Completion

Generate a response from a given conversation.

<CodeGroup>
  ```python Default theme={null}
  from llama_index.core.llms import ChatMessage, MessageRole

  message = ChatMessage(role=MessageRole.USER, content="Tell me a joke.")
  resp = llm.chat([message])

  print(resp)
  ```

  ```python Streaming theme={null}
  from llama_index.core.llms import ChatMessage, MessageRole

  message = ChatMessage(role=MessageRole.USER, content="Tell me a joke.")
  resp = llm.stream_chat([message])

  for r in resp:
      print(r.delta, end="")
  ```

  ```python Async theme={null}
  from llama_index.core.llms import ChatMessage, MessageRole

  message = ChatMessage(role=MessageRole.USER, content="Tell me a joke.")
  resp = await llm.achat([message])

  print(resp)
  ```

  ```python Async Streaming theme={null}
  from llama_index.core.llms import ChatMessage, MessageRole

  message = ChatMessage(role=MessageRole.USER, content="Tell me a joke.")
  resp = await llm.astream_chat([message])

  async for r in resp:
      print(r.delta, end="")
  ```
</CodeGroup>

### Completion

Generate a response from a given prompt.

<CodeGroup>
  ```python Default theme={null}
  prompt = "Draft a cover letter for a role in software engineering."
  resp = llm.complete(prompt)

  print(resp)
  ```

  ```python Streaming theme={null}
  prompt = "Draft a cover letter for a role in software engineering."
  resp = llm.stream_complete(prompt)

  for r in resp:
      print(r.delta, end="")
  ```

  ```python Async theme={null}
  prompt = "Draft a cover letter for a role in software engineering."
  resp = await llm.acomplete(prompt)

  print(resp)
  ```

  ```python Async Streaming theme={null}
  prompt = "Draft a cover letter for a role in software engineering."
  resp = await llm.astream_complete(prompt)

  async for r in resp:
      print(r.delta, end="")
  ```
</CodeGroup>


# OpenAI Node.js SDK
Source: https://friendli.ai/docs/examples/sdks/openai/nodejs

Use the OpenAI Node.js SDK with FriendliAI endpoints. Migrate existing Node.js apps by changing the base URL. Covers chat, streaming, and tool calls.

You can use [**OpenAI Node.js SDK**](https://github.com/openai/openai-node) to interact with FriendliAI.
This makes migration of existing applications already using OpenAI particularly easy.

## How to Use

Before you start, ensure the `baseURL` and `apiKey` refer to FriendliAI.
FriendliAI is fully compatible with the OpenAI SDK, so you can follow the examples below.

Choose one of the [available models](https://friendli.ai/models?products=SERVERLESS) for the `model` parameter.

<CodeGroup>
  ```bash npm theme={null}
  npm i openai
  ```

  ```bash yarn theme={null}
  yarn add openai
  ```

  ```bash pnpm theme={null}
  pnpm add openai
  ```
</CodeGroup>

### Chat Completion

Chat completion API that generates a response from a given conversation.

Choose the example that best fits your needs:

<CodeGroup>
  ```ts Default theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.API_KEY,
  });

  async function main() {
    const completion = await client.chat.completions.create({
      model: "zai-org/GLM-5.3",
      messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "Hello!" },
      ],
    });

    console.log(completion.choices[0]);
  }

  main();
  ```

  ```ts Streaming theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.API_KEY,
  });

  async function main() {
    const completion = await client.chat.completions.create({
      model: "zai-org/GLM-5.3",
      messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "Hello!" },
      ],
      stream: true,
    });

    for await (const chunk of completion) {
      console.log(chunk.choices[0].delta.content);
    }
  }

  main();
  ```

  ```ts Functions theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.API_KEY,
  });

  async function main() {
    const messages = [
      { role: "user", content: "What's the weather like in Boston today?" },
    ];
    const tools = [
      {
        type: "function",
        function: {
          name: "get_current_weather",
          description: "Get the current weather in a given location",
          parameters: {
            type: "object",
            properties: {
              location: {
                type: "string",
                description: "The city and state, e.g. San Francisco, CA",
              },
              unit: { type: "string", enum: ["celsius", "fahrenheit"] },
            },
            required: ["location"],
          },
        },
      },
    ];

    const completion = await client.chat.completions.create({
      model: "zai-org/GLM-5.3",
      messages: messages,
      tools: tools,
      tool_choice: "auto",
    });

    console.log(completion);
  }

  main();
  ```

  ```ts Logprobs theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.API_KEY,
  });

  async function main() {
    const completion = await client.chat.completions.create({
      model: "zai-org/GLM-5.3",
      messages: [{ role: "user", content: "Hello!" }],
      logprobs: true,
      top_logprobs: 2,
    });

    console.log(completion.choices[0].message);
    console.log(completion.choices[0].logprobs);
  }

  main();
  ```
</CodeGroup>


# OpenAI Python SDK
Source: https://friendli.ai/docs/examples/sdks/openai/python

Use the OpenAI Python SDK with FriendliAI endpoints. Migrate existing Python apps by changing the base URL. Covers chat, streaming, and tool calling.

You can use [**OpenAI Python SDK**](https://github.com/openai/openai-python) to interact with FriendliAI.
This makes migration of existing applications already using OpenAI particularly easy.

## How to Use

Before you start, ensure the `base_url` and `api_key` refer to FriendliAI.
FriendliAI is fully compatible with the OpenAI SDK, so you can follow the examples below.

Choose one of the [available models](https://friendli.ai/models?products=SERVERLESS) for the `model` parameter.

```bash theme={null}
pip install -qU openai
```

### Chat Completion

Chat completion API that generates a response from a given conversation.

Choose the example that best fits your needs:

<CodeGroup>
  ```python Default theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/serverless/v1",
      api_key=os.environ.get("API_KEY")
  )

  completion = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Hello!"}
      ]
  )

  print(completion.choices[0].message)
  ```

  ```python Streaming theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/serverless/v1",
      api_key=os.environ.get("API_KEY")
  )

  completion = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Hello!"}
      ],
      stream=True
  )

  for chunk in completion:
      print(chunk.choices[0].delta)
  ```

  ```python Functions theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/serverless/v1",
      api_key=os.environ.get("API_KEY")
  )

  tools = [
      {
          "type": "function",
          "function": {
              "name": "get_current_weather",
              "description": "Get the current weather in a given location",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The city and state, e.g. San Francisco, CA",
                      },
                      "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                  },
                  "required": ["location"],
              },
          }
      }
  ]

  completion = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      messages=[
          {"role": "user", "content": "What's the weather like in Boston today?"}
      ],
      tools=tools,
      tool_choice="auto"
  )

  print(completion)
  ```

  ```python Logprobs theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/serverless/v1",
      api_key=os.environ.get("API_KEY")
  )

  completion = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      messages=[
          {"role": "user", "content": "Hello!"}
      ],
      logprobs=True,
      top_logprobs=2
  )

  print(completion.choices[0].message)
  print(completion.choices[0].logprobs)
  ```
</CodeGroup>


# Use Your SDK with FriendliAI
Source: https://friendli.ai/docs/examples/sdks/overview

Browse FriendliAI SDK integrations for OpenAI, LangChain, LlamaIndex, LiteLLM, Vercel AI SDK, and Weaviate. Pick the SDK that fits your stack.

FriendliAI gives you flexible, powerful tools to integrate AI models into your projects. FriendliAI supports a variety of popular SDKs and frameworks, making it easy to add FriendliAI's capabilities to your existing workflows and applications.

Integration options include LiteLLM for unified LLM interactions, Vercel AI SDK for web application development, LangChain for building AI-driven applications, and an OpenAI-compatible API for developers familiar with OpenAI's interface. These integrations let you use FriendliAI's AI models across many use cases—from simple chat apps to complex AI systems—without changing your existing tools or workflow.

## Create an API Key

<Steps>
  <Step title="Create a Friendli Suite Account">
    If you haven't already, [sign up](https://auth.friendli.ai/sign-up) for an account.
  </Step>

  <Step title="Sign In">
    Then, [sign in](https://auth.friendli.ai). Friendli Suite opens your dashboard.
  </Step>

  <Step title="Create a FriendliAI API Key">
    1. In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**.
    2. In the upper-right corner, click <Icon icon="plus" /> **Create API Key**.
    3. (Optional) Name your API key and set an expiration date.
    4. Click **Create Key**.
    5. Click <Icon icon="copy" /> **Copy**.
  </Step>
</Steps>

## Choose Your SDK

Choose the SDK you want to work with:

<CardGroup>
  <Card title="OpenAI (Python)" href="/docs/examples/sdks/openai/python">
    <img alt="OpenAI" />

    <img alt="OpenAI" />
  </Card>

  <Card title="OpenAI (Node.js)" href="/docs/examples/sdks/openai/nodejs">
    <img alt="OpenAI" />

    <img alt="OpenAI" />
  </Card>

  <Card title="LangChain (Python)" href="/docs/examples/sdks/langchain/python">
    <img alt="LangChain" />
  </Card>

  <Card title="LangChain (Node.js)" href="/docs/examples/sdks/langchain/nodejs">
    <img alt="LangChain" />
  </Card>

  <Card title="Weaviate (Python)" href="/docs/examples/sdks/weaviate/python">
    <img alt="Weaviate" />
  </Card>

  <Card title="Weaviate (Node.js)" href="/docs/examples/sdks/weaviate/nodejs">
    <img alt="Weaviate" />
  </Card>

  <Card title="Vercel AI SDK" href="/docs/examples/sdks/vercel-ai">
    <img alt="Vercel" />

    <img alt="Vercel" />
  </Card>

  <Card title="LlamaIndex" href="/docs/examples/sdks/llamaindex">
    <img alt="LlamaIndex" />
  </Card>

  <Card title="LiteLLM" href="/docs/examples/sdks/litellm">
    <img alt="LiteLLM" />

    <img alt="LiteLLM" />
  </Card>

  <Card title="Linkup" href="/docs/examples/sdks/linkup">
    <img alt="Linkup" />

    <img alt="Linkup" />
  </Card>
</CardGroup>


# Vercel AI SDK
Source: https://friendli.ai/docs/examples/sdks/vercel-ai

Use the Vercel AI SDK with FriendliAI for streaming chat UIs in Next.js and React. Connect to Model APIs or Dedicated Endpoints with minimal setup.

You can use [**Vercel AI SDK**](https://sdk.vercel.ai) to interact with FriendliAI.
This makes migration of existing applications already using Vercel AI SDK particularly easy.

## How to Use

Before you start, ensure you've already obtained the `API_KEY` from the [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).

<CodeGroup>
  ```bash npm theme={null}
  npm i ai @friendliai/ai-provider
  ```

  ```bash yarn theme={null}
  yarn add ai @friendliai/ai-provider
  ```

  ```bash pnpm theme={null}
  pnpm add ai @friendliai/ai-provider
  ```
</CodeGroup>

### Instantiation

Instantiate your models using a FriendliAI provider instance.
Choose the example for your endpoint type:

<CodeGroup>
  ```ts Model APIs {2-4} theme={null}
  import { friendli } from "@friendliai/ai-provider";

  // Automatically select Model APIs
  const model = friendli("zai-org/GLM-5.3");
  ```

  ```ts Dedicated Endpoints {2-7} theme={null}
  import { createFriendli } from "@friendliai/ai-provider";

  const friendli = createFriendli({
    baseURL: "dedicated",
  });

  // Replace YOUR_ENDPOINT_ID with the ID of your endpoint, e.g. "zbimjgovmlcb"
  const model = friendli("YOUR_ENDPOINT_ID");
  ```
</CodeGroup>

To target a specific endpoint type explicitly, pass `baseURL` when creating the provider with `createFriendli`. Accepted values include `"serverless"`, `"dedicated"`, or a full API URL (for example, `https://api.friendli.ai/serverless/v1`).

### Example: Generating Text

Generate a response with the `generateText` function:

```ts theme={null}
import { friendli } from "@friendliai/ai-provider";
import { generateText } from "ai";

const { text } = await generateText({
  model: friendli("zai-org/GLM-5.3"),
  prompt: "Write a vegetarian lasagna recipe for 4 people.",
});

console.log(text);
```

### Example: Using Enforcing Patterns (Regex)

Specify a specific pattern (e.g., CSV), character sets, or specific language characters (e.g., Korean Hangul characters) for your LLM's output.
Pass the regex as a string via `providerOptions.friendliai.regex`:

```ts {6-10} theme={null}
import { friendli } from "@friendliai/ai-provider";
import { generateText } from "ai";

const { text } = await generateText({
  model: friendli("zai-org/GLM-5.3"),
  prompt: "조선 왕조의 첫번째 왕은 누구입니까 (Who is the first king of the Joseon Dynasty)?",
  providerOptions: {
    friendliai: {
      regex: "[\n ,.?!0-9\uac00-\ud7af]*",
    },
  },
});

console.log(text);
```

## OpenAI Compatibility

You can also use `@ai-sdk/openai` as the APIs are OpenAI-compatible.

```ts theme={null}
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const friendli = createOpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.API_KEY,
});

const { text } = await generateText({
  model: friendli.chat("zai-org/GLM-5.3"),
  prompt: "Say hello in one short sentence.",
});
```

If you are using Dedicated Endpoints:

```ts {4-5} theme={null}
import { createOpenAI } from "@ai-sdk/openai";

const friendli = createOpenAI({
  baseURL: "https://api.friendli.ai/dedicated/v1",
  apiKey: process.env.API_KEY,
});
```

## Further Resources

* [Implementing a simple streaming chat with Next.js](https://sdk.vercel.ai/examples/next-app/basics/streaming-text-generation)
* [Build a Next.js app with the Vercel AI SDK](https://sdk.vercel.ai/docs/getting-started/nextjs-app-router)
* [Explore the Vercel AI SDK Core Reference](https://sdk.vercel.ai/docs/ai-sdk-core/overview)


# FriendliAI + Weaviate (Node.js)
Source: https://friendli.ai/docs/examples/sdks/weaviate/nodejs

Build RAG apps with FriendliAI and Weaviate in Node.js. Combine vector search with Friendli Engine inference to reduce hallucinations in responses.

Integration with [**Weaviate**](https://github.com/weaviate/weaviate) enables performing Retrieval Augmented Generation (RAG) directly within the Weaviate database.
This combines the power of [**Friendli Engine**](https://friendli.ai/why-friendliai) and Weaviate's efficient storage and fast retrieval capabilities to generate personalized and context-aware responses.

## How to Use

Before you start, ensure you've already obtained the `API_KEY` from the [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).
Also, set up your Weaviate instance following this [guide](https://weaviate.io/developers/weaviate/starter-guides/which-weaviate).
Your Weaviate instance must be configured with the FriendliAI generative AI integration (`generative-friendliai`) module.

<CodeGroup>
  ```bash npm theme={null}
  npm i weaviate-client
  ```

  ```bash yarn theme={null}
  yarn add weaviate-client
  ```

  ```bash pnpm theme={null}
  pnpm add weaviate-client
  ```
</CodeGroup>

### Instantiation

Now you can instantiate a [Weaviate collection](https://weaviate.io/developers/weaviate/manage-data/collections) using the model.
Choose the example for your endpoint type:

You can specify one of the [available models](https://friendli.ai/models?products=SERVERLESS) for the Model APIs.
If you don't specify a model, the default model (i.e. `zai-org/GLM-5.3`) is used.

<CodeGroup>
  ```ts Model APIs theme={null}
  import weaviate from "weaviate-client"

  const client = await weaviate.connectToWeaviateCloud(
    'WEAVIATE_INSTANCE_URL',  // your Weaviate instance URL
    {
      authCredentials: new weaviate.ApiKey('WEAVIATE_INSTANCE_APIKEY'),
      headers: {
        'X-Friendli-Api-Key': process.env.API_KEY,
      }
    }
  )

  await client.collections.create({
    name: 'DemoCollection',
    generative: weaviate.configure.generative.friendliai({
      model: 'zai-org/GLM-5.3'
    }),
    // Additional parameters ...
  });

  client.close()
  ```

  ```ts Dedicated Endpoints theme={null}
  import weaviate from "weaviate-client"

  const client = await weaviate.connectToWeaviateCloud(
    'WEAVIATE_INSTANCE_URL',  // your Weaviate instance URL
    {
      authCredentials: new weaviate.ApiKey('WEAVIATE_INSTANCE_APIKEY'),
      headers: {
        'X-Friendli-Api-Key': process.env.API_KEY,
        "X-Friendli-Baseurl": "https://api.friendli.ai/dedicated",
      }
    }
  )

  await client.collections.create({
    name: 'DemoCollection',
    generative: weaviate.configure.generative.friendliai({
      model: 'YOUR_ENDPOINT_ID'
    }),
    // Additional parameters ...
  });

  client.close()
  ```

  ```ts Dedicated Endpoints with Adapter Route theme={null}
  import weaviate from "weaviate-client"

  const client = await weaviate.connectToWeaviateCloud(
    'WEAVIATE_INSTANCE_URL',  // your Weaviate instance URL
    {
      authCredentials: new weaviate.ApiKey('WEAVIATE_INSTANCE_APIKEY'),
      headers: {
        'X-Friendli-Api-Key': process.env.API_KEY,
        "X-Friendli-Baseurl": "https://api.friendli.ai/dedicated",
      }
    }
  )

  await client.collections.create({
    name: 'DemoCollection',
    generative: weaviate.configure.generative.friendliai({
      model: 'YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE'
    }),
    // Additional parameters ...
  });

  client.close()
  ```
</CodeGroup>

#### Configurable Parameters

Configure the following generative parameters to customize the model behavior.

```ts theme={null}
await client.collections.create({
  name: 'DemoCollection',
  generative: weaviate.configure.generative.friendliai({
    model: 'zai-org/GLM-5.3',
    maxTokens: 500,
    temperature: 0.7,
  }),
  // Additional parameters ...
});
```

### Retrieval Augmented Generation

After configuring Weaviate, perform RAG operations, either with the single prompt or grouped task method.

#### Single Prompt

To generate text for each object in the search results, use the single prompt method.
The example below generates outputs for each of the n search results, where n is specified by the limit parameter.

When creating a single prompt query, use braces `{}` to interpolate the object properties you want Weaviate to pass on to the language model.
For example, to pass on the object's title property, include `{title}` in the query.

```ts theme={null}
let myCollection = client.collections.get('DemoCollection');

const singlePromptResults = await myCollection.generate.nearText(
  ['A holiday film'],
  {
    singlePrompt: `Translate this into French: {title}`,
  },
  {
    limit: 2,
  }
);

for (const obj of singlePromptResults.objects) {
  console.log(obj.properties['title']);
  console.log(`Generated output: ${obj.generated}`);  // Note that the generated output is per object
}
```

#### Grouped Task

To generate one text for the entire set of search results, use the grouped task method.
In other words, when you have n search results, the generative model generates one output for the entire group.

```ts theme={null}
let myCollection = client.collections.get('DemoCollection');
const groupedTaskResults = await myCollection.generate.nearText(
  ['A holiday film'],
  {
    groupedTask: `Write a fun tweet to encourage readers to check out these films.`,
  },
  {
    limit: 2,
  }
);

console.log(`Generated output: ${groupedTaskResults.generated}`);  // Note that the generated output is per query
for (const obj of groupedTaskResults.objects) {
  console.log(obj.properties['title']);
}
```

### Further Resources

Once the integrations are configured at the collection, the data management and search operations in Weaviate work identically to any other collection.
See the following model-agnostic examples:

* [How-to manage data guides show how to perform data operations](https://weaviate.io/developers/weaviate/manage-data/create).
* [How-to search guides show how to perform search operations](https://weaviate.io/developers/weaviate/search/basics).


# FriendliAI + Weaviate (Python)
Source: https://friendli.ai/docs/examples/sdks/weaviate/python

Build RAG apps with FriendliAI and Weaviate in Python. Combine vector search with Friendli Engine inference for context-aware, grounded responses.

Integration with [**Weaviate**](https://github.com/weaviate/weaviate) enables performing Retrieval Augmented Generation (RAG) directly within the Weaviate database.
This combines the power of [**Friendli Engine**](https://friendli.ai/why-friendliai) and Weaviate's efficient storage and fast retrieval capabilities to generate personalized and context-aware responses.

## How to Use

Before you start, ensure you've already obtained the `API_KEY` from the [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).
Also, set up your Weaviate instance following this [guide](https://weaviate.io/developers/weaviate/starter-guides/which-weaviate).
Your Weaviate instance must be configured with the FriendliAI generative AI integration (`generative-friendliai`) module.

```bash theme={null}
pip install -qU weaviate-client
```

### Instantiation

Now you can instantiate a [Weaviate collection](https://weaviate.io/developers/weaviate/manage-data/collections) using the model.
Choose the example for your endpoint type:

You can specify one of the [available models](https://friendli.ai/models?products=SERVERLESS) for the Model APIs.
If you don't specify a model, the default model (i.e. `zai-org/GLM-5.3`) is used.

<CodeGroup>
  ```python Model APIs theme={null}
  import weaviate
  import os
  from weaviate.classes.init import Auth
  from weaviate.classes.config import Configure

  headers = {
      "X-Friendli-Api-Key": os.getenv("API_KEY"),
  }

  client = weaviate.connect_to_weaviate_cloud(
      cluster_url=weaviate_url,                     # `weaviate_url`: your Weaviate URL
      auth_credentials=Auth.api_key(weaviate_key),  # `weaviate_key`: your Weaviate API key
      headers=headers
  )

  client.collections.create(
      "DemoCollection",
      generative_config=Configure.Generative.friendliai(
          model = "zai-org/GLM-5.3",
      )
      # Additional parameters not shown
  )

  client.close()
  ```

  ```python Dedicated Endpoints theme={null}
  import weaviate
  import os
  from weaviate.classes.init import Auth
  from weaviate.classes.config import Configure

  headers = {
      "X-Friendli-Api-Key": os.getenv("API_KEY"),
      "X-Friendli-Baseurl": "https://api.friendli.ai/dedicated",
  }

  client = weaviate.connect_to_weaviate_cloud(
      cluster_url=weaviate_url,                     # `weaviate_url`: your Weaviate URL
      auth_credentials=Auth.api_key(weaviate_key),  # `weaviate_key`: your Weaviate API key
      headers=headers
  )

  client.collections.create(
      "DemoCollection",
      generative_config=Configure.Generative.friendliai(
          model = "YOUR_ENDPOINT_ID",
      )
      # Additional parameters not shown
  )

  client.close()
  ```

  ```python Dedicated Endpoints with Adapter Route theme={null}
  import weaviate
  import os
  from weaviate.classes.init import Auth
  from weaviate.classes.config import Configure

  headers = {
      "X-Friendli-Api-Key": os.getenv("API_KEY"),
      "X-Friendli-Baseurl": "https://api.friendli.ai/dedicated",
  }

  client = weaviate.connect_to_weaviate_cloud(
      cluster_url=weaviate_url,                     # `weaviate_url`: your Weaviate URL
      auth_credentials=Auth.api_key(weaviate_key),  # `weaviate_key`: your Weaviate API key
      headers=headers
  )

  client.collections.create(
      "DemoCollection",
      generative_config=Configure.Generative.friendliai(
          model = "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE",
      )
      # Additional parameters not shown
  )

  client.close()
  ```
</CodeGroup>

#### Configurable Parameters

Configure the following generative parameters to customize the model behavior.

```python theme={null}
from weaviate.classes.config import Configure

client.collections.create(
    "DemoCollection",
    generative_config=Configure.Generative.friendliai(
        # These parameters are optional
        model = "zai-org/GLM-5.3",
        max_tokens = 500,
        temperature = 0.7,
    )
)
```

### Retrieval Augmented Generation

After configuring Weaviate, perform RAG operations, either with the single prompt or grouped task method.

#### Single Prompt

To generate text for each object in the search results, use the single prompt method.
The example below generates outputs for each of the n search results, where n is specified by the limit parameter.

When creating a single prompt query, use braces `{}` to interpolate the object properties you want Weaviate to pass on to the language model.
For example, to pass on the object's title property, include `{title}` in the query.

```python theme={null}
collection = client.collections.get("DemoCollection")

response = collection.generate.near_text(
    query="A holiday film",  # The model provider integration will automatically vectorize the query
    single_prompt="Translate this into French: {title}",
    limit=2
)

for obj in response.objects:
    print(obj.properties["title"])
    print(f"Generated output: {obj.generated}")  # Note that the generated output is per object
```

#### Grouped Task

To generate one text for the entire set of search results, use the grouped task method.
In other words, when you have n search results, the generative model generates one output for the entire group.

```python theme={null}
collection = client.collections.get("DemoCollection")

response = collection.generate.near_text(
    query="A holiday film",  # The model provider integration will automatically vectorize the query
    grouped_task="Write a fun tweet to encourage readers to check out these films.",
    limit=2
)

print(f"Generated output: {response.generated}")  # Note that the generated output is per query
for obj in response.objects:
    print(obj.properties["title"])
```

### Further Resources

Once the integrations are configured at the collection, the data management and search operations in Weaviate work identically to any other collection.
See the following model-agnostic examples:

* [How-to manage data guides show how to perform data operations](https://weaviate.io/developers/weaviate/manage-data/create).
* [How-to search guides show how to perform search operations](https://weaviate.io/developers/weaviate/search/basics).


# Reasoning
Source: https://friendli.ai/docs/guides/capabilities/reasoning

Control how a model reasons on FriendliAI. Compare controllable and always-on reasoning models, then set the reasoning effort, budget, and parsing behavior.

Some models can generate a chain of thought before they respond. These models use this intermediate step to split a prompt into more manageable parts, test different approaches, and arrive at a conclusion with more rigor.

This *reasoning*—sometimes called *thinking*—improves response quality, especially for complex tasks. However, it also increases the model's token usage and its response times.

Depending on the model, you may be able to control whether and how the model reasons. If the model supports it, FriendliAI can also parse reasoning content for you, such that you can clearly distinguish reasoning from the rest of the response.

## Reasoning Model Types

There are two types of reasoning models:

* **Controllable reasoning model**: With a controllable reasoning model, you use a parameter to control whether the model reasons. If you turn reasoning on, the model generates a chain of thought before it responds. If you turn it off, the model responds without generating one.

  To learn more, see [`enable_thinking`](#enable_thinking).

* **Always-on reasoning model**: An always-on reasoning model always reasons, meaning it generates a chain of thought before every response. You can't turn reasoning off.

By contrast, non-reasoning models can't generate chains of thought, meaning they always respond without generating them.

## Reasoning Parameters

You can control whether and how a model reasons with reasoning parameters, which you pass as either chat template keyword arguments or request body parameters. How you pass each depends on the parameter.

To learn more, see [Control a Capability](/docs/guides/introduction/get-started#control-a-capability).

Depending on the model, you can control reasoning with some or all of the following parameters:

| Parameter           | Data Type | Pass As                        |
| ------------------- | --------- | ------------------------------ |
| `enable_thinking`   | boolean   | Chat template keyword argument |
| `parse_reasoning`   | boolean   | Request body parameter         |
| `include_reasoning` | boolean   | Request body parameter         |
| `reasoning_effort`  | string    | Request body parameter         |
| `reasoning_budget`  | integer   | Request body parameter         |
| `clear_thinking`    | boolean   | Chat template keyword argument |

To see some specific examples, see [Choose a Model](/docs/examples/models/overview#choose-a-model), choose one, and see its Control Reasoning page.

### `enable_thinking`

If the model's controllable, you can turn reasoning on and off by setting the `enable_thinking` parameter to `true` or `false`, respectively. It controls whether the model reasons before it generates a response.

For complex tasks, where response quality matters, turn reasoning on. For simpler tasks, you can try turning it off to reduce token usage and response times.

To turn reasoning on for a controllable reasoning model, such as `zai-org/GLM-5.2`, you can use the following code:

<CodeGroup>
  ```python OpenAI Python SDK wrap highlight={15-19} theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLIAI_API_KEY"],
  )

  completion = client.chat.completions.create(
    model="zai-org/GLM-5.2",
    messages=[
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."},
    ],
    extra_body={
      "chat_template_kwargs": {
        "enable_thinking": True,
      },
    },
  )

  print(completion.choices[0].message)
  ```

  ```javascript OpenAI JavaScript SDK wrap highlight={14-16} theme={null}
  import OpenAI from "openai"

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.FRIENDLIAI_API_KEY,
  })

  const completion = await client.chat.completions.create({
    model: "zai-org/GLM-5.2",
    messages: [
      { role: "system", content: "You are a friendly assistant." },
      { role: "user", content: "Describe FriendliAI in one sentence." },
    ],
    "chat_template_kwargs": {
      "enable_thinking": true,
    },
  })

  console.log(completion.choices[0].message)
  ```

  ```bash cURL wrap highlight={10-12} theme={null}
  curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
    -d '{
      "model": "zai-org/GLM-5.2",
      "messages": [
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Describe FriendliAI in one sentence."}
      ],
      "chat_template_kwargs": {
        "enable_thinking": true
      }
  }'
  ```
</CodeGroup>

### `parse_reasoning`

If the model supports it, you can turn parsing on and off by setting the `parse_reasoning` parameter to `true` or `false`, respectively. It controls whether reasoning's parsed in the response.

Without parsing, a reasoning model returns its chain of thought alongside the rest of the response, and usually separates the two with some kind of delimiter. FriendliAI's parser uses this delimiter to separate them; if there's no delimiter, the parser can't separate the two. If you turn parsing on, reasoning's parsed into both `reasoning` and `reasoning_content`, separate from `content`.

To turn parsing on, you can use the following code:

<CodeGroup>
  ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLIAI_API_KEY"],
  )

  completion = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."},
    ],
    extra_body={
      "parse_reasoning": True,
    },
  )

  print(completion.choices[0].message)
  ```

  ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
  import OpenAI from "openai"

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.FRIENDLIAI_API_KEY,
  })

  const completion = await client.chat.completions.create({
    model: "zai-org/GLM-5.3",
    messages: [
      { role: "system", content: "You are a friendly assistant." },
      { role: "user", content: "Describe FriendliAI in one sentence." },
    ],
    "parse_reasoning": true,
  })

  console.log(completion.choices[0].message)
  ```

  ```bash cURL wrap highlight={10} theme={null}
  curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
    -d '{
      "model": "zai-org/GLM-5.3",
      "messages": [
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Describe FriendliAI in one sentence."}
      ],
      "parse_reasoning": true
  }'
  ```
</CodeGroup>

### `include_reasoning`

If you turn parsing on, you can include or exclude reasoning by setting the `include_reasoning` parameter to `true` or `false`, respectively. It controls whether the reasoning's included in the response.

If you need reasoning content (for example, to display to users), include it in the response. If you don't, exclude it. Note that your choice doesn't change the number of tokens the model uses to generate a response. It's the same, whether or not you include reasoning.

To include reasoning, you can use the following code:

<CodeGroup>
  ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLIAI_API_KEY"],
  )

  completion = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."},
    ],
    extra_body={
      "include_reasoning": True,
    },
  )

  print(completion.choices[0].message)
  ```

  ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
  import OpenAI from "openai"

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.FRIENDLIAI_API_KEY,
  })

  const completion = await client.chat.completions.create({
    model: "zai-org/GLM-5.3",
    messages: [
      { role: "system", content: "You are a friendly assistant." },
      { role: "user", content: "Describe FriendliAI in one sentence." },
    ],
    "include_reasoning": true,
  })

  console.log(completion.choices[0].message)
  ```

  ```bash cURL wrap highlight={10} theme={null}
  curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
    -d '{
      "model": "zai-org/GLM-5.3",
      "messages": [
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Describe FriendliAI in one sentence."}
      ],
      "include_reasoning": true
  }'
  ```
</CodeGroup>

### `reasoning_effort`

If the model supports it, you can set the model's reasoning effort by setting the `reasoning_effort` parameter to a value the model supports. It controls the model's reasoning effort for each response.

The greater the reasoning effort, the longer the chain of thought. On complex tasks, these longer chains of thought improve response quality. Note that it also increases the number of completion tokens and response times.

To set the reasoning effort to a supported value, such as `high`, you can use the following code:

<CodeGroup>
  ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLIAI_API_KEY"],
  )

  completion = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."},
    ],
    extra_body={
      "reasoning_effort": "high",
    },
  )

  print(completion.choices[0].message)
  ```

  ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
  import OpenAI from "openai"

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.FRIENDLIAI_API_KEY,
  })

  const completion = await client.chat.completions.create({
    model: "zai-org/GLM-5.3",
    messages: [
      { role: "system", content: "You are a friendly assistant." },
      { role: "user", content: "Describe FriendliAI in one sentence." },
    ],
    "reasoning_effort": "high",
  })

  console.log(completion.choices[0].message)
  ```

  ```bash cURL wrap highlight={10} theme={null}
  curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
    -d '{
      "model": "zai-org/GLM-5.3",
      "messages": [
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Describe FriendliAI in one sentence."}
      ],
      "reasoning_effort": "high"
  }'
  ```
</CodeGroup>

### `reasoning_budget`

If the model supports it, you can set the model's reasoning budget by setting the `reasoning_budget` parameter to an integer. It controls the model's reasoning budget for each response.

The reasoning budget is the maximum number of reasoning tokens the model can use to generate a response. When the model reaches the budget, it stops thinking—sometimes mid-thought. For this reason, choose a budget thoughtfully.

To set the reasoning budget to 10,000 tokens, you can use the following code:

<CodeGroup>
  ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLIAI_API_KEY"],
  )

  completion = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."},
    ],
    extra_body={
      "reasoning_budget": 10000,
    },
  )

  print(completion.choices[0].message)
  ```

  ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
  import OpenAI from "openai"

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.FRIENDLIAI_API_KEY,
  })

  const completion = await client.chat.completions.create({
    model: "zai-org/GLM-5.3",
    messages: [
      { role: "system", content: "You are a friendly assistant." },
      { role: "user", content: "Describe FriendliAI in one sentence." },
    ],
    "reasoning_budget": 10000,
  })

  console.log(completion.choices[0].message)
  ```

  ```bash cURL wrap highlight={10} theme={null}
  curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
    -d '{
      "model": "zai-org/GLM-5.3",
      "messages": [
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Describe FriendliAI in one sentence."}
      ],
      "reasoning_budget": 10000
  }'
  ```
</CodeGroup>

### `clear_thinking`

If the model supports it, you can clear or preserve reasoning by setting the `clear_thinking` parameter to `true` or `false`, respectively. It controls whether reasoning's cleared from the model's context window.

If reasoning matters to your workflow or use case, you can preserve it and add it to the model's context window. Otherwise, you can clear it. Note that preserving reasoning increases token usage.

To clear reasoning, you can use the following code:

<CodeGroup>
  ```python OpenAI Python SDK wrap highlight={15-19} theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["FRIENDLIAI_API_KEY"],
  )

  completion = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."},
    ],
    extra_body={
      "chat_template_kwargs": {
        "clear_thinking": True,
      },
    },
  )

  print(completion.choices[0].message)
  ```

  ```javascript OpenAI JavaScript SDK wrap highlight={14-16} theme={null}
  import OpenAI from "openai"

  const client = new OpenAI({
    baseURL: "https://api.friendli.ai/serverless/v1",
    apiKey: process.env.FRIENDLIAI_API_KEY,
  })

  const completion = await client.chat.completions.create({
    model: "zai-org/GLM-5.3",
    messages: [
      { role: "system", content: "You are a friendly assistant." },
      { role: "user", content: "Describe FriendliAI in one sentence." },
    ],
    "chat_template_kwargs": {
      "clear_thinking": true,
    },
  })

  console.log(completion.choices[0].message)
  ```

  ```bash cURL wrap highlight={10-12} theme={null}
  curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
    -d '{
      "model": "zai-org/GLM-5.3",
      "messages": [
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Describe FriendliAI in one sentence."}
      ],
      "chat_template_kwargs": {
        "clear_thinking": true
      }
  }'
  ```
</CodeGroup>


# Data Privacy and Security
Source: https://friendli.ai/docs/guides/data-handling

Learn how FriendliAI handles your data. Inference requests and responses are never used for training or shared with third parties.

FriendliAI keeps your data private and secure. The contents of your inference requests and responses are processed only to deliver the Friendli Model APIs and Friendli Dedicated Endpoints services—and for no other purpose.

* We do **not** share request or response contents with third parties.
* We do **not** use request or response contents to train models.

<Note>
  For full details on how FriendliAI handles and protects your data, see our [Privacy Policy](https://friendli.ai/privacypolicy).
</Note>


# Autoscaling
Source: https://friendli.ai/docs/guides/dedicated-endpoints/autoscaling

Configure autoscaling for Friendli Dedicated Endpoints to automatically adjust GPU replicas based on traffic and latency thresholds.

Friendli Dedicated Endpoints provide autoscaling that automatically adjusts computational resources based on your traffic patterns, helping you optimize both performance and costs.

<RoundedBorderBox>
  <img alt="Autoscaling Config" />
</RoundedBorderBox>

## How Autoscaling Works

* **Minimum Replicas**:
  * When set to 0, the endpoint enters sleeping status during periods of inactivity, helping to minimize costs.
  * When set to a value greater than 0, the endpoint maintains at least that number of active replicas at all times.
* **Maximum Replicas**: Defines the upper limit of replicas that can be created to handle increased traffic load.
* **Cooldown Period**: Measured in seconds; if no requests are received during this period, the endpoint transitions to sleeping status.

## Scaling Policies

<Danger>
  We highly recommend using the **Default** autoscaling type, as it performs reliably for most workloads.
  Performance degradation or unexpected charges may occur with other configurations if you don't fully understand your workload characteristics.
</Danger>

* **Default** (Recommended): This is the best choice for the majority of users. It operates reliably across most workloads with no configuration required, using our internal expertise to provide a balanced approach to performance and cost.
* **Request count**: This is an advanced option for users who have a deep understanding of their workload characteristics and require granular control over scaling behavior.
  * As you define the number of requests a single worker will handle, cost prediction becomes more straightforward and intuitive.
  * This method can serve as a foundation for implementing your own custom autoscaling logic by dynamically changing the threshold via an API, targeting custom metrics.

## Benefits of Autoscaling

* **Cost Optimization**: Pay only for the resources you need for your workload.
* **Performance Management**: Handle traffic spikes efficiently.
* **Resource Efficiency**: Maintain optimal resource utilization for your workload.


# Endpoints
Source: https://friendli.ai/docs/guides/dedicated-endpoints/endpoints

Configure, monitor, and manage Friendli Dedicated Endpoints. Learn about endpoint configuration options, statuses, and the metrics you can monitor.

An endpoint is a running deployment of your model on a dedicated GPU. This page covers what you can configure on an endpoint and what you can monitor.

## What You Can Configure

When you create or update an endpoint, you can configure the following options:

* **Name**: The name of the endpoint.
* **Model**: The model to serve, from **Hugging Face** or [uploaded models](/docs/guides/dedicated-endpoints/models), with optional [LoRA adapters](/docs/guides/dedicated-endpoints/lora-serving).
* **Instance type**: The GPU type and count for the endpoint.
* **[Scaling configuration](/docs/guides/dedicated-endpoints/autoscaling)**: The range of replicas used to scale with traffic.
* **[Online Quantization](/docs/guides/dedicated-endpoints/online-quantization)**: Improves serving efficiency with FriendliAI's proprietary quantization method. Select off, 8-bit, or 4-bit.
* **[Speculative Decoding](/docs/guides/dedicated-endpoints/speculative-decoding)**: Speeds up generation by drafting candidate tokens and verifying them in parallel, using a draft model or n-gram speculation.
* **Host KV Cache**: Additional host memory for KV cache storage, extending total KV capacity beyond GPU memory limits (may add to startup time).
* **Engine configuration**: Special token handling and maximum batch size.
* **Request logging**: Whether to log request content (default: off).
* **[Reasoning Parser](/docs/guides/capabilities/reasoning#parse_reasoning)**: The default `parse_reasoning` behavior, applied when the argument isn't provided in a request.
* **Custom Chat Template**: A [Jinja](https://jinja.palletsprojects.com/en/stable/) template that overrides the model's default template.
* **[Version comment](/docs/guides/dedicated-endpoints/versioning)**: An optional note describing each deployed version of the endpoint.

## What You Can Monitor

For each endpoint, you can monitor the following:

### Status

An endpoint moves through the following statuses:

* **Initializing** — The endpoint is starting up after creation, across three phases: initializing the GPU, downloading the model, and initializing the engine.
* **Running** — At least one replica is available to serve requests.
* **Updating** — The endpoint is applying a change to its spec.
* **Sleeping** — The endpoint freed its GPUs after the cooldown period with no requests.
* **Waking up** — The endpoint is returning from sleeping to running, doing the same work as initializing. You can trigger a wake-up manually or by sending a request.
* **Terminated** — The endpoint has been terminated.
* **Failed** — The endpoint has failed initialization for some reason.

### Versions

Each endpoint keeps a [deployment history](/docs/guides/dedicated-endpoints/versioning), where every version captures a snapshot of its configuration with a comment. You can compare changes between versions and roll back to a previous one without downtime.

### Metrics

The **Metrics** tab provides charts for monitoring performance and usage:

* Processed requests
* Processed tokens
* Time to first token
* Time per output token
* Request latency
* Number of replicas
* Cost per million tokens
* Overall traffic (2xx, 4xx, and 5xx responses)

<Note>
  Some charts may not be available depending on the model type.
</Note>

### KV Cache Size

You can see the endpoint's current KV cache size. To make it larger, enable the **Host KV Cache** option, which extends total KV capacity beyond GPU memory limits.


# Dedicated Endpoints FAQ and Troubleshooting
Source: https://friendli.ai/docs/guides/dedicated-endpoints/faq

Answers to common questions about Friendli Dedicated Endpoints, including model compatibility, GPU requirements, billing, and troubleshooting tips.

## Integrations

<AccordionGroup>
  <Accordion title="How do I integrate a Hugging Face account?">
    1. Sign in to Hugging Face, then navigate to [Access Tokens](https://huggingface.co/settings/tokens).

    2. Create a new token. You can use a fine-grained token. In this case, make sure the token has view permission for the repository you'd like to use.

    3. Integrate the key in [Friendli Suite > Personal Settings > Integrations](https://friendli.ai/suite/~/setting/integrations).

    <Info>
      If you revoke / invalidate the key, you will have to update the key to avoid disrupting ongoing deployments, or to launch a new inference deployment.
    </Info>
  </Accordion>
</AccordionGroup>

## Using a 3rd-Party Model

<AccordionGroup>
  <Accordion title="How can I use a Hugging Face repository as a model?">
    <img alt="HF Artifact as a Model" />

    * Use the repository id of the model. You can select the entry from the list of autocompleted model repositories.
    * You can select a specific branch, or manually enter a commit hash.
  </Accordion>
</AccordionGroup>

## Format Requirements

<AccordionGroup>
  <Accordion title="What are the format requirements for a model?">
    * A model should be in safetensors format.
    * The model should NOT be nested inside another directory.
    * Including other arbitrary files (that are not in the list) is totally fine. However, those files will not be downloaded nor used.

    | Required | Filename                  | Description                                                                                                          |
    | -------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------- |
    | Yes      | *safetensors*             | Model weight, e.g. model.safetensors. Use model.safetensors.index.json for split safetensors files                   |
    | Yes      | config.json               | Model config that includes the architecture. ([Supported Models on FriendliAI](https://friendli.ai/models))          |
    | No       | tokenizer.json            | Tokenizer for the model                                                                                              |
    | No       | tokenizer\_config.json    | Tokenizer config. This should be present & have a `chat_template` field for the Friendli Engine to provide chat APIs |
    | No       | special\_tokens\_map.json | Tokenizer's special tokens to their corresponding token strings                                                      |
  </Accordion>
</AccordionGroup>

## Troubleshooting

### Inference Request Errors

<AccordionGroup>
  <Accordion title="Common error codes for inference requests">
    Below is a table of common error codes you might encounter when making inference-related API requests.

    | Code  | Name                    | Cause                                                                                                                                                    | Suggested Solution                                                                                                                                                   |
    | ----- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `400` | *Bad Request*           | The request is malformed or missing required fields.                                                                                                     | Check your request payload. Ensure it is valid JSON with all required fields.                                                                                        |
    | `401` | *Unauthorized*          | Missing or invalid API key. The request lacks proper authentication.                                                                                     | Include a valid Personal API key in the `Authorization` header. Verify the key is active and correct.                                                                |
    | `403` | *Forbidden*             | The API key is valid, but the request is not allowed. This can happen if the key lacks access to the endpoint, or if your team has no remaining credits. | Confirm team access and the `X-Friendli-Team` header if needed. If your team has no remaining credits, [purchase credits](/docs/guides/suite/credits).               |
    | `404` | *Not Found*             | The specified endpoint or resource does not exist. This typically occurs when the `endpoint_id` or `team_id` is invalid.                                 | Verify the `endpoint_id` and model name in your request. Ensure they match an existing, non-deleted deployment. Also check for typos in your endpoint ID or team ID. |
    | `422` | *Unprocessable Entity*  | The request is syntactically correct but semantically invalid (e.g. exceeding token limits, invalid parameter values).                                   | Adjust your request (e.g. reduce `max_tokens`, correct parameter values) and try again.                                                                              |
    | `429` | *Too Many Requests*     | You have exceeded rate limits for your plan.                                                                                                             | Reduce request frequency or upgrade your plan for higher limits. Wait before retrying after a 429 error.                                                             |
    | `500` | *Internal Server Error* | A server-side error occurred while processing the request.                                                                                               | Retry the request after a short delay. If the error persists, check endpoint health in the overview dashboard or contact FriendliAI support.                         |

    #### Quick Checklist Before Retrying

    * Verify the endpoint URL, `endpoint_id`, and (if applicable) `X-Friendli-Team` header
    * Include the `Authorization` header with a valid key
    * Confirm the target deployment exists, is healthy, and is not deleted
    * Validate request JSON and required fields; reduce `max_tokens` if needed
    * Confirm your team has credits
    * Check rate limits; add retry with backoff when receiving `429`
  </Accordion>
</AccordionGroup>

### Model Selection Errors

<AccordionGroup>
  <Accordion title="You don't have access to this gated model">
    <img alt="Troubleshooting - No Access" />

    The repository is gated. Follow the steps and gain approval from the owner using Hugging Face Hub.
  </Accordion>

  <Accordion title="The repository / artifact is invalid">
    <img alt="Troubleshooting - Invalid Repo" />

    <img alt="Troubleshooting - Invalid Artifact" />

    The model does not meet the requirements. Check if the model follows a correct safetensors format. See the [format requirements](#format-requirements) for details.
  </Accordion>

  <Accordion title="The architecture is not supported">
    <img alt="Troubleshooting - Unsupported" />

    The model architecture is not supported. Refer to the [Supported Models](https://friendli.ai/models?products=DEDICATED) page.
  </Accordion>
</AccordionGroup>

### Endpoint Lifecycle

<AccordionGroup>
  <Accordion title="Why was my endpoint suddenly terminated?">
    Endpoints that remain in a sleep state for 48 hours are automatically terminated.

    * When `min_replicas = 0`, the endpoint enters a sleep state after the cooldown period if no requests are received.
    * A notification is sent after 24 hours of sleep, and the endpoint is terminated after another 24 hours if not reactivated.
  </Accordion>
</AccordionGroup>

This page may not cover all cases. If your issue persists, [contact support](mailto:support@friendli.ai).


# Introducing Friendli Dedicated Endpoints
Source: https://friendli.ai/docs/guides/dedicated-endpoints/introduction

Run custom or open-source generative AI models on dedicated GPU hardware with Friendli Dedicated Endpoints. No shared resources or infra management.

Friendli Dedicated Endpoints let you deploy custom or open-source AI models on GPU instances dedicated entirely to you—for consistent performance, full control over your deployment with no infrastructure to manage.

## What Are Friendli Dedicated Endpoints

Each endpoint runs your model on its own GPU instance. With Friendli Dedicated Endpoints, you can:

* **Bring your own model**: Run your own model or deploy any model from [Hugging Face](https://huggingface.co).
* **Choose dedicated resources**: Select the GPU type for your workload. Each instance is fully dedicated to your model—no shared resources.
* **Scale reliably**: Trusted by leading companies for robust performance on production workloads.
* **Pay per second**: You are billed only for the time your model runs. See [Pricing](/docs/guides/dedicated-endpoints/pricing) for details.

## Next Steps

<CardGroup>
  <Card title="QuickStart" icon="rocket" href="/docs/guides/dedicated-endpoints/quickstart">
    Deploy your first endpoint.
  </Card>

  <Card title="Pricing" icon="tag" href="/docs/guides/dedicated-endpoints/pricing">
    Review per-second pricing.
  </Card>

  <Card title="Browse Models" icon="layer-group" href="https://friendli.ai/models?products=DEDICATED">
    Explore available models.
  </Card>
</CardGroup>


# Multi-LoRA Serving
Source: https://friendli.ai/docs/guides/dedicated-endpoints/lora-serving

Learn how to deploy LoRA models from Hugging Face Hub to Friendli Dedicated Endpoints for efficient inference, including a quick guide for FLUX LoRA models.

This document explains how to deploy LoRA models available on Hugging Face to Friendli Dedicated Endpoints.
Friendli Dedicated Endpoints support deploying LoRA adapters for both text generation and FLUX models.

## FLUX LoRA Quick Deployment Guide

This tutorial demonstrates how to deploy the FLUX LoRA model [multimodalart/flux-tarot-v1](https://huggingface.co/multimodalart/flux-tarot-v1), which is trained to generate images in the style of Rider–Waite Tarot cards.

Friendli offers a convenient one-click deployment feature, Deploy-Model, that streamlines the process of serving LoRA adapters from the Hugging Face Hub on Dedicated Endpoints. To deploy a specific model, use a URL in the format `https://friendli.ai/deploy-model/{hf-model-id}`.

For example, to deploy the FLUX LoRA model mentioned above, use [this link](https://friendli.ai/deploy-model/multimodalart/flux-tarot-v1). This will launch the deployment workflow, allowing you to quickly serve and experiment with the model on Friendli.

<Frame>
  <img alt="LoRA Model Deployment" />
</Frame>

Clicking the link above displays a screen like the one shown. Click the **Deploy now** button here to deploy the LoRA model to Friendli Dedicated Endpoints.
Once the deployment is complete, a screen like the one below appears. Click the **Go to Suite** button to navigate to the playground where you can use the LoRA model.

<CardGroup>
  <Card>
    Original Generated Image
  </Card>

  <Card>
    LoRA Generated Image
  </Card>
</CardGroup>

## Advanced: Deploying LoRA Models with Custom Settings

While the quick deployment method described above is convenient, you can also deploy LoRA endpoints with custom settings. This allows you to specify the GPU instance type, endpoint name, scaling options, and more.

<Steps>
  <Step title="Sign In to Friendli Suite">
    Sign in to your [Friendli Suite](https://friendli.ai/suite) account and navigate to the [Dedicated Endpoints](https://friendli.ai/suite/~/dedicated-endpoints).
    If your team has no remaining credits, [purchase credits](/docs/guides/suite/credits) before deploying.

    <Frame>
      <img alt="Friendli Suite Endpoint List" />
    </Frame>
  </Step>

  <Step title="Navigate to the Endpoint Creation Page">
    Create a new project, then click the **New Endpoint** button.
    You'll see a screen like the one below. Enter an Endpoint Name, for example, "My New LoRA Endpoint".

    <Frame>
      <img alt="Create Endpoint" />
    </Frame>
  </Step>

  <Step title="Select the Base Model">
    Friendli Suite currently supports LoRA adapters trained within the Suite and those available on the Hugging Face Hub.
    Since this tutorial doesn't cover fine-tuning, it focuses on deploying LoRA adapters from the Hugging Face Hub.
    First, in the Base Model section, select **Hugging Face**, then select the base model for the LoRA adapter you want to deploy.

    <Tip>
      There are several ways to find the base model of a LoRA adapter. The most common method is to check the model tree on the Hugging Face model page.
    </Tip>

    This example deploys the `predibase/tldr_content_gen` adapter.

    <Frame>
      <img alt="Hugging Face LoRA Adapter Page" />
    </Frame>

    On the [Hugging Face model page](https://huggingface.co/predibase/tldr_content_gen) for this adapter, you can find the Model tree on the right side. This shows the base model used.
    In this case, the adapter is based on the `mistralai/Mistral-7B-v0.1` model.

    <Frame>
      <img alt="Hugging Face Model Tree" />
    </Frame>

    Enter the identified base model name into the model input field on the Endpoint Create page.
  </Step>

  <Step title="Select the LoRA Adapter">
    Now it's time to select the LoRA adapter.

    <Frame>
      <img alt="Endpoint Create Page With Base Model Selected" />
    </Frame>

    Once the base model is selected, the **Add LoRA adapter** button will become active. Click it to open the modal window for adding LoRA adapters.

    <Frame>
      <img alt="Add LoRA Adapter Modal" />
    </Frame>

    Select **Hugging Face adapters** and enter the Hugging Face Model ID of the adapter. For this tutorial, it's `predibase/tldr_content_gen`.

    <Frame>
      <img alt="Select Hugging Face LoRA Adapter" />
    </Frame>

    After adding the adapter, your screen should look like this. Now, select the instance type, configure the autoscaling options appropriately, and click the **Deploy** button.

    <Tip>
      For details on other options, refer to the [Deploy with Hugging Face Models](/docs/guides/dedicated-endpoints/deploy-with-huggingface#step-1-create-a-new-endpoint) documentation.
    </Tip>
  </Step>

  <Step title="Experiment with the Deployed Adapter Model">
    <Frame>
      <img alt="Endpoint Overview" />
    </Frame>

    Once the endpoint is deployed, you'll see a screen like this. Navigate to the Playground page to quickly compare the adapter model and the base model.

    <Frame>
      <img alt="Endpoint Playground" />
    </Frame>

    In the Playground, use the highlighted dropdown menu to switch between the adapter model and the base model for experimentation and comparison.

    That's it! You have successfully deployed a LoRA adapter on Friendli Dedicated Endpoints and experimented with it in the Playground.

    Now you can explore deploying multiple adapters on a single endpoint with [Multi-LoRA Serving](#multi-lora-serving), or use the API to send requests to the model and integrate it into your applications.
  </Step>
</Steps>

## Multi-LoRA Serving

You can serve multiple LoRA adapters on a single base model using Friendli Dedicated Endpoints.


# Models
Source: https://friendli.ai/docs/guides/dedicated-endpoints/models

Manage models for Friendli Dedicated Endpoints. Upload directly, or load from Hugging Face repositories.

Friendli Dedicated Endpoints can deploy models from a variety of sources:

* [Hugging Face](https://huggingface.co) repositories
* Models you upload directly

## How to Upload a Model

You can upload your own model files directly to FriendliAI using the Friendli CLI, then deploy them to a Dedicated Endpoint.

<Steps>
  <Step title="Install the CLI">
    Install the Friendli CLI, specifying the exact version:

    ```bash theme={null}
    pip install friendli-client==2.0.0a16
    ```
  </Step>

  <Step title="Prepare Your API Key">
    The CLI requires authentication. Create an API key in [Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys) and set it as an environment variable:

    ```bash theme={null}
    export API_KEY=<YOUR_API_KEY>
    ```
  </Step>

  <Step title="Confirm Your Project ID">
    Identify the Project ID where you want to upload your model, and set it as an environment variable:

    ```bash theme={null}
    export PROJECT_ID=<YOUR_PROJECT_ID>
    ```
  </Step>

  <Step title="Run the Upload Command">
    Upload your local model directory to FriendliAI:

    ```bash theme={null}
    friendli --token $API_KEY model push --project $PROJECT_ID --name new-model /path/to/model
    ```

    * `--name`: The name to assign to the model on Friendli.
    * `/path/to/model`: The local path to your model files.
  </Step>
</Steps>

<Note>
  * Identical files are deduplicated automatically and won't be re-uploaded.
  * Some files may be skipped if they are unnecessary or unsupported.
</Note>


# Online Quantization
Source: https://friendli.ai/docs/guides/dedicated-endpoints/online-quantization

Automatically quantize models to 4-bit or 8-bit precision at deploy time on Friendli Dedicated Endpoints. No pre-quantized checkpoint needed.

Online Quantization quantizes your model at runtime using FriendliAI's proprietary method, improving speed and reducing cost with little to no loss in accuracy. This lets you select lower-VRAM GPU instances without sacrificing performance.

You can configure the precision level with the following options:

* **Off**: Serve the model at its original precision.
* **4-bit**: Quantize to 4-bit precision for the largest savings in memory and cost.
* **8-bit**: Quantize to 8-bit precision for a balance between savings and accuracy.

<Note>
  Some models (e.g., those already quantized) may not be compatible with Online Quantization.
  Not all models support all target precisions. Some may only support 8-bit.\
  In certain cases, specific GPU instance types may not be available when this option is enabled.
</Note>


# GPUs and Pricing
Source: https://friendli.ai/docs/guides/dedicated-endpoints/pricing

View Friendli Dedicated Endpoints pricing by GPU type. Covers supported instance types, per-second billing, and how autoscaling affects costs.

Dedicated Endpoints offer flexible monthly billing based on **actual usage**.

## Supported Instance Types

<Warning>
  New GPU prices are effective October 1, 2026. Until then, you can compare current and new rates. To learn more, see [Pricing > Dedicated Endpoints](https://friendli.ai/pricing/dedicated-endpoints).
</Warning>

<DedicatedGpuPricingTable />

<Info>
  Contact sales for a discounted custom pricing plan for your enterprise.
</Info>

## How Does Billing Work for Dedicated Endpoints?

Billing is measured per GPU-second for each **replica**. An endpoint can run multiple replicas at once, each metered independently for the time it's running.

For each replica:

* Billing accrues once the replica has finished initializing. The startup work—provisioning GPUs, downloading the model, and initializing the engine—is not billed.
* Once running, the replica is billed for as long as it stays up, even when it's idle and receiving no traffic.
* Billing stops as soon as the replica begins shutting down, such as during scale-down, sleep, or termination.

Therefore, scaling up adds cost for each new replica once it's running. An endpoint that always runs at least one replica keeps accruing charges.

## How Does Autoscaling Affect My Costs?

Autoscaling adjusts the number of replicas to match traffic, and each replica is metered by the same rules above. Your cost rises and falls with the number of running replicas—for example, running 2 replicas instead of 1 doubles your GPU cost.

## Best Practices for Cost Management

* **Monitor running endpoints**: Regularly review which endpoints are running and how many replicas they use, so you don't pay for capacity you don't need.
* **Enable sleeping for endpoints with intermittent traffic**: Set the minimum replica count to 0 so an endpoint sleeps when idle and wakes automatically on the next request.


# QuickStart: Friendli Dedicated Endpoints
Source: https://friendli.ai/docs/guides/dedicated-endpoints/quickstart

Get started with Friendli Dedicated Endpoints. Create a project, pick a model, deploy an endpoint, and generate your first inference response.

Get started with [Friendli Dedicated Endpoints](/docs/guides/dedicated-endpoints/introduction). This quickstart walks you through launching your first endpoint and sending your first request.

## 1. Sign Up or Sign In

Create an account or sign in at [Friendli Suite](https://friendli.ai/suite).

## 2. Navigate to the Dedicated Endpoints Page

Go to [Dedicated Endpoints](https://friendli.ai/suite/~/dedicated-endpoints) to see your endpoint list.

<RoundedBorderBox>
  <img alt="Sidebar" />
</RoundedBorderBox>

## 3. Prepare Your Model

Select a model to serve from [Hugging Face](https://huggingface.co), or upload your own.

<img alt="Hugging Face" />

## 4. Deploy Your Endpoint

Deploy the model from step 3 with your selected GPU. You can also configure replicas and optimization options.

<RoundedBorderBox>
  <img alt="Create Endpoint" />
</RoundedBorderBox>

<br />

<RoundedBorderBox>
  <img alt="Endpoint Detail" />
</RoundedBorderBox>

## 5. Generate Responses

Generate responses in two ways: the playground or the endpoint URL.

Use the playground tab to test your model in a chat-style interface.

<RoundedBorderBox>
  <img alt="Endpoint Playground" />
</RoundedBorderBox>

For programmatic use, send requests to your endpoint address—shown on the endpoint information tab—through our [API](/docs/openapi). See [Manage Your FriendliAI API Keys](/docs/guides/suite/personal-api-keys) to create an API key.

<CodeGroup>
  ```python OpenAI Python SDK theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      api_key=os.getenv("API_KEY"),
      base_url="https://api.friendli.ai/dedicated/v1",
  )

  chat_completion = client.chat.completions.create(
      model="YOUR_ENDPOINT_ID",
      messages=[
          {
              "role": "user",
              "content": "Tell me how to make a delicious pancake"
          }
      ]
  )
  print(chat_completion.choices[0].message.content)
  ```

  ```sh curl theme={null}
  curl -X POST https://api.friendli.ai/dedicated/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "X-Friendli-Team: $TEAM_ID" \
    -H "Authorization: Bearer $API_KEY" \
    -d '{
      "model": "YOUR_ENDPOINT_ID",
      "messages": [
        {
          "role": "user",
          "content": "Python is a popular"
        }
      ]
    }'
  ```
</CodeGroup>

<Note>
  For a more detailed tutorial, refer to our guide for using [Hugging Face models](/docs/guides/dedicated-endpoints/deploy-with-huggingface).
</Note>


# Request Queueing
Source: https://friendli.ai/docs/guides/dedicated-endpoints/request-queueing

Learn how to configure request queueing to keep performance predictable when a Friendli Dedicated Endpoint receives more traffic than it can serve.

Request queueing keeps your endpoint's performance predictable even under traffic that exceeds its capacity. Instead of accepting every request and overloading the endpoint, it holds excess requests in a queue, so the requests being processed maintain consistent performance.

## Queueing Threshold

A queueing threshold defines the capacity at which an endpoint starts queueing requests. Friendli Dedicated Endpoints currently support one threshold type:

* **Request count**: The average number of in-flight requests each replica should handle. The endpoint multiplies this value by the current number of running replicas to get its capacity. Once the number of in-flight requests exceeds that capacity, the extra requests are queued rather than routed to a replica.

<Note>
  The appropriate threshold depends on your model, GPU instance, and workload characteristics, so tune it to the point where the endpoint meets your target performance.
</Note>

## Queue Timeout

Queue timeout is optional and controls how long a request can stay in the queue:

* **Not set**: Queued requests wait until capacity frees up.
* **Set**: When a queued request has waited longer than the timeout, the endpoint returns a `429 Too Many Requests` response, so the client can retry or fall back.

Set a queue timeout when you would rather reject a request than let it wait too long.

## Set Up Request Queueing

<Steps>
  <Step title="Find the Endpoint Features Section">
    While creating or updating an endpoint, go to **Endpoint Features**.
  </Step>

  <Step title="Enable Request Queueing">
    Turn on **Request Queueing**.
  </Step>

  <Step title="Configure the Threshold and Timeout">
    Enter a **Request Count Threshold** (minimum 1) and, optionally, a **Queue timeout** in seconds. Leave the timeout empty or set it to 0 for **No Limit**.
  </Step>

  <Step title="Deploy or Update the Endpoint">
    Click **Deploy** for a new endpoint, or **Update** to apply changes to an existing one.
  </Step>
</Steps>


# Speculative Decoding
Source: https://friendli.ai/docs/guides/dedicated-endpoints/speculative-decoding

Speed up LLM inference on Friendli Dedicated Endpoints with speculative decoding using proprietary draft models and N-gram token prediction.

Speculative decoding speeds up generation by drafting candidate tokens and verifying them with the target model in parallel, so the model accepts more tokens per forward pass. Friendli Dedicated Endpoints support two methods.

## Draft-Model Method

You can enable speculative decoding by pairing the target model with a pre-trained draft model. This improves inference efficiency by allowing a fast draft model to propose multiple tokens that the larger target model verifies in parallel. As a result, the model can accept multiple tokens per forward pass, increasing throughput.

<Note>This feature is currently limited to a curated list of target models.</Note>

## N-gram Method

Toggle the switch to enable N-gram speculative decoding. When enabled, the system uses past tokens to pre-generate future tokens. For predictable tasks, this can deliver substantial performance gains.

You can also set the `Maximum N-gram Size`, which defines how many tokens are predicted in advance. We recommend keeping the default value of 3.

<Note>Higher values can further reduce latency when successful. However, predicting too many tokens at once may lower prediction efficiency and, in extreme cases, even increase latency.</Note>


# Versioning
Source: https://friendli.ai/docs/guides/dedicated-endpoints/versioning

Use endpoint versioning on Friendli Dedicated Endpoints to track deployment history, roll back to previous configurations, and update without downtime.

Versioning lets you roll out and roll back endpoints without downtime, so you can update configurations safely and revert to a previous state whenever needed.

## Why Use Versioning

The versioning feature in Friendli Dedicated Endpoints helps you manage all changes to your deployed endpoints safely and transparently. When you update the configuration—like changing the model, engine settings, or autoscaling—FriendliAI creates a new version instead of replacing the current one.

* **Zero-Downtime Updates**: Safely apply changes while the current version continues to serve traffic.

* **One-Click Rollbacks**: Instantly revert to a previous stable configuration if issues occur.

* **Easy-to-Follow History**: Each version shows who made the change, when it was made, and what was changed. This makes audits and debugging easier.

Each version captures a full snapshot of the deployment, including:

* Model name and artifact source
* Accelerator type and count
* Autoscaling and engine settings
* Metadata (creator, timestamps, comments)

<RoundedBorderBox>
  <img alt="Updated Version Configuration Modal" />
</RoundedBorderBox>

## How to Use Versioning

1. **Initial Deployment**: Deploy your model for the first time via the platform or webhook. This creates version `v0`.

<RoundedBorderBox>
  <img alt="Initial Version (v0) Running" />
</RoundedBorderBox>

2. **Apply Configuration Updates**: Changing any setting—such as model, accelerator type, or autoscaling—triggers a new version (`v1`, `v2`, etc.).

3. **Browse Version History**: View the full version list by clicking the **Versions** tab on the endpoint detail page. You'll see which version is current or in progress.

<RoundedBorderBox>
  <img alt="Applying Version v2" />
</RoundedBorderBox>

4. **View Configuration Details**: Click **View configs** to see a version's full settings. Updates from the previous version are marked with a blue badge for easy comparison.

<RoundedBorderBox>
  <img alt="Viewing Version v1 Details" />
</RoundedBorderBox>

## How to Roll Back to a Previous Version

To roll back, select a previous version from the version history and click **Rollback**.

<RoundedBorderBox>
  <img alt="Rollback" />
</RoundedBorderBox>

The system creates a new version (`vN+1`) using the selected version's settings. This new version becomes the current one, allowing you to quickly revert to a known good state.

### When an Update Fails

Update failures can occur due to various reasons, such as:

* **Configuration Errors**: Invalid settings or unsupported configurations can prevent the update.
* **Resource Limitations**: Insufficient resources (like GPU availability) can block the update.
* **Network Issues**: Temporary network problems can interrupt the update process.

When you attempt to update an endpoint and the process fails, the system does not automatically apply the changes.

Instead, it logs the error and lets you troubleshoot the issue without affecting the live endpoint. This keeps your endpoint operational without disruption.


# Get Started with FriendliAI
Source: https://friendli.ai/docs/guides/introduction/get-started

Get started with FriendliAI. Create an API key, send your first request with Model APIs, and set up your favorite coding agent or SDK.

Create a FriendliAI API key, send your first request, and learn to control a model's capabilities. Once you complete these steps, you're ready to [choose a popular model](/docs/examples/models/overview) and [use your favorite agent](/docs/examples/agents/overview) or [SDK](/docs/examples/sdks/overview) with FriendliAI.

With [Friendli Model APIs](/docs/guides/model-apis/introduction), you get access to a curated set of popular, open-weight models that are ready for you today. If you want to run any model—including your own—on dedicated GPUs, try [Friendli Dedicated Endpoints](/docs/guides/dedicated-endpoints/introduction).

To get started, complete the following steps.

## Create an API Key

<Steps>
  <Step title="Create a Friendli Suite Account">
    If you haven't already, [sign up](https://auth.friendli.ai/sign-up) for an account.
  </Step>

  <Step title="Sign In">
    Then, [sign in](https://auth.friendli.ai). Friendli Suite opens your dashboard.
  </Step>

  <Step title="Create a FriendliAI API Key">
    1. In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**.
    2. In the upper-right corner, click <Icon icon="plus" /> **Create API Key**.
    3. (Optional) Name your API key and set an expiration date.
    4. Click **Create Key**.
    5. Click <Icon icon="copy" /> **Copy**.
  </Step>
</Steps>

## Send a Request

<Steps>
  <Step title="Set Up Your API Key">
    In your terminal, run the following command:

    ```bash theme={null}
    export FRIENDLIAI_API_KEY="<FRIENDLIAI_API_KEY>"
    ```

    Replace `<FRIENDLIAI_API_KEY>` with your API key.
  </Step>

  <Step title="Install the OpenAI SDK">
    In your terminal, run the following command:

    <CodeGroup>
      ```bash OpenAI Python SDK theme={null}
      pip install openai
      ```

      ```bash OpenAI JavaScript SDK theme={null}
      npm install openai
      ```
    </CodeGroup>
  </Step>

  <Step title="Send Your Request to FriendliAI">
    Run the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ]
      }'
      ```
    </CodeGroup>

    <Check>
      FriendliAI returns a response, similar to the following:

      ```json expandable wrap theme={null}
      {
        "id": "chatcmpl-04eb40ba7fc14cf8a51b7869d9cfa76f",
        "object": "chat.completion",
        "choices": [
          {
            "index": 0,
            "message": {
              "role": "assistant",
              "content": "FriendliAI is a generative AI infrastructure company that provides optimized solutions to help businesses deploy large language models faster, more efficiently, and at a lower cost.",
              "reasoning": "...",
              "reasoning_content": "..."
            },
            "logprobs": null,
            "finish_reason": "stop"
          }
        ],
        "created": 1781892285,
        "usage": {
          "completion_tokens": 412,
          "prompt_tokens": 27,
          "prompt_tokens_details": {
            "cached_tokens": 0
          },
          "total_tokens": 439
        },
        "model": "zai-org/GLM-5.3"
      }
      ```
    </Check>
  </Step>
</Steps>

## Control a Capability

You can control most capabilities—such as streaming or reasoning—with parameters, which you pass as either request body parameters or chat template keyword arguments. How you pass each depends on the parameter:

<Steps>
  <Step title="Turn Streaming On" icon="dot">
    Some request body parameters are processed by the Friendli Engine and natively by an SDK.

    For example, to turn streaming on, run the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      stream = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        stream=True,
      )

      for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const stream = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "stream": true,
      })

      for await (const chunk of stream) {
        process.stdout.write(chunk.choices[0].delta?.content || "")
      }
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "stream": true
      }'
      ```
    </CodeGroup>
  </Step>

  <Step title="Set the Reasoning Budget" icon="dot">
    Other request body parameters are processed exclusively by the Friendli Engine and are used to further shape the response.

    For example, to set the reasoning budget to 10,000 tokens, run the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-17} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "reasoning_budget": 10000,
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "reasoning_budget": 10000,
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "reasoning_budget": 10000
      }'
      ```
    </CodeGroup>
  </Step>

  <Step title="Clear Reasoning" icon="dot">
    Chat template keyword arguments are always forwarded to the model's chat template and used to construct the prompt to the model's specifications.

    For example, to clear reasoning, run the following code:

    <CodeGroup>
      ```python OpenAI Python SDK wrap highlight={15-19} theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.environ["FRIENDLIAI_API_KEY"],
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You are a friendly assistant."},
          {"role": "user", "content": "Describe FriendliAI in one sentence."},
        ],
        extra_body={
          "chat_template_kwargs": {
            "clear_thinking": True,
          },
        },
      )

      print(completion.choices[0].message)
      ```

      ```javascript OpenAI JavaScript SDK wrap highlight={14-16} theme={null}
      import OpenAI from "openai"

      const client = new OpenAI({
        baseURL: "https://api.friendli.ai/serverless/v1",
        apiKey: process.env.FRIENDLIAI_API_KEY,
      })

      const completion = await client.chat.completions.create({
        model: "zai-org/GLM-5.3",
        messages: [
          { role: "system", content: "You are a friendly assistant." },
          { role: "user", content: "Describe FriendliAI in one sentence." },
        ],
        "chat_template_kwargs": {
          "clear_thinking": true,
        },
      })

      console.log(completion.choices[0].message)
      ```

      ```bash cURL wrap highlight={10-12} theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H 'Content-Type: application/json' \
        -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You are a friendly assistant."},
            {"role": "user", "content": "Describe FriendliAI in one sentence."}
          ],
          "chat_template_kwargs": {
            "clear_thinking": true
          }
      }'
      ```
    </CodeGroup>
  </Step>
</Steps>

Congratulations. You learned to control a model's capabilities and are ready to build your next project.

## Use Popular Models and Your Favorite Agent or SDK

You're ready to start building with your favorite coding agent or SDK to connect to fast, cost-efficient, and reliable open-weight models.

<CardGroup>
  <Card title="Models" icon="brain" href="/docs/examples/models/overview">
    Use popular FriendliAI models.
  </Card>

  <Card title="Agents" icon="robot" href="/docs/examples/agents/overview">
    Use your favorite agent with FriendliAI.
  </Card>

  <Card title="SDKs" icon="code" href="/docs/examples/sdks/overview">
    Use your favorite SDK with FriendliAI.
  </Card>
</CardGroup>


# Build with FriendliAI
Source: https://friendli.ai/docs/guides/introduction/overview

Build with FriendliAI for fast, affordable, and reliable AI inference. Start with Model APIs, then deploy any model on dedicated GPUs with Dedicated Endpoints.

FriendliAI's inference is powered by the Friendli Engine, optimized for speed and cost. In a few steps, you get access to a popular set of open-weight models that are comparable to the frontier models you're used to. When you're ready for more, you can deploy any model, including your own, on dedicated GPUs.

## Start Your Journey

Create an API key, choose a model, and send your first request with [Friendli Model APIs](/docs/guides/model-apis/introduction). FriendliAI's [OpenAI-compatible Chat Completions API](/docs/openapi/model-apis/chat-completions) works with most coding agents and SDKs. You can also try the [Anthropic-compatible Messages API](/docs/openapi/model-apis/messages) <Badge>Beta</Badge>.

<Card title="Model APIs" icon="cloud" href="/docs/guides/introduction/get-started">
  Start with a curated set of models available at usage-based pricing.
</Card>

## Run Any Model on Dedicated GPUs

If you want to run any model, including your own, you're ready for [Friendli Dedicated Endpoints](/docs/guides/dedicated-endpoints/introduction). With Dedicated Endpoints, you can browse over 610,000 models and deploy your choice on dedicated GPUs.

<Card title="Dedicated Endpoints" icon="server" href="/docs/guides/dedicated-endpoints/quickstart">
  Deploy any model on dedicated GPUs reserved for you.
</Card>

## Resources

<CardGroup>
  <Card title="Reference" icon="book" href="/docs/openapi/introduction">
    Learn more about the API and its operations and parameters.
  </Card>

  <Card title="Examples" icon="code" href="/docs/examples/models/overview">
    See what you can do with real-world use cases.
  </Card>
</CardGroup>

<CardGroup>
  <ModelsCard />

  <PricingCard />

  <Card title="Blog" icon="newspaper" href="https://friendli.ai/blog">
    Read the latest posts from FriendliAI.
  </Card>
</CardGroup>


# Introducing Friendli Model APIs
Source: https://friendli.ai/docs/guides/model-apis/introduction

Get started with Friendli Model APIs to access popular AI models via API. No infrastructure setup or GPU management required.

Friendli Model APIs let you access frontier open-source models through a ready-to-use API endpoint—no deployment required, and you pay only for what you use.

## What Are Friendli Model APIs

Friendli Model APIs provide instant access to a curated set of models, powered by [Friendli Engine](https://friendli.ai/why-friendliai) for high-performance, cost-efficient inference. With Friendli Model APIs, you can:

* **Access popular models instantly**: Use state-of-the-art models without downloading, hosting, or optimizing them yourself.
* **Integrate in a few lines of code**: Connect through any OpenAI-compatible client.
* **Pay per token, not per GPU**: You are billed only for what you use, with no idle GPU cost. See [Pricing](/docs/guides/model-apis/pricing) for details.

## Next Steps

<CardGroup>
  <Card title="QuickStart" icon="rocket" href="/docs/guides/model-apis/quickstart">
    Make your first API call in minutes.
  </Card>

  <Card title="Pricing" icon="tag" href="/docs/guides/model-apis/pricing">
    Review usage-based pricing.
  </Card>

  <Card title="Browse Models" icon="layer-group" href="https://friendli.ai/models?products=SERVERLESS">
    Explore available models.
  </Card>
</CardGroup>


# Models and Pricing
Source: https://friendli.ai/docs/guides/model-apis/pricing

View Friendli Model APIs pricing per model. Compare token-based and audio-based rates across text and audio models.

Model APIs are often more economical, with access to a wide range of models. Pricing varies by model type—text models are charged by processed tokens of your request, while audio models are charged by the duration of processed audio.

## Text Models

Text models are charged by the **number of processed tokens**.

<ModelApisTokenBasedTable />

## Audio Models

Audio models are charged based on the **duration of processed audio**. Charges are calculated per second and aggregated into a per-minute rate for clarity.

<ModelApisAudioMinuteBasedTable />

<Note>
  For custom pricing, contact [support@friendli.ai](mailto:support@friendli.ai).
</Note>


# QuickStart: Friendli Model APIs
Source: https://friendli.ai/docs/guides/model-apis/quickstart

Get started with Friendli Model APIs in minutes. Explore popular AI models, experiment in a chat-style playground, and make your first API call with no setup required.

Get started with [Friendli Model APIs](/docs/guides/model-apis/introduction) in two ways: interact with models in the playground, or call the API directly from your application. This quickstart walks you through both.

## Explore Models in the Playground

Try models directly in Friendli Suite. The playground provides an interactive experience where you can test prompts, inspect responses, and fine-tune inference settings.

### 1. Sign Up or Sign In

Create an account or sign in at [Friendli Suite](https://friendli.ai/suite).

### 2. Navigate to the Model APIs Page

Go to the [Model APIs page](https://friendli.ai/suite/~/model-apis) to see all available models.

<RoundedBorderBox>
  <img alt="Model APIs" />
</RoundedBorderBox>

### 3. Select a Model

Select a model from the list. Each model page includes a brief overview and usage details.

<RoundedBorderBox>
  <img alt="Model APIs Overview" />
</RoundedBorderBox>

### 4. Send a Prompt

The playground offers a chat-style interface with built-in tools such as a calculator and [Linkup web search](https://friendli.ai/blog/linkup-partnership). You can also adjust inference parameters like temperature and top-p to fine-tune the model's behavior.

<RoundedBorderBox>
  <img alt="Playground" />
</RoundedBorderBox>

<br />

<RoundedBorderBox>
  <img alt="Model APIs Chat" />
</RoundedBorderBox>

## Use the API in Your Application

To use a model in your application, call it directly through the API. Each model page includes ready-to-use example code you can paste into your application.

### 1. Sign Up or Sign In

Create an account or sign in at [Friendli Suite](https://friendli.ai/suite).

### 2. Create a Personal API Key

You can create and manage API keys in: [Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).

<RoundedBorderBox>
  <img alt="Personal API Keys" />
</RoundedBorderBox>

Set your key as an environment variable:

```shell theme={null}
export API_KEY="YOUR API KEY HERE"
```

### 3. Install the OpenAI SDK

```shell theme={null}
pip install openai
```

### 4. Send an API Request

You can now start sending API requests right away.

<CodeGroup>
  ```python OpenAI Python SDK theme={null}
  import os

  from openai import OpenAI

  client = OpenAI(
      api_key=os.getenv("API_KEY"),
      base_url="https://api.friendli.ai/serverless/v1",
  )

  completion = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      extra_body={
          "parse_reasoning": True,
          "chat_template_kwargs": {"enable_thinking": True},
      },
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Hello!"},
      ],
  )

  print("Reasoning: ", completion.choices[0].message.reasoning_content)
  print(completion.choices[0].message.content)
  ```

  ```sh curl theme={null}
  curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $API_KEY" \
    -d '{
    "model": "zai-org/GLM-5.3",
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"}
      ],
      "stream": true
  }'
  ```
</CodeGroup>


# Rate Limits
Source: https://friendli.ai/docs/guides/model-apis/rate-limits

Understand Friendli Model APIs rate limits and usage tiers. Tiers are based on lifetime spend and increase automatically as your usage grows.

Friendli Model APIs apply rate limits based on your usage tier. Higher tiers unlock higher request rates and output length limits.

## Tier-Based API Rate Limits

Tiers are based on **lifetime spending and update automatically**. As your lifetime spend grows, your tier increases. You can move up instantly by purchasing additional credits.

<RateLimitTiersTable />

<Info>
  **Adaptive Rate Limits**: Rate limits are applied dynamically based on overall platform conditions.
</Info>

<Warning>
  'Output Token Length' is how much the model can write in response. It's different from 'Context Length', which is the sum of the input and output tokens.
</Warning>


# Multi-Modality
Source: https://friendli.ai/docs/guides/multi-modality

Process text, images, audio, and video with FriendliAI multimodal APIs. Includes vision, transcription, and image generation endpoint guides.

FriendliAI supports multimodal workflows across text, image, audio, and video. \
Use the comprehensive guides below to get started with each modality.

## Quick Navigation

* [Image Generation](#image-generation) - Generate images from text prompts
* [Vision (Image Understanding)](#vision-image-understanding) - Analyze and understand images
* [Video Understanding](#video-understanding) - Process and analyze video content
* [Audio and Speech](#audio-and-speech) - Convert audio to text and analyze audio

### Image Generation

Transform text prompts into high-quality visuals with FriendliAI's image generation capabilities.

#### Representative Models

We support various trending image generation models including:

* [FLUX.1-dev](https://friendli.ai/models?baseModel=black-forest-labs/FLUX.1-dev)
* [FLUX.1-schnell](https://friendli.ai/models?baseModel=black-forest-labs/FLUX.1-schnell)
* [See all image generation models](https://friendli.ai/models?input=TEXT\&output=IMAGE)

#### API Usage

<CodeGroup>
  ```bash curl theme={null}
  curl -L -X POST "https://api.friendli.ai/dedicated/v1/images/generations" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $API_KEY" \
    --data-raw '{
      "model": "YOUR_ENDPOINT_ID",
      "prompt": "An orange Lamborghini driving down a hill road at night with a beautiful ocean view in the background.",
      "num_inference_steps": 10,
      "guidance_scale": 3.5
    }'
  ```

  ```python OpenAI Python SDK theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/dedicated/v1",
      api_key=os.environ.get("API_KEY"),
  )

  images = client.images.generate(
      model="YOUR_ENDPOINT_ID",
      prompt="An orange Lamborghini driving down a hill road at night with a beautiful ocean view in the background.",
      extra_body={
          "num_inference_steps": 10,
          "guidance_scale": 3.5
      }
  )

  print(images.data[0].url)
  ```
</CodeGroup>

### Vision (Image Understanding)

Analyze and understand images using FriendliAI's vision capabilities.

#### Representative Models

We support various trending vision models including:

* **Qwen2.5-VL**
* **InternVL3**
* [See all vision models](https://friendli.ai/models?input=IMAGE\&output=TEXT)

#### Supported Image Formats

Supports formats supported by the PIL library:

* JPEG (.jpeg and .jpg)
* PNG (.png)
* AVIF (.avif)

#### API Usage

<CodeGroup>
  ```python URL-based image theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/dedicated/v1",
      api_key=os.environ.get("API_KEY"),
  )

  image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"

  completion = client.chat.completions.create(
      model="YOUR_ENDPOINT_ID",
      messages=[
          {
              "role": "user",
              "content": [
                  {
                      "type": "text",
                      "text": "What kind of animal is shown in the image?",
                  },
                  {"type": "image_url", "image_url": {"url": image_url}},
              ],
          },
      ],
      stream=False
  )

  print(completion.choices[0].message.content)
  ```

  ```python Base64-encoded image theme={null}
  import base64, requests, os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/dedicated/v1",
      api_key=os.environ.get("API_KEY"),
  )

  image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"
  image_media_type = "image/jpg"
  image_base64 = base64.standard_b64encode(requests.get(image_url).content).decode(
      "utf-8"
  )

  completion = client.chat.completions.create(
      model="YOUR_ENDPOINT_ID",
      messages=[
          {
              "role": "user",
              "content": [
                  {
                      "type": "text",
                      "text": "What kind of animal is shown in the image?",
                  },
                  {
                      "type": "image_url",
                      "image_url": {
                          "url": f"data:{image_media_type};base64,{image_base64}"
                      },
                  },
              ],
          },
      ],
  )

  print(completion.choices[0].message.content)
  ```
</CodeGroup>

### Video Understanding

Process and analyze video content with FriendliAI's video understanding capabilities.

#### Representative Models

We support various video understanding models including:

* **Qwen2.5-VL**
* [See all video models](https://friendli.ai/models?input=VIDEO\&output=TEXT)

#### Video Requirements

* Videos must be hosted at publicly accessible URLs
* We recommend HTTPS URLs for security
* Consider video file size and processing time implications
* Some models may have specific resolution or duration requirements

#### API Usage

By default, video fetching timeout is 30 seconds. To increase the timeout value, [contact us](mailto:support@friendli.ai).

<CodeGroup>
  ```python Single Video Input theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/dedicated/v1",
      api_key=os.environ.get("API_KEY"),
  )

  video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"

  completion = client.chat.completions.create(
      model="YOUR_ENDPOINT_ID",
      messages=[
          {
              "role": "user",
              "content": [
                  {
                      "type": "text",
                      "text": "What's in this video?",
                  },
                  {
                      "type": "video_url",
                      "video_url": {"url": video_url},
                  },
              ],
          },
      ],
      temperature=0,
      max_tokens=100,
  )

  print(completion.choices[0].message.content)
  ```

  ```python Multi-Video Input theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.friendli.ai/dedicated/v1",
      api_key=os.environ.get("API_KEY"),
  )

  video_url_1 = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
  video_url_2 = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"

  completion = client.chat.completions.create(
      model="YOUR_ENDPOINT_ID",
      messages=[
          {
              "role": "user",
              "content": [
                  {
                      "type": "text",
                      "text": "Describe the characters in each video concisely.",
                  },
                  {
                      "type": "video_url",
                      "video_url": {"url": video_url_2},
                  },
                  {
                      "type": "video_url",
                      "video_url": {"url": video_url_1},
                  },
              ],
          },
      ],
      temperature=0,
      max_tokens=100,
  )

  print(completion.choices[0].message.content)
  ```
</CodeGroup>

### Audio and Speech

Convert audio files to text and perform various AI tasks with FriendliAI's audio capabilities.

#### Representative Models

We support various trending audio models including:

* **Whisper Large V3**
* **Qwen2-Audio**
* **Ultravox**
* [See all audio models](https://friendli.ai/models?input=AUDIO\&output=TEXT)

#### Supported Audio Formats

Our platform supports a wide range of audio formats compatible with the **librosa library**:

* **MP3** (.mp3)
* **WAV** (.wav)
* **FLAC** (.flac)
* **OGG** (.ogg)
* And many other standard audio formats

#### API Usage

By default, audio input is limited to 30 seconds. To enable longer audio inputs, [contact us](mailto:support@friendli.ai).

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.friendli.ai/dedicated/v1/audio/transcriptions \
    -H "Authorization: Bearer $API_KEY" \
    -H 'Content-Type: multipart/form-data' \
    -F file=@/path/to/audio/file.mp3 \
    -F model="YOUR_ENDPOINT_ID"
  ```

  ```python OpenAI Python SDK theme={null}
  from openai import OpenAI
  import os

  client = OpenAI(
    base_url="https://api.friendli.ai/dedicated/v1",
    api_key=os.getenv("API_KEY"),
  )
  audio_file= open("/path/to/file/audio.mp3", "rb")

  transcription = client.audio.transcriptions.create(
      model="YOUR_ENDPOINT_ID",
      file=audio_file
  )

  print(transcription.text)
  ```
</CodeGroup>

### API References

For detailed API specifications, refer to:

* [Image Generation API Reference](/docs/openapi/dedicated/inference/image-generations)
* [Image/Video/Audio Understanding API Reference](/docs/openapi/dedicated/inference/chat-completions)
* [Audio Transcriptions API Reference](/docs/openapi/dedicated/inference/audio-transcriptions)


# OpenAI Compatibility
Source: https://friendli.ai/docs/guides/openai-compatibility

Use official OpenAI Python and Node.js SDKs with FriendliAI endpoints. Migrate existing OpenAI applications by changing the base URL and API key.

Friendli Model APIs and Friendli Dedicated Endpoints are [OpenAI-compatible](/docs/openapi/introduction). \
You can migrate existing applications with minimal effort, still using the official OpenAI SDKs.

## Specify the Base URL and API Key

Initialize the OpenAI client using FriendliAI's base URL and your Personal API key.

* **Model APIs**: `https://api.friendli.ai/serverless/v1`.
* **Dedicated Endpoints**: `https://api.friendli.ai/dedicated/v1`.

Get your Personal API key in [Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).

<CodeGroup>
  ```python Python theme={null}
  client = OpenAI(
      api_key=os.getenv("API_KEY"),
      base_url="https://api.friendli.ai/serverless/v1",
  )
  ```

  ```javascript Node.js theme={null}
  const client = new OpenAI({
    apiKey: process.env.API_KEY,
    baseURL: "https://api.friendli.ai/serverless/v1",
  });
  ```
</CodeGroup>

## Usage

Choose any model available on Friendli Model APIs or Dedicated Endpoints.

### Completions API

Generate text completions using a simple prompt-based approach.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import os

  client = OpenAI(
      api_key=os.getenv("API_KEY"),
      base_url="https://api.friendli.ai/serverless/v1",
  )

  completion = client.completions.create(
      model="zai-org/GLM-5.3",
      prompt="Tell me a funny joke about programming.",
      max_tokens=100,
      temperature=0.7,
  )
  print(completion.choices[0].text)
  ```

  ```javascript Node.js theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.API_KEY,
    baseURL: "https://api.friendli.ai/serverless/v1",
  });

  async function main() {
    const completion = await client.completions.create({
      model: "zai-org/GLM-5.3",
      prompt: "Tell me a funny joke about programming.",
      max_tokens: 100,
      temperature: 0.7,
    });

    console.log(completion.choices[0].text);
  }

  main().catch(console.error);
  ```
</CodeGroup>

### Chat Completions API

Generate chat completions using a conversational message-based approach.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import os

  client = OpenAI(
      api_key=os.getenv("API_KEY"),
      base_url="https://api.friendli.ai/serverless/v1",
  )

  completion = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Tell me a funny joke."},
      ],
      stream=False,
  )
  print(completion.choices[0].message.content)
  ```

  ```javascript Node.js theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.API_KEY,
    baseURL: "https://api.friendli.ai/serverless/v1",
  });

  async function main() {
    const completion = await client.chat.completions.create({
      model: "zai-org/GLM-5.3",
      messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "Tell me a funny joke." },
      ],
    });

    console.log(completion.choices[0].message.content);
  }

  main().catch(console.error);
  ```
</CodeGroup>

### Streaming Mode

Receive responses in real-time, enabling a better user experience for long responses.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  import os

  client = OpenAI(
      api_key=os.getenv("API_KEY"),
      base_url="https://api.friendli.ai/serverless/v1",
  )

  stream = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Tell me a funny joke."},
      ],
      stream=True,
  )

  for chunk in stream:
      print(chunk.choices[0].delta.content or "", end="", flush=True)
  ```

  ```javascript Node.js theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.API_KEY,
    baseURL: "https://api.friendli.ai/serverless/v1",
  });

  async function main() {
    const stream = await client.chat.completions.create({
      model: "zai-org/GLM-5.3",
      messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "Tell me a funny joke." },
      ],
      stream: true,
    });

    for await (const chunk of stream) {
      process.stdout.write(chunk.choices[0].delta?.content || "");
    }
  }

  main().catch(console.error);
  ```
</CodeGroup>


# Structured Outputs
Source: https://friendli.ai/docs/guides/structured-outputs

Generate JSON outputs conforming to a schema using FriendliAI Structured Outputs. Works on all chat-capable models with response_format support.

FriendliAI offers structured outputs capability with two core guarantees:

* **Model-agnostic**: Supported on **all** chat‑capable models on Friendli.
* **High schema fidelity**: Generates outputs that reliably conform to your provided schemas.

## What Is Structured Outputs

Structured Outputs ensures LLMs return predictable, machine‑readable results (e.g., JSON) instead of free‑form text. This is essential for workflows that require validation or downstream automation.

## Structured Outputs with FriendliAI

* **Schema‑aligned generation**: High‑accuracy adherence to your JSON Schema.
* **Flexible modes**: Select strict or loose JSON mode, or apply regex constraints as needed.
* **OpenAI compatible**: Use standard `response_format` options with OpenAI SDKs.

### Structured Outputs Parameters

| Type          | Description                                                         | Name at OpenAI                                                                                |
| ------------- | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `json_schema` | The model returns a JSON object that conforms to the given schema.  | [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#introduction) |
| `json_object` | The model can return any JSON object.                               | [JSON mode](https://platform.openai.com/docs/guides/structured-outputs#json-mode)             |
| `regex`       | The model returns a string that conforms to the given regex schema. | N/A                                                                                           |

### Supported JSON Schemas

We support **all seven standard JSON schema types** (`null`, `boolean`, `number`, `integer`, `string`, `object`, `array`). The supported JSON schema keywords are listed below.

<Tip>Using unsupported or unexpected JSON schema keywords may result in them being ignored, triggering an error, or causing undefined behavior.</Tip>

#### Type-Specific Keywords

* `integer`
  * `exclusiveMinimum`, `exclusiveMaximum`, `minimum`, `maximum` (Note: these are not supported in `number`)
* `string`
  * `pattern`
  * `format`
    * Supported values: `uuid`, `date-time`, `date`, `time`, `uri`
* `object`
  * `properties`
  * `additionalProperties` is ignored, and is always set to `False`.
  * `required`: We support both required and optional properties, but have these limitations:
    * The sequence of the properties is fixed.
    * The first property should be `required`. If not, the first required property is moved to the first position.
* `array`
  * `items`
  * `minItems`: We support only `0` or `1` for `minItems`.

#### Constant Values and Enumerated Values

`const` and `enum` only support constant values of `null`, `boolean`, `number`, and `string`.

#### Schema Composition

We support only `anyOf` for [schema composition](https://json-schema.org/understanding-json-schema/reference/combining).

#### Referencing Subschemas

We only support referencing (`$ref`) to "internal" subschemas. These subschemas must be defined within `$defs`, and the value of `$ref` must be a valid URI pointing to a subschema.

#### Annotations

JSON schema annotations such as `title` or `description` are accepted but ignored.

## Simple Example

<Tabs>
  <Tab title="json_schema">
    This example provides a step-by-step guide on how to create a structured output response in JSON format.
    We use Python and the `pydantic` library to define a schema for the output in this example.

    <Steps>
      <Step title="Define Object Schema">
        Define a schema that contains information about a dish.

        ```python theme={null}
        from pydantic import BaseModel

        class Result(BaseModel):
            dish: str
            cuisine: str
            calories: int
        ```
      </Step>

      <Step title="Asking the Model for Structured Output">
        Call structured output and use schema to structure the response.

        <CodeGroup>
          ```python {17-22} OpenAI Python SDK theme={null}
          import os
          from openai import OpenAI

          client = OpenAI(
              base_url="https://api.friendli.ai/serverless/v1",
              api_key=os.getenv("API_KEY"),
          )

          completion = client.chat.completions.create(
              model="zai-org/GLM-5.3",
              messages=[
                  {
                      "role": "user",
                      "content": "Suggest a popular Italian dish in JSON format.",
                  },
              ],
              response_format={
                  "type": "json_schema",
                  "json_schema": {
                      "schema": Result.model_json_schema(),
                  }
              }
          )
          ```

          ```bash {12-25} curl theme={null}
          curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
            -H "Authorization: Bearer $API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "model": "zai-org/GLM-5.3",
              "messages": [
                {
                  "role": "user",
                  "content": "Suggest a popular Italian dish in JSON format."
                }
              ],
              "response_format": {
                "type": "json_schema",
                "json_schema": {
                  "schema": {
                    "type": "object",
                    "properties": {
                      "dish": {"type": "string"},
                      "cuisine": {"type": "string"},
                      "calories": {"type": "integer"}
                    },
                    "required": ["dish", "cuisine", "calories"]
                  }
                }
              }
            }'
          ```
        </CodeGroup>
      </Step>

      <Step title="Using Structured Output Results">
        You can use the output in the following way.

        ```python theme={null}
        response = completion.choices[0].message.content
        print(response)
        ```

        The code output result is as follows.

        ```json Result: theme={null}
        {
            "dish": "Spaghetti Bolognese",
            "cuisine": "Italian",
            "calories": 540
        }
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="json_object">
    This example demonstrates how to generate an arbitrary JSON object response without a predefined schema.

    <Warning>
      In `json_object` mode, the response may start with `{` or `[` and can be any arbitrary JSON object (dictionary) or array. If you need predictable results, we recommend using `json_schema`.
    </Warning>

    <CodeGroup>
      ```bash curl theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H "Authorization: Bearer $API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {"role": "system", "content": "You MUST answer with JSON."},
            {"role": "user", "content": "Generate a lasagna recipe. (very short)"}
          ],
          "response_format": {"type": "json_object"}
        }'
      ```

      ```python OpenAI Python SDK theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
        base_url="https://api.friendli.ai/serverless/v1",
        api_key=os.getenv("API_KEY"),
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=[
          {"role": "system", "content": "You MUST answer with JSON."},
          {"role": "user", "content": "Generate a lasagna recipe. (very short)"},
        ],
        response_format={"type": "json_object"},
      )

      print(completion.choices[0].message.content)
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Regex">
    This example shows how to generate output that matches a specific regular expression pattern.

    <CodeGroup>
      ```bash curl theme={null}
      curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
        -H "Authorization: Bearer $API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "zai-org/GLM-5.3",
          "messages": [
            {
              "role": "user",
              "content": "조선 왕조의 첫번째 왕은 누구입니까 (Who is the first king of the Joseon Dynasty)?"
            }
          ],
          "response_format": {
            "type": "regex",
            "schema": "[\\n ,.?!0-9\\uac00-\\ud7af]*"
          }
        }'
      ```

      ```python OpenAI Python SDK theme={null}
      import os
      from openai import OpenAI

      client = OpenAI(
          base_url="https://api.friendli.ai/serverless/v1",
          api_key=os.getenv("API_KEY"),
      )

      completion = client.chat.completions.create(
          model="zai-org/GLM-5.3",
          messages=[
              {
                  "role": "user",
                  "content": "조선 왕조의 첫번째 왕은 누구입니까 (Who is the first king of the Joseon Dynasty)?",
              },
          ],
          # Korean characters and numbers are allowed in the response.
          response_format={"type": "regex", "schema": "[\n ,.?!0-9\uac00-\ud7af]*"},
      )

      print(completion.choices[0].message.content)
      ```
    </CodeGroup>
  </Tab>
</Tabs>

## Advanced Examples

For more advanced use cases, see our blog: [Structured Output for LLM Agents](https://friendli.ai/blog/structured-output-llm-agents).


# Account Suspension
Source: https://friendli.ai/docs/guides/suite/account-suspension

Find out why your Friendli Suite account was suspended, what access is restricted, and the steps to resolve billing or policy-related suspensions.

Your account may be suspended for the following reasons:

1. **Billing issue:** A payment for an outstanding balance could not be processed with your default payment method.
2. **Suspicious or abusive activity:** If the system detects suspicious usage or abuse, your account may be suspended without prior notice.

## What Happens When Your Account Is Suspended

When your account is suspended, the following restrictions apply:

* Model APIs access is disabled.
* Dedicated Endpoint creation/restart is blocked.
* All running Dedicated Endpoints are terminated.
* Access to most platform pages is restricted until the outstanding balance is cleared.
* Uploading or retrieving custom models is not available.

## Resolving Suspension

If your account is suspended due to failed payments:

1. Go to [**Friendli Suite > Team Settings > Billing**](https://friendli.ai/suite/~/setting/billing/overview).
2. Pay any outstanding invoices.

Once the outstanding balance is cleared, your account access will be restored automatically.

If your account suspension appears incorrect, [contact support](mailto:support@friendli.ai) for further assistance.


# Billing and Payments
Source: https://friendli.ai/docs/guides/suite/billing-payments

Understand Friendli Suite prepaid billing, manage payment methods, view invoices, and learn how credits are applied to usage.

Friendli Suite applies available [credits](/docs/guides/suite/credits) to your usage first. Teams use prepaid credits: if your team has no remaining credits, API requests stop until you add more credits.

## How Billing Works

1. Credits are applied to your usage first, based on the [order of consumption](/docs/guides/suite/credits#order-of-consumption).
2. If your team has no remaining credits, API requests stop until you add more credits, either manually or through Auto Recharge.

### Outstanding Balances

How an outstanding (negative) balance is handled:

* **Prepaid billing (default)**: If your balance reaches \$0 (or goes slightly negative due to asynchronous billing), API requests stop until you add credits.
  * Credits you add are applied to any outstanding balance first.
  * Want to use **postpaid billing** instead? [Contact us](mailto:support@friendli.ai).

## Accepted Payment Methods

* **Credit and debit cards:** All major credit and debit cards are accepted, excluding prepaid cards.
* **ACH:** Available upon request. [Contact us](mailto:support@friendli.ai) to set up ACH payments.
* **AWS Marketplace:** You can make payments directly through the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-o2qc23xrtfo36).

<Note>
  Splitting a single charge across multiple payment methods is not currently supported.
</Note>

## Managing Payment Methods

One of your registered payment methods must be set as the **default payment method**.

* You can add, remove, or change the default payment method **within the same type** at any time (for example, between two credit cards).
* You cannot delete the last remaining payment method.
* Switching the **type** of the default payment method (for example, from ACH to a credit card, or from AWS Marketplace to ACH) is not supported in self-service. [Contact us](mailto:support@friendli.ai) for assistance.

## Invoices

You can view and download all previous invoices and current usage from the **[Billing History](https://friendli.ai/suite/~/setting/billing/history)** tab in [**Friendli Suite > Team Settings > Billing**](https://friendli.ai/suite/~/setting/billing/overview).

You can add your business name or other details to invoices via Stripe's customer portal, which is accessible from the **[Profile](https://friendli.ai/suite/~/setting/billing/profile)** tab in [**Friendli Suite > Team Settings > Billing**](https://friendli.ai/suite/~/setting/billing/overview).


# Credits
Source: https://friendli.ai/docs/guides/suite/credits

Learn about Friendli Suite credit types, including promotional and purchased credits, their consumption order, expiration rules, and how to redeem promo codes.

## Credit Types

### Promotional Credits

* **Source:** Redeeming a promo code, etc.
* **Usage scope:** May be restricted to a specific product, model, or GPU.
* **Expiration:** Usually comes with an expiration date.

### Prepaid Credits

* **Source:** One-time purchase (\$10 minimum) or Auto Recharge
* **Usage scope:** No restrictions
* **Expiration:** None

<Note>
  Bulk Discounts: [contact sales](https://friendli.ai/contact) for volume pricing
</Note>

## Order of Consumption

Promotional credits are always consumed before prepaid credits. Among promotional credits, those with a limited usage scope and earlier expiration dates are consumed first.

1. Promotional credits with an expiration date and limited usage scope (if usage is applicable)
2. Promotional credits with an expiration date and unlimited usage scope
3. Promotional credits without an expiration date and limited usage scope (if usage is applicable)
4. Promotional credits without an expiration date and unlimited usage scope
5. Prepaid credits

## Auto Recharge

Auto Recharge keeps your prepaid account running without interruption by automatically topping up credits when your balance runs low. Configure it with two settings:

* **Recharge threshold:** When your credit balance falls below this amount, a recharge is triggered automatically.
* **Restore credit balance to:** The target balance to refill up to when a recharge is triggered.

With Auto Recharge enabled, FriendliAI replenishes your balance before it runs out, so you can keep using the platform without interruption.

## When You Have No Credits

* Purchase credits (\$10 minimum) to continue using the platform.
* Adding credits also unlocks [higher rate limits](/docs/guides/model-apis/rate-limits) for the Model APIs.

## Redeeming Promo Codes

To redeem a promo code:

1. Go to [**Friendli Suite > Team Settings > Billing**](https://friendli.ai/suite/~/setting/billing/overview).
2. Click the **Redeem Promo Code** button in the top-right corner.

<Frame>
  <img alt="Redeem Promo Code" />
</Frame>

Once you enter a valid code, the credits are applied immediately. Credits are team-based. All members in your team can use them.

<Note>
  Only team admins can access the Billing page.
</Note>

If your code is not accepted, check the following:

* **Expired code:** The promo code may have passed its expiration date.
* **Already redeemed:** A team member may have already used this code.
* **Invalid code:** Verify the code is entered correctly.
* **Limit reached:** All available uses for this code may have been claimed.

If you encounter any issues, [contact support](mailto:support@friendli.ai).

## Viewing Credit History

You can view all of your previous credit records from the **[Credit Grants](https://friendli.ai/suite/~/setting/billing/credit)** tab in [**Friendli Suite > Team Settings > Billing**](https://friendli.ai/suite/~/setting/billing/overview).


# Enterprise Plan
Source: https://friendli.ai/docs/guides/suite/enterprise-plan

FriendliAI's Enterprise plan with reserved GPUs, higher API rate limits, private deployments, and enterprise-grade security and support.

Built for teams that need **scale, control, and contractual guarantees**.

The FriendliAI Enterprise plan is **not a fixed bundle**.

It is a **contract-based framework** that enables specific capabilities based on your deployment, scale, and compliance requirements.

## Is Enterprise Right for You?

Enterprise is a good fit if your team needs:

* Guaranteed or reserved GPU capacity
* Higher or custom Model APIs rate limits
* Priority access to high-demand GPU types
* Private deployments (VPC or on-prem)
* Region-specific or compliance-driven infrastructure

Let's design an Enterprise setup that fits your requirements.

### What's Included with All Enterprise Plans

* Enterprise-grade security and compliance commitments

### Enabled by Contract

Select the capabilities you need across the following areas.

| **Scale & Reliability**                                                                                      | **Control & Deployment**                                                             | **Enterprise Commitments**                                                                          |
| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| • Custom Model APIs rate limits<br />• Priority access to high-demand GPU types<br />• Reserved GPU capacity | • Custom region deployments<br />• VPC deployments<br />• On-prem deployment options | • Dedicated support channels<br />• Named Customer Success ownership<br />• Custom commercial terms |

## Talk to Us

Enterprise plans are designed and priced based on your specific requirements.

[Contact us](https://friendli.ai/contact) to discuss capacity, deployment, and contractual options.


# Manage Your FriendliAI API Keys
Source: https://friendli.ai/docs/guides/suite/personal-api-keys

Create, rename, and revoke FriendliAI API keys in Friendli Suite to authenticate your API requests. Covers optional key names and expiration dates.

To authenticate your requests to the FriendliAI API, you must first create a FriendliAI API key. When you create an API key, you can optionally name it and set an expiration date. You can later rename your API key or revoke it.

If you haven't already, [sign up](https://auth.friendli.ai/sign-up) for a Friendli Suite account. Then, [sign in](https://auth.friendli.ai).

## Create an API Key

<Steps>
  <Step title="Open the API Keys Page">
    In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**. Friendli Suite opens the **API Keys** page.
  </Step>

  <Step title="Open the Create API Key Form">
    In the upper-right corner, click <Icon icon="plus" /> **Create API Key**.
  </Step>

  <Step title="Create Your API Key">
    1. (Optional) Name your API key.
    2. (Optional) Set an expiration date:
       1. Check **Set expiration date**.
       2. Select an option. For example, select **30 days**.
       3. If you selected **Custom**, select a date.
    3. Click **Create Key**.

    Friendli Suite creates your API key and displays it to you.
  </Step>

  <Step title="Copy Your API Key">
    To the right of your API key, click <Icon icon="copy" /> **Copy**.

    Save your API key to a secure location. You won't be able to see it again.
  </Step>
</Steps>

## Rename an API Key

<Steps>
  <Step title="Open the API Keys Page">
    In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**. Friendli Suite opens the **API Keys** page. Your API keys may be listed across multiple pages.
  </Step>

  <Step title="Find Your API Key">
    On the current page, try to find your API key. If your API key isn't on the current page, use the pagination buttons to navigate to the page it's on.
  </Step>

  <Step title="Open the Rename Form">
    To the right of your API key, click <Icon icon="ellipsis" /> **Actions**. Then, click **Rename**.
  </Step>

  <Step title="Rename Your API Key">
    Rename your API key. Then, click **Save**.
  </Step>
</Steps>

## Revoke an API Key

<Steps>
  <Step title="Open the API Keys Page">
    In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**. Friendli Suite opens the **API Keys** page. Your API keys may be listed across multiple pages.
  </Step>

  <Step title="Find Your API Key">
    On the current page, try to find your API key. If your API key isn't on the current page, use the pagination buttons to navigate to the page it's on.
  </Step>

  <Step title="Open the Revoke Form">
    To the right of your API key, click <Icon icon="ellipsis" /> **Actions**. Then, click **Revoke**.
  </Step>

  <Step title="Revoke Your API Key">
    Enter your API key's name. Then, click **Revoke**.
  </Step>
</Steps>


# Reserve GPUs
Source: https://friendli.ai/docs/guides/suite/reserved-gpus

Reserve GPU capacity on Friendli Suite for a set period, deploy endpoints on your reserved GPUs, then extend or shut them down before the reservation ends.

With [Friendli Dedicated Endpoints](/docs/guides/dedicated-endpoints/introduction), you can deploy any model on dedicated GPUs reserved for you. Reserve them ahead of time at a fixed price for a set period of time. When that period ends, FriendliAI reclaims the GPUs.

To understand how reservations are billed, see the following table:

| Charge                           | How It's Billed                                                                                                                                                                                                   |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Your reservation                 | A flat fee, paid upfront, for the GPU type, count, and period you commit to.                                                                                                                                      |
| GPU time within your reservation | Not billed. Your reservation covers it.                                                                                                                                                                           |
| GPU time beyond your reservation | Billed per second at [on-demand rates](/docs/guides/dedicated-endpoints/pricing#supported-instance-types). Prepaid teams buy credits upfront, and postpaid teams pay as the overage accumulates on their balance. |

You must be a team owner or admin to reserve GPUs. To reserve GPUs, complete the following steps:

<Steps>
  <Step title="Sign In to Your Friendli Suite Account">
    If you haven't already, [sign in](https://auth.friendli.ai). Friendli Suite opens your dashboard.
  </Step>

  <Step title="Open the GPUs Tab">
    In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, under **Team Settings**, click **[Reserved GPUs](https://friendli.ai/suite/~/setting/reserved-gpus/gpus)**. Friendli Suite opens the **Reserved GPUs** > **GPUs** tab.
  </Step>

  <Step title="Request GPUs">
    1. On the **GPUs** tab, find an in-stock option.
    2. Click **Request**. Friendli Suite opens the request form.
    3. Select the number of days you want to reserve the GPUs.
    4. Select when you want to start your reservation.
    5. Click **Submit**.

    FriendliAI reviews your request. To track its status, open the **Reserved GPUs** > **[My Reservations](https://friendli.ai/suite/~/setting/reserved-gpus/my-reservations)** tab.

    If you don't see an option that works for you, click **Get a custom quote** and complete the request form.
  </Step>

  <Step title="Pay for Your Reservation">
    If FriendliAI approves your request, pay with the payment method on file. To add or change a payment method, see [Billing and Payments](/docs/guides/suite/billing-payments#managing-payment-methods).

    Once your payment is confirmed, Friendli Suite assigns the capacity to your team, and you can use it from your chosen start date.
  </Step>

  <Step title="Deploy to Your Reserved GPUs">
    Create an endpoint and set its **[Instance type](/docs/guides/dedicated-endpoints/endpoints#what-you-can-configure)** to the GPU type and count you reserved. Your reservation covers the endpoint's GPU time.

    To learn more, see [QuickStart: Friendli Dedicated Endpoints](/docs/guides/dedicated-endpoints/quickstart).
  </Step>
</Steps>

Before your reservation ends, extend it or shut down the endpoints you no longer need. Otherwise, FriendliAI chooses which of your endpoints to scale down.


# Usage and Costs
Source: https://friendli.ai/docs/guides/suite/usage-and-costs

Retrieve team-level usage and cost data through the Friendli Suite API. Covers authentication, rate limits, and practical examples with cURL.

Friendli Suite exposes two administration endpoints that let you track consumption programmatically:

* **Cost** (`GET /v1/team/cost`) — rated dollar amounts for your team, grouped by billing line item.
* **Usage** (`GET /v1/team/usage`) — metered quantities such as token counts and GPU time, with filters by model, product type, and user.

Use the cost endpoint when you need dollar figures. Use the usage endpoint when you need raw consumption numbers, for example, to build a dashboard, do chargeback accounting, or compare token volume across models.

## Authentication

Both endpoints require a **Personal API key** (for example, `flp_XXX`) passed as a bearer token in the `Authorization` header.

If you do not already have one, generate it from **[Friendli Suite > Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys)**. For the full steps, see the [Manage Your FriendliAI API Keys](/docs/guides/suite/personal-api-keys) guide.

```bash theme={null}
export FRIENDLIAI_API_KEY="<FRIENDLIAI_API_KEY>"
```

### Specifying a Team

To target a specific team, pass the team ID in the `X-Friendli-Team` header. Omit it to use your default team.

```bash theme={null}
curl https://api.friendli.ai/v1/team/cost \
  -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
  -H "X-Friendli-Team: <team_id>"
```

## Rate Limits and Polling

Both endpoints return `429` if you call them too frequently. Wait at least 5 minutes between repeated calls.

## Cost API

`GET /v1/team/cost` returns daily cost buckets for a time range you specify. Each bucket contains the total cost in USD and, when grouped by line item, a breakdown by billing category.

### Parameters

| Parameter         | Type               | Required | Description                                                                                                                         |
| ----------------- | ------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `start_time`      | string (date-time) | Yes      | RFC 3339 timestamp in UTC. The time portion must be zeroed out (for example, `2026-01-01T00:00:00Z`). No earlier than one year ago. |
| `end_time`        | string (date-time) | Yes      | RFC 3339 timestamp in UTC, zeroed out. Must not be later than midnight UTC of the next day.                                         |
| `bucket_width`    | string             | No       | Width of each time bucket. Only `1d` is supported. Defaults to `1d`.                                                                |
| `limit`           | integer            | No       | Number of buckets to return. Range 1–35, default 7.                                                                                 |
| `page`            | string             | No       | Pagination cursor from the `next_page` field of a previous response.                                                                |
| `group_by`        | string             | No       | Group costs by field. Only `line_item` is supported.                                                                                |
| `X-Friendli-Team` | string (header)    | No       | Team ID to run the request as.                                                                                                      |

### Example: Get Daily Cost Totals

```bash theme={null}
curl --request GET \
  --url "https://api.friendli.ai/v1/team/cost?start_time=2026-06-01T00:00:00Z&end_time=2026-06-08T00:00:00Z" \
  --header "Authorization: Bearer $FRIENDLIAI_API_KEY"
```

```json expandable theme={null}
{
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "start_time": "2026-06-01T00:00:00Z",
      "end_time": "2026-06-02T00:00:00Z",
      "results": [
        {
          "total": "67.89123016"
        }
      ]
    },
    {
      "start_time": "2026-06-02T00:00:00Z",
      "end_time": "2026-06-03T00:00:00Z",
      "results": [
        {
          "total": "87.2309217066666666"
        }
      ]
    }
  ]
}
```

### Example: Cost Broken Down by Line Item

Pass `group_by=line_item` to see which billing categories drove the cost.

```bash theme={null}
curl --request GET \
  --url "https://api.friendli.ai/v1/team/cost?start_time=2026-06-01T00:00:00Z&end_time=2026-06-08T00:00:00Z&group_by=line_item" \
  --header "Authorization: Bearer $FRIENDLIAI_API_KEY"
```

```json expandable theme={null}
{
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "start_time": "2026-06-01T00:00:00Z",
      "end_time": "2026-06-02T00:00:00Z",
      "results": [
        {
          "total": "0.3575",
          "quantity": "0.09166666666666666",
          "unit_price": "390.0",
          "line_item": "depuldflrhkh24n, h100, gpu time cost"
        },
        {
          "total": "50.43951744",
          "quantity": "193.998144",
          "unit_price": "26.0",
          "line_item": "zai-org/GLM-5.3, cached input tokens cost"
        }
      ]
    }
  ]
}
```

## Usage API

`GET /v1/team/usage` returns usage buckets with metered quantities: request counts, token counts, and GPU time. You can filter and group by model, product type, and user.

### Parameters

| Parameter         | Type               | Required | Description                                                                                                                     |
| ----------------- | ------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `start_time`      | string (date-time) | Yes      | RFC 3339 timestamp in UTC. Must fall on a 5-minute boundary. No earlier than one year ago. Alignment depends on `bucket_width`. |
| `end_time`        | string (date-time) | Yes      | RFC 3339 timestamp in UTC on a 5-minute boundary. Its minute value must match `start_time`.                                     |
| `bucket_width`    | string             | No       | `1d`, `1h`, or `5m`. Defaults to `1d`.                                                                                          |
| `limit`           | integer            | No       | Number of buckets. Defaults and maximums depend on `bucket_width`: `1d` → 7 / max 31, `1h` → 24 / max 168, `5m` → 12 / max 288. |
| `page`            | string             | No       | Pagination cursor from `next_page`.                                                                                             |
| `group_by`        | array of strings   | No       | Group usage by one or more of: `model`, `product_type`, `user_id`.                                                              |
| `models`          | array of strings   | No       | Filter to specific model IDs or endpoint IDs.                                                                                   |
| `user_ids`        | array of strings   | No       | Filter to specific users.                                                                                                       |
| `product_types`   | array of strings   | No       | Filter to `model_apis` or `dedicated_endpoints`.                                                                                |
| `gpu_types`       | array of strings   | No       | Filter to specific GPU types. Implies `product_types=dedicated_endpoints`.                                                      |
| `X-Friendli-Team` | string (header)    | No       | Team ID to run the request as.                                                                                                  |

### Example: Daily Usage Totals

```bash theme={null}
curl --request GET \
  --url "https://api.friendli.ai/v1/team/usage?start_time=2026-06-01T00:00:00Z&end_time=2026-06-08T00:00:00Z&bucket_width=1d" \
  --header "Authorization: Bearer $FRIENDLIAI_API_KEY"
```

```json expandable theme={null}
{
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "start_time": "2026-06-01T00:00:00Z",
      "end_time": "2026-06-02T00:00:00Z",
      "results": [
        {
          "num_model_requests": 4230,
          "input_tokens": 203290924,
          "input_cached_tokens": 193998144,
          "output_tokens": 1110820,
          "gpu_usage": {
            "h100": 330
          },
          "processed_audio_length_ms": 0
        }
      ]
    }
  ]
}
```

### Example: Usage by Model and Product Type

Group by `model` and `product_type` to see which models consumed the most tokens across both Model APIs and Dedicated Endpoints.

```bash theme={null}
curl --request GET \
  --url "https://api.friendli.ai/v1/team/usage?start_time=2026-06-01T00:00:00Z&end_time=2026-06-08T00:00:00Z&bucket_width=1d&group_by=model&group_by=product_type" \
  --header "Authorization: Bearer $FRIENDLIAI_API_KEY"
```

```json expandable theme={null}
{
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "start_time": "2026-06-01T00:00:00Z",
      "end_time": "2026-06-02T00:00:00Z",
      "results": [
        {
          "product_type": "dedicated_endpoints",
          "model": "depuldflrhkh24n",
          "num_model_requests": 0,
          "input_tokens": 0,
          "input_cached_tokens": 0,
          "output_tokens": 0,
          "gpu_usage": {
            "h100": 330
          },
          "processed_audio_length_ms": 0
        },
        {
          "product_type": "model_apis",
          "model": "zai-org/GLM-5.3",
          "num_model_requests": 2651,
          "input_tokens": 203257788,
          "input_cached_tokens": 193998144,
          "output_tokens": 938369,
          "processed_audio_length_ms": 0
        }
      ]
    }
  ]
}
```

### Example: Usage by User

Group by `user_id` to break down consumption per user. This shows who is driving usage across both Model APIs and Dedicated Endpoints.

```bash theme={null}
curl --request GET \
  --url "https://api.friendli.ai/v1/team/usage?start_time=2026-06-01T00:00:00Z&end_time=2026-06-08T00:00:00Z&bucket_width=1d&group_by=user_id&group_by=product_type" \
  --header "Authorization: Bearer $FRIENDLIAI_API_KEY"
```

```json expandable theme={null}
{
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "start_time": "2026-06-01T00:00:00Z",
      "end_time": "2026-06-02T00:00:00Z",
      "results": [
        {
          "product_type": "dedicated_endpoints",
          "user_id": "AvBI1gXTuRux",
          "num_model_requests": 1568,
          "input_tokens": 32928,
          "input_cached_tokens": 0,
          "output_tokens": 167805,
          "gpu_usage": {
            "h100": 330
          },
          "processed_audio_length_ms": 0
        },
        {
          "product_type": "model_apis",
          "user_id": "YmM5eyQ3J23q",
          "num_model_requests": 501,
          "input_tokens": 47473901,
          "input_cached_tokens": 45342336,
          "output_tokens": 314746,
          "processed_audio_length_ms": 0
        }
      ]
    }
  ]
}
```

#### Who Is `user_id`?

The `user_id` field identifies who consumed the usage, but the attribution depends on the product type:

* **Model APIs**: The `user_id` is the owner of the Personal API key that made the inference request. If multiple team members use the same API key, all usage is attributed to that key's owner.
* **Dedicated Endpoints — token consumption**: The `user_id` is the owner of the Personal API key that sent the request, same as Model APIs.
* **Dedicated Endpoints — GPU time**: GPU time in the `gpu_usage` field is attributed to the user who **created the endpoint**, not the user who sent inference requests to it. If Alice creates an endpoint and Bob sends requests to it, the token usage appears under Bob's `user_id` but the GPU time appears under Alice's.

<Warning>
  When grouping by `user_id`, Dedicated Endpoints GPU time is attributed to the endpoint creator, not the API caller. If your chargeback model assumes per-request attribution, handle GPU time separately in your aggregation logic. Use `group_by=model` together with `group_by=user_id` to separate endpoint-level GPU time from token usage per user.
</Warning>


# Tool Calling
Source: https://friendli.ai/docs/guides/tool-calling

Use OpenAI-compatible tool calling on FriendliAI endpoints. Broad model support, strict schema enforcement, and parallel tool call examples.

FriendliAI provides OpenAI-compatible tool calling with two core guarantees:

* **Broad model coverage**: Works across most chat‑capable models. No custom parsers required.
* **High accuracy**: Ensures reliable tool-call responses that align with your provided schemas.

## What Is Tool Calling

Tool calling (also called function calling) connects LLMs to external systems, enabling real‑time data access and action execution—a capability essential for agentic workflows.

<Frame>
  <img alt="Function Calling" />
</Frame>

## Broad Model Coverage

FriendliAI supports tool calling for a wide range of open‑source and commercial models. \
You can browse available models on our [Models page](https://friendli.ai/models) and try them out with the Playground.

## Tool Calling with FriendliAI

### Tool Calling Parameters

To enable tool calling, use the `tools`, `tool_choice`, and `parallel_tool_calls` parameters.

| Parameter             | Description                                                            | Default |
| --------------------- | ---------------------------------------------------------------------- | ------- |
| `tools`               | The list of tool objects that define the functions the model can call. | -       |
| `tool_choice`         | Determines the tool calling behavior of the model.                     | `auto`  |
| `parallel_tool_calls` | Whether to let the model issue tool calls in parallel.                 | `True`  |

By default, the model decides whether to call a function and which one to use.
With the `tool_choice` parameter, you can explicitly instruct the model to use a specific function.

* `none`: Disable the use of tools.
* `auto`: Enable the model to decide whether to use tools and which ones to use.
* `required`: Force the model to use a tool, but the model chooses which one.
* Named tool choice: Force the model to use a specific tool. It must be in the following format:

  ```json theme={null}
  {
    "type": "function",
    "function": {
      "name": "get_current_weather" // The function name you want to specify
    }
  }
  ```

### Response Schema

FriendliAI follows the OpenAI function calling schema. Tool calls are returned in `choices[].message.tool_calls[]` with each item containing a `function.name` and JSON‑stringified `function.arguments`. After executing a tool, append a new message with role: `tool`, the matching `tool_call_id`, and the tool result in content.

## Simple Example

The example below walks through five steps:

1. Define a tool (`get_weather`) that retrieves weather information.

2. Ask a question that triggers tool use.

3. Let the model select the tool.

4. Execute the tool.

5. Generate the final answer using the tool result.

<a href="https://colab.research.google.com/github/friendliai/examples/blob/main/tutorials/function-calling/basic-function-calling.ipynb">
  <img alt="Open In Colab" />
</a>

<Steps>
  <Step title="Tool Definition">
    Define a function that the model can call (`get_weather`) with a JSON Schema.

    The function requires the following parameters:

    * `location`: The location to look up weather information for.
    * `date`: The date to look up weather information for.

    This definition is included in the `tools` array and passed to the model.

    ```python theme={null}
    tools = [
      {
          "type": "function",
          "function": {
              "name": "get_weather",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {"type": "string"},
                      "date": {"type": "string", "format": "date"}
                  },
              },
          },
      }
    ]
    ```
  </Step>

  <Step title="Calling the Model">
    When a user asks a question, this request is passed to the model as a `messages` array.
    For example, the request "What's the weather like in Paris today?" would be passed as:

    ```python theme={null}
    from datetime import datetime

    today = datetime.now()
    messages = [
        {"role": "system", "content": f"You are a helpful assistant. today is {today}."},
        {"role": "user", "content": "What's the weather like in Paris today?"}
    ]
    ```

    Call the model using the `tools` and `messages` defined above.

    <CodeGroup>
      ```python OpenAI Python SDK theme={null}
      import os
      from openai import OpenAI

      token = os.getenv("API_KEY") or "<YOUR_API_KEY>"

      client = OpenAI(
          base_url = "https://api.friendli.ai/serverless/v1",
          api_key = token
      )

      completion = client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=messages,
        tools=tools,
      )

      print(completion.choices[0].message.tool_calls)
      ```
    </CodeGroup>
  </Step>

  <Step title="Executing the Tool">
    The API caller runs the tool based on the function call information of the model.
    For example, the `get_weather` function is executed as follows:

    ```python theme={null}
    import json
    import random

    def get_weather(location: str, date: str):
        temperature = random.randint(60, 80)
        return {"temperature": temperature, "forecast": "sunny"}

    tool_call = completion.choices[0].message.tool_calls[0]

    tool_response = locals()[tool_call.function.name](**json.loads(tool_call.function.arguments))
    print(tool_response)
    ```

    ```python Result: theme={null}
    {'temperature': 65, 'forecast': 'sunny'}
    ```
  </Step>

  <Step title="Adding Tool Responses">
    Add the tool's response to the `messages` array and pass it back to the model.

    1. Append tool call information
    2. Append the tool's execution result

    This ensures the model has all the necessary information to generate a response.

    ```python theme={null}
    model_response = completion.choices[0].message

    # Append the response from the model
    messages.append(
        {
            "role": model_response.role,
            "tool_calls": [
                tool_call.model_dump()
                for tool_call in model_response.tool_calls
            ]
        }
    )

    # Append the response from the tool
    messages.append(
        {
            "role": "tool",
            "content": json.dumps(tool_response),
            "tool_call_id": tool_call.id
        }
    )

    print(json.dumps(messages, indent=2))
    ```
  </Step>

  <Step title="Generating the Final Response">
    The model generates the final response based on the tool's output:

    <CodeGroup>
      ```python OpenAI Python SDK theme={null}
      next_completion = client.chat.completions.create(
          model="zai-org/GLM-5.3",
          messages=messages,
          tools=tools
      )

      print(next_completion.choices[0].message.content)
      ```
    </CodeGroup>

    ```text Final output: theme={null}
    According to the forecast, it's going to be a sunny day in Paris with a temperature of 65 degrees.
    ```
  </Step>
</Steps>

## Advanced Examples

Follow these blog posts to learn more about how to use tool calling with FriendliAI:

* Building an AI Agent for Google Calendar (<Tooltip>[Part 1](https://friendli.ai/blog/ai-agent-google-calendar)</Tooltip> / <Tooltip>[Part 2](https://friendli.ai/blog/calendar-agent-vercel)</Tooltip>)
* Friendli Tools Blog Series (<Tooltip>[Part 1](https://friendli.ai/blog/llm-function-calling)</Tooltip> / <Tooltip>[Part 2](https://friendli.ai/blog/ai-agents-function-calling)</Tooltip> / <Tooltip>[Part 3](https://friendli.ai/blog/friendli-tools-llama3-outperforms-gpt4o)</Tooltip>)


# Build an Agent with Gradio
Source: https://friendli.ai/docs/guides/tutorials/build-an-agent-with-gradio

Build and deploy an AI agent with Friendli Model APIs and Gradio in under 50 lines of Python. Includes chat UI setup.

## Goals

* Build your own AI agent using [**Friendli Model APIs**](https://friendli.ai/product/model-apis) and [**Gradio**](https://www.gradio.app) in less than 50 LoC
* Share your AI agent with the world and gather feedback

> [**Gradio**](https://www.gradio.app) is the fastest way to demo your model with a friendly web interface.

## Getting Started

1. Go to [**Friendli Suite**](https://friendli.ai/suite), and create an account.
2. Grab a [Personal API key](https://friendli.ai/suite/~/setting/keys) to use Friendli Model APIs within an agent.

## Step 1. Prerequisite

Install dependencies.

```bash theme={null}
pip install openai gradio
```

## Step 2. Launch Your Agent

Build your own AI agent using **Friendli Model APIs** and **Gradio**.

* Gradio provides a `ChatInterface` that implements a chatbot UI running the `chat_function`.
  * More information about the *chat\_function(message, history)*
    > *The input function should accept two parameters: a string input message and list of two-element lists of the form \[\[user\_message, bot\_message], ...] representing the chat history, and return a string response.*
* Implement the `chat_function` using Friendli Model APIs.
  * Here, we used the `zai-org/GLM-5.3` model.
  * Feel free to explore [other available models](https://friendli.ai/models?products=SERVERLESS).

```python theme={null}
from openai import OpenAI
import gradio as gr

friendli_client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key="YOUR_API_KEY"
)

def chat_function(message, history):
    messages = []
    for user, chatbot in history:
        messages.append({"role" : "user", "content": user})
        messages.append({"role" : "assistant", "content": chatbot})
    messages.append({"role": "user", "content": message})

    stream = friendli_client.chat.completions.create(
        model="zai-org/GLM-5.3",
        messages=messages,
        stream=True
    )
    res = ""
    for chunk in stream:
        res += chunk.choices[0].delta.content or ""
        yield res

css = """
.gradio-container {
    max-width: 800px !important;
    margin-top: 100px !important;
}

.pending {
    display: none !important;
}

.sm {
    box-shadow: None !important;
}

#component-2 {
    height: 400px !important;
}
"""

with gr.Blocks(theme=gr.themes.Soft(), css=css) as friendli_agent:
    gr.ChatInterface(chat_function)

friendli_agent.launch()
```

## Step 3. Deploy Your Agent

For the temporary deployment, change the last line of the code.

```python theme={null}
friendli_agent.launch(share=True)
```

For the permanent deployment, you can use [Hugging Face Spaces](https://huggingface.co/spaces)!


# Build an Agent with LangChain
Source: https://friendli.ai/docs/guides/tutorials/build-an-agent-with-langchain

Create an AI agent using LangChain and Friendli Model APIs with tool calling. Step-by-step tutorial with code examples in Python.

## Introduction

This tutorial walks you through creating an Agent using LangChain and Model APIs.

## Setup

```bash theme={null}
pip install -qU langchain-openai langchain-community langchain wikipedia
```

Get your [Personal API key](https://friendli.ai/suite/~/setting/keys) to use Friendli Model APIs.

```python theme={null}
import getpass
import os

if not os.environ.get("API_KEY"):
    os.environ["API_KEY"] = getpass.getpass("Enter your Personal API key: ")
```

## Instantiation

```python theme={null}
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="zai-org/GLM-5.3",
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["API_KEY"],
)
```

## Create Agent with LangChain

### Step 1. Create Tool

```python theme={null}
from langchain_community.tools import WikipediaQueryRun
from langchain_community.utilities import WikipediaAPIWrapper

api_wrapper = WikipediaAPIWrapper(top_k_results=1, doc_content_chars_max=100)
wiki = WikipediaQueryRun(api_wrapper=api_wrapper)
tools = [wiki]
```

### Step 2. Create Prompt

```python theme={null}
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        MessagesPlaceholder("chat_history"),
        ("user", "{input}"),
        ("placeholder", "{agent_scratchpad}"),
    ]
)

prompt.messages
```

### Step 3. Create Agent

```python theme={null}
from langchain.agents import AgentExecutor
from langchain.agents import create_tool_calling_agent

agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
```

### Step 4. Run the Agent

```python theme={null}
chat_history = []

while True:
    user_input = input("Enter your message: ")
    result = agent_executor.invoke(
        {"input": user_input, "chat_history": chat_history},
    )
    chat_history.append({"role": "user", "content": user_input})
    chat_history.append({"role": "assistant", "content": result["output"]})
```

When you run the code, it will wait for your input.
After inputting, it will wait and output the result.
When you ask a question about a specific Wikipedia topic, it will automatically call the Wikipedia tool and output the result.

```text final result theme={null}
Enter your Personal API key: ··········
Enter your message: hello

> Entering new AgentExecutor chain...
Hello, it's nice to meet you. I'm here to help with any questions or topics you'd like to discuss. Is there something in particular you'd like to talk about, or do you need assistance with something?

> Finished chain.
Enter your message: What does the Linux kernel do?

> Entering new AgentExecutor chain...

Invoking: `wikipedia` with `{'query': 'Linux kernel'}`
responded: The Linux kernel is the core component of the Linux operating system. It acts as a bridge between the computer hardware and the user space applications. The kernel manages the system's hardware resources, such as memory, CPU, and I/O devices. It provides a set of interfaces and APIs that allow user space applications to interact with the hardware.

Page: Linux kernel
Summary: The Linux kernel is a free and open source,: 4  UNIX-like kernel that isThe Linux kernel is a free and open source, UNIX-like kernel that is responsible for managing the system's hardware resources, such as memory, CPU, and I/O devices. It provides a set of interfaces and APIs that allow user space applications to interact with the hardware. The kernel is the core component of the Linux operating system, and it plays a crucial role in ensuring the stability and security of the system.

> Finished chain.
Enter your message:
```

## Full Example Code

```python theme={null}
import getpass
import os

from langchain_openai import ChatOpenAI

from langchain_community.tools import WikipediaQueryRun
from langchain_community.utilities import WikipediaAPIWrapper

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

from langchain.agents import AgentExecutor
from langchain.agents import create_tool_calling_agent

if not os.environ.get("API_KEY"):
    os.environ["API_KEY"] = getpass.getpass("Enter your Personal API key: ")

llm = ChatOpenAI(
    model="zai-org/GLM-5.3",
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.environ["API_KEY"],
)

api_wrapper = WikipediaAPIWrapper(top_k_results=1, doc_content_chars_max=100)
wiki = WikipediaQueryRun(api_wrapper=api_wrapper)
tools = [wiki]

# Get the prompt to use - you can modify this!
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        MessagesPlaceholder("chat_history"),
        ("user", "{input}"),
        ("placeholder", "{agent_scratchpad}"),
    ]
)

agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

chat_history = []

while True:
    user_input = input("Enter your message: ")
    result = agent_executor.invoke(
        {"input": user_input, "chat_history": chat_history},
    )
    chat_history.append({"role": "user", "content": user_input})
    chat_history.append({"role": "assistant", "content": result["output"]})
```


# Getting Started with EXAONE 4.0
Source: https://friendli.ai/docs/guides/tutorials/getting-started-with-exaone-4.0

Deploy and run LG AI Research's EXAONE 4.0 models on Friendli Dedicated Endpoints. Covers authentication, inference, reasoning, and optimization.

As an official launch partner with [LG AI Research](https://www.lgresearch.ai/), [FriendliAI](https://friendli.ai/) provides full Day 0 support for the new EXAONE 4.0 models on [Dedicated Endpoints](https://friendli.ai/product/dedicated-endpoints). This guide walks you through how to run the EXAONE 4.0 models on FriendliAI. You will learn how to authenticate, send inference requests, configure model parameters, and optimize for both performance and cost. You will also learn how to enable or disable reasoning at request time.

## Introduction

[LGAI-EXAONE/EXAONE-4.0.1-32B](https://huggingface.co/LGAI-EXAONE/EXAONE-4.0.1-32B) is the latest evolution in LG AI Research's EXAONE series. Optimized for real-world reasoning, generation, and enterprise applications, EXAONE 4.0 brings advanced capabilities across a wide range of use cases, from intelligent agents to enterprise automation and research.

To learn more, check out our [partnership announcement](https://friendli.ai/blog/lg-ai-research-partnership-exaone-4.0) and [LG AI Research's EXAONE page](https://www.lgresearch.ai/exaone/).

## Overview

[Friendli Dedicated Endpoints](https://friendli.ai/product/dedicated-endpoints) provide:

* High-throughput with consistent, guaranteed performance
* 50%+ GPU savings
* Full control over GPU resources, 99.99% availability
* Ideal for high-volume production applications and long-running services

## Prerequisites

Before you get started, ensure that you have the following:

1. A FriendliAI account

Sign up or sign in at [Friendli Suite](https://friendli.ai/suite).

2. A Personal API key

You can create and manage API keys in: [Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).

<RoundedBorderBox>
  <img alt="Personal API Keys" />
</RoundedBorderBox>

Set your key as an environment variable:

```shell theme={null}
export API_KEY="YOUR API KEY HERE"
```

3. Install the OpenAI SDK

```shell theme={null}
pip install openai
```

## Run EXAONE 4.0 on Dedicated Endpoints for Maximum Performance

[Friendli Dedicated Endpoints](/docs/guides/dedicated-endpoints/introduction) give you full control over deployment, scaling, and hardware selection, making them ideal for production workloads and mission-critical applications.

1. ### Go to [Dedicated Endpoint creation page](https://friendli.ai/suite/~/dedicated-endpoints/create).

<RoundedBorderBox>
  <img alt="Create Endpoint" />
</RoundedBorderBox>

2. ### Select your base model and multi-LoRA adapters.

Search for the model you want to deploy. For EXAONE 4.0 models, just type "EXAONE-4.0" in the search bar and select the variant you want.

You can also apply as many [Multi-LoRA Adapters](https://friendli.ai/blog/how-to-use-hugging-face-multi-lora-adapters) as you want.

<RoundedBorderBox>
  <img alt="Select Model" />
</RoundedBorderBox>

You can deploy any of the [610,000+ supported models](https://friendli.ai/models), including your own custom fine-tuned models. Deploy directly from [our Models page](https://friendli.ai/models), [Hugging Face](https://friendli.ai/blog/huggingface-partnership) model repositories.

3. ### Configure endpoint features.

Here you can customize your endpoint to:

* Set [Online Quantization](https://friendli.ai/blog/online-quantization) for higher throughput and a lower number of GPUs.
* Enable [N-gram Speculative Decoding](https://friendli.ai/blog/n-gram-speculative-decoding) for faster Time-per-Output-Token (TPOT).

<RoundedBorderBox>
  <img alt="Endpoint Features" />
</RoundedBorderBox>

4. ### Select the GPU type.

<RoundedBorderBox>
  <img alt="Select GPU Type" />
</RoundedBorderBox>

5. ### Customize autoscaling parameters.

FriendliAI lets you customize the autoscaling parameters to tailor endpoints to your workloads.

<RoundedBorderBox>
  <img alt="Autoscaling Config" />
</RoundedBorderBox>

6. ### Configure the inference engine.

Customize the inference engine to match your application's requirements. These settings control how the model processes inputs, handles tokens, and logs requests during use. You can:

* Add special tokens
* Skip special tokens
* Set maximum batch size
* Log request content

<RoundedBorderBox>
  <img alt="Engine Config" />
</RoundedBorderBox>

7. ### Deploy.

Click **Deploy** to deploy your Dedicated Endpoint and start using your model.

8. ### Send inference requests.

You can immediately try the model on Dedicated Endpoints in Playground, which provides a chat-style interface for quick experimentation. In Playground, you can set a system prompt and adjust parameters such as token length, temperature, Top P, and frequency penalty.

<RoundedBorderBox>
  <img alt="Playground" />
</RoundedBorderBox>

You can also start sending API requests right away. Copy the endpoint ID from the endpoint overview page into the `model` field in your requests.

<RoundedBorderBox>
  <img alt="Endpoint ID" />
</RoundedBorderBox>

Example code using the OpenAI SDK:

```py theme={null}
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("API_KEY"),
    base_url="https://api.friendli.ai/dedicated/v1",
)

stream = client.chat.completions.create(
    model="your-dedicated-endpoint-id",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

Example `curl` script:

```shell theme={null}
curl -X POST https://api.friendli.ai/dedicated/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-dedicated-endpoint-id",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ],
    "stream": true
  }'
```

9. ### Monitor the endpoint behavior.

Real-time metrics and logs give you immediate visibility into system behavior, making it far easier to understand and resolve issues quickly.

* You can view full metrics and request activities.
  * Monitor real-time throughput, latency, tokens processed, and replica counts over time.
  * Review request activity and troubleshoot issues more quickly.

<RoundedBorderBox>
  <img alt="Metrics" />
</RoundedBorderBox>

* View specific request and response content (when explicitly enabled).
  * Get a clearer view of how the model is behaving.
  * Spot and investigate requests that may require attention.

<RoundedBorderBox>
  <img alt="Logs" />
</RoundedBorderBox>

## Enabling and Disabling Reasoning

EXAONE 4.0 models support an explicit reasoning (or "thinking") mode, which allows the model to internally reason step-by-step before producing a final answer. You can enable or disable this behavior at request time, depending on whether you want maximum reasoning quality or fast, deterministic responses.

<Info>
  By default, EXAONE 4.0 does not use reasoning when the `enable_thinking` parameter is not specified.
</Info>

### When to Enable Reasoning

Enable reasoning when:

* You want higher-quality answers for complex or open-ended questions.
* Creativity and exploration are more important than determinism.
* Higher latency or token usage is acceptable.

When reasoning is enabled:

* Use temperature=1.0 and top\_p=1.0 for best performance.

```py theme={null}
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("API_KEY"),
    base_url="https://api.friendli.ai/dedicated/v1",
)

stream = client.chat.completions.create(
    model="your-dedicated-endpoint-id",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
    temperature=1.0,
    top_p=1.0,
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

### When to Disable Reasoning

Disable reasoning when:

* You want fast, predictable, and deterministic outputs.
* The task is simple (e.g., classification, extraction, short factual answers).
* You want minimal token usage.

When reasoning is disabled:

* Set enable\_thinking=False.
* Use temperature=0 for deterministic behavior.
* top\_p can be omitted or left at its default.

```py theme={null}
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("API_KEY"),
    base_url="https://api.friendli.ai/dedicated/v1",
)

stream = client.chat.completions.create(
    model="your-dedicated-endpoint-id",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    temperature=0,
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

Choosing the right mode lets you balance answer quality, determinism, latency, and cost for your specific use case.

## Conclusion

EXAONE 4.0 brings a new level of capabilities across a wide range of use cases, from intelligent agents to enterprise automation and research. FriendliAI makes it easy to deploy them from Day 0. With full control and maximum efficiency through Dedicated Endpoints, FriendliAI provides the infrastructure, tooling, and reliability needed to build and scale production-grade AI systems. With flexible configuration options, seamless deployment workflows, and real-time observability, you can confidently bring AI-powered applications to life and optimize them for your team's workflow, performance needs, and budget.

Ready to get started? Sign in to [Friendli Suite](https://friendli.ai/suite) and launch your first EXAONE 4.0 deployment today.


# Getting Started with Nemotron 3
Source: https://friendli.ai/docs/guides/tutorials/getting-started-with-nemotron-3

Deploy and run NVIDIA Nemotron 3 on Friendli Dedicated Endpoints. Includes setup, API usage, chat completions, and performance benchmarks.

<RoundedBorderBox>
  <img alt="Nemotron 3 Tutorial" />
</RoundedBorderBox>

As an official launch partner for NVIDIA Nemotron 3 Nano and Nemotron 3 Super, [FriendliAI](https://friendli.ai/) provides high-performance deployment options for the Nemotron 3 family on [Friendli Dedicated Endpoints](https://friendli.ai/product/dedicated-endpoints). This guide focuses on how to run Nemotron 3 models on FriendliAI. You will learn how to authenticate, deploy the model, send inference requests, configure model parameters, and optimize for both performance and cost. You will also learn how to enable or disable reasoning at request time.

If you are exploring which Nemotron model to deploy, visit our [Nemotron landing page](https://friendli.ai/models/nemotron), where you can browse supported Nemotron models on FriendliAI and jump directly into deployment options.

## Introduction

Nemotron 3 is a family of **high-performance**, **high-efficiency** foundation models designed for **agentic AI applications**. It uses a **hybrid Mamba–Transformer Mixture-of-Experts (MoE)** architecture and supports **1M-token context windows**. With Nemotron 3, you can build reliable, high-throughput agents that operate across complex workflows, multi-document reasoning, and long-duration tasks.

FriendliAI participated in the Nemotron 3 Nano launch and now provides Day 0 support for Nemotron 3 Super. While Nano is optimized for efficient targeted workloads, Nemotron 3 Super is purpose-built for more advanced multi-agent systems, complex tool-calling, and production-scale agentic AI workloads. This tutorial covers deploying the Nemotron 3 model family on Friendli Dedicated Endpoints.

To learn more, check out [our Nemotron 3 Nano launch announcement](https://friendli.ai/blog/nvidia-nemotron-3-partnership), [our Nemotron 3 Super launch announcement](https://friendli.ai/blog/nvidia-nemotron-3-super), and [NVIDIA's Nemotron 3 page](https://developer.nvidia.com/blog/inside-nvidia-nemotron-3-techniques-tools-and-data-that-make-it-efficient-and-accurate).

## Friendli Dedicated Endpoints

[Friendli Dedicated Endpoints](/docs/guides/dedicated-endpoints/introduction) give you full control over deployment, scaling, and hardware selection, making them ideal for production workloads and mission-critical applications.

* High-throughput with consistent, guaranteed performance
* 50%+ GPU savings
* Full control over GPU resources, 99.99% availability
* Ideal for high-volume production applications, long-running services, and advanced agentic workloads such as multi-agent systems and complex tool-calling

## Prerequisites

Before you get started, ensure that you have the following:

1. A FriendliAI account

Sign up or sign in at [Friendli Suite](https://friendli.ai/suite).

2. A Personal API key

You can create and manage API keys in: [Personal Settings > API Keys](https://friendli.ai/suite/~/setting/keys).

<RoundedBorderBox>
  <img alt="Personal API Keys" />
</RoundedBorderBox>

Set your key as an environment variable:

```shell theme={null}
export API_KEY="YOUR API KEY HERE"
```

3. Install the OpenAI SDK

```shell theme={null}
pip install openai
```

## Run Nemotron 3 on Dedicated Endpoints for Maximum Performance

1. ### Go to [Dedicated Endpoint creation page](https://friendli.ai/suite/~/dedicated-endpoints/create).

<RoundedBorderBox>
  <img alt="Create Endpoint" />
</RoundedBorderBox>

2. ### Select your base model and multi-LoRA adapters.

Search for the model you want to deploy. For Nemotron 3 models, type `Nemotron-3` in the search bar and select the variant you want to run.

<RoundedBorderBox>
  <img alt="Select Model" />
</RoundedBorderBox>

You can deploy any of the [610,000+ supported models](https://friendli.ai/models), including your own custom fine-tuned models. Deploy directly from [our Models page](https://friendli.ai/models), [Hugging Face](https://friendli.ai/blog/huggingface-partnership) model repositories.

3. ### Select the GPU type.

Select the GPU type that best matches your latency, throughput, and budget requirements for your Nemotron 3 deployment.

<RoundedBorderBox>
  <img alt="Select GPU Type" />
</RoundedBorderBox>

4. ### Customize autoscaling parameters.

FriendliAI lets you customize the autoscaling parameters to tailor endpoints to your workloads.

<RoundedBorderBox>
  <img alt="Autoscaling Config" />
</RoundedBorderBox>

5. ### Configure the inference engine.

Customize the inference engine to match your application's requirements. You can set maximum batch size for beta models.

<RoundedBorderBox>
  <img alt="Engine Config" />
</RoundedBorderBox>

6. ### Deploy.

Click **Deploy** to deploy your Dedicated Endpoint and start using your model.

7. ### Send inference requests.

As with Model APIs, you can immediately try the model on Dedicated Endpoints in Playground, which provides a chat-style interface for quick experimentation. You can also customize the system prompt and tune various parameters to explore different behaviors and response styles.

<RoundedBorderBox>
  <img alt="Playground" />
</RoundedBorderBox>

You can also start sending API requests right away. Copy the endpoint ID from the endpoint overview page into the `model` field in your requests.

<RoundedBorderBox>
  <img alt="Endpoint ID" />
</RoundedBorderBox>

Example code using the OpenAI SDK:

```py theme={null}
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("API_KEY"),
    base_url="https://api.friendli.ai/dedicated/v1",
)

stream = client.chat.completions.create(
    model="your-dedicated-endpoint-id",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Summarize the architectural advantages of Nemotron 3 Super for long-context, tool-using agent workflows."},
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

Example `curl` script:

```shell theme={null}
curl -X POST https://api.friendli.ai/dedicated/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-dedicated-endpoint-id",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Summarize the architectural advantages of Nemotron 3 Super for long-context, tool-using agent workflows."
      }
    ],
    "stream": true
  }'
```

8. ### Monitor the endpoint behavior.

Real-time metrics and logs give you immediate visibility into system behavior, making it far easier to understand and resolve issues quickly.

* You can view full metrics and request activities.
  * Monitor real-time throughput, latency, tokens processed, and replica counts over time.
  * Review request activity and troubleshoot issues more quickly.

<RoundedBorderBox>
  <img alt="Metrics" />
</RoundedBorderBox>

* View specific request and response content (when explicitly enabled).
  * Get a clearer view of how the model is behaving.
  * Spot and investigate requests that may require attention.

<RoundedBorderBox>
  <img alt="Logs" />
</RoundedBorderBox>

## Enabling and Disabling Reasoning

Nemotron 3 models support an explicit reasoning (or "thinking") mode, which allows the model to internally reason step-by-step before producing a final answer. You can enable or disable this behavior at request time, depending on whether you want maximum reasoning quality or fast, deterministic responses.

<Info>
  By default, Nemotron 3 uses reasoning when the `enable_thinking` parameter is not specified.
</Info>

### When to Enable Reasoning

Enable reasoning when:

* You want higher-quality answers for complex or open-ended questions.
* Creativity and exploration are more important than determinism.
* Higher latency or token usage is acceptable.

When reasoning is enabled:

* Use temperature=1.0 and top\_p=1.0 for best performance.

```py theme={null}
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("API_KEY"),
    base_url="https://api.friendli.ai/dedicated/v1",
)

stream = client.chat.completions.create(
    model="your-dedicated-endpoint-id",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Summarize the architectural advantages of Nemotron 3 Super for long-context, tool-using agent workflows."},
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
    temperature=1.0,
    top_p=1.0,
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

### When to Disable Reasoning

Disable reasoning when:

* You want fast, predictable, and deterministic outputs.
* The task is simple (e.g., classification, extraction, short factual answers).
* You want minimal token usage.

When reasoning is disabled:

* Set enable\_thinking=False.
* Use temperature=0 for deterministic behavior.
* top\_p can be omitted or left at its default.

```py theme={null}
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("API_KEY"),
    base_url="https://api.friendli.ai/dedicated/v1",
)

stream = client.chat.completions.create(
    model="your-dedicated-endpoint-id",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    temperature=0,
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

Choosing the right mode lets you balance answer quality, determinism, latency, and cost for your specific use case.

## Conclusion

Nemotron 3 unlocks a new generation of high-performance, long-context, agent-ready AI capabilities, and FriendliAI makes it fast and easy to deploy them.

As an official launch partner for Nemotron 3 models, FriendliAI provides the infrastructure, tooling, and reliability needed to build and scale production-grade AI systems across the Nemotron 3 family. With flexible configuration options, seamless deployment workflows, and real-time observability, you can confidently bring Nemotron-powered applications to life and optimize them for your team's workflow, performance needs, and budget.

Ready to get started? Sign in to [Friendli Suite](https://friendli.ai/suite) and launch your first Nemotron 3 deployment today.


# Tool Calling with Model APIs
Source: https://friendli.ai/docs/guides/tutorials/tool-calling-with-model-apis

Implement tool calling with Friendli Model APIs. Tutorial covers defining tools, handling function calls, and building multi-turn agent loops.

## Goals

* Use tool calling to build your own AI agent with [**Friendli Model APIs**](https://friendli.ai/product/model-apis)
* Feel free to make your own custom tools!

## Getting Started

1. Go to [**Friendli Suite**](https://friendli.ai/suite), and create an account.
2. Grab a [Personal API key](https://friendli.ai/suite/~/setting/keys) to use Friendli Model APIs within an agent.

## Step 1. Playground UI

Experience tool calling on the Playground!

<RoundedBorderBox>
  <img alt="Sidebar" />
</RoundedBorderBox>

<br />

<RoundedBorderBox>
  <img alt="Web Search Tool" />
</RoundedBorderBox>

1. On your left sidebar, click the **Model APIs** option to access the playground page.
2. You will see the models available as Model APIs. Select the one you want and click the endpoint.
3. Click the **Tools** button, select the Search tool, and enter a query to see the response. 😀

## Step 2. Build a Custom Tool

Build your own creative tool. This section shows you how to make a custom tool that retrieves temperature information. (The completed code snippet is at the bottom.)

1. **Define a function for using as a custom tool**

```python theme={null}
def get_temperature(location: str) -> int:
    """Mock function that returns the city temperature"""
    if "new york" in location.lower():
        return 45
    if "san francisco" in location.lower():
        return 72
    return 30
```

2. **Send a function calling inference request**

   1. Add your input as a `user` role message.
   2. The information about the custom function (e.g., `get_temperature`) goes into the tools option. JSON schema describes the function's parameters.
   3. The response includes the `arguments` field, which contains values extracted from the user's input that can be used as parameters of the custom function.

   ```python theme={null}
   # pip install openai

   import os
   from openai import OpenAI

   token = os.environ.get("API_KEY") or "YOUR_API_KEY"
   client = OpenAI(
       api_key=token,
       base_url="https://api.friendli.ai/serverless/v1",
   )
   user_prompt = "I live in New York. What should I wear for today's weather?"

   messages = [
       {
           "role": "user",
           "content": user_prompt,
       },
   ]

   tools=[
       {
           "type": "function",
           "function": {
               "name": "get_temperature",
               "description": "Get the temperature information in a given location.",
               "parameters": {
                   "type": "object",
                   "properties": {
                       "location": {
                           "type": "string",
                           "description": "The name of current location e.g., New York",
                       },
                   },
               },
           },
       },
   ]

   chat = client.chat.completions.create(
       model="zai-org/GLM-5.3",
       messages=messages,
       tools=tools,
       temperature=0,
       frequency_penalty=1,
   )

   print(chat)
   ```

3. **Generate the final response using the tool calling results**

   1. Add the `tool_calls` response as an `assistant` role message.
   2. Add the result obtained by calling the `get_temperature` function as a `tool` message to the Chat API again.

   ```python theme={null}
   import json

   func_kwargs = json.loads(chat.choices[0].message.tool_calls[0].function.arguments)
   temperature_info = get_temperature(**func_kwargs)

   messages.append(
       {
           "role": "assistant",
           "tool_calls": [
               tool_call.model_dump()
               for tool_call in chat.choices[0].message.tool_calls
           ]
       }
   )
   messages.append(
       {
           "role": "tool",
           "content": str(temperature_info),
           "tool_call_id": chat.choices[0].message.tool_calls[0].id
       }
   )

   chat_w_info = client.chat.completions.create(
       model="zai-org/GLM-5.3",
       tools=tools,
       messages=messages,
   )

   for choice in chat_w_info.choices:
       print(choice.message.content)
   ```

* **Complete Code Snippet**

  ```python theme={null}
  # pip install openai

  import json
  import os
  from openai import OpenAI

  token = os.environ.get("API_KEY") or "YOUR_API_KEY"
  client = OpenAI(
      api_key=token,
      base_url="https://api.friendli.ai/serverless/v1",
  )
  user_prompt = "I live in New York. What should I wear for today's weather?"

  messages = [
      {
          "role": "user",
          "content": user_prompt,
      },
  ]

  tools=[
      {
          "type": "function",
          "function": {
              "name": "get_temperature",
              "description": "Get the temperature information in a given location.",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The name of current location e.g., New York",
                      },
                  },
              },
          },
      },
  ]

  chat = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      messages=messages,
      tools=tools,
      temperature=0,
      frequency_penalty=1,
  )

  def get_temperature(location: str) -> int:
      """Mock function that returns the city temperature"""
      if "new york" in location.lower():
          return 45
      if "san francisco" in location.lower():
          return 72
      return 30

  func_kwargs = json.loads(chat.choices[0].message.tool_calls[0].function.arguments)
  temperature_info = get_temperature(**func_kwargs)

  messages.append(
      {
          "role": "assistant",
          "tool_calls": [
              tool_call.model_dump()
              for tool_call in chat.choices[0].message.tool_calls
          ]
      }
  )
  messages.append(
      {
          "role": "tool",
          "content": str(temperature_info),
          "tool_call_id": chat.choices[0].message.tool_calls[0].id
      }
  )

  chat_w_info = client.chat.completions.create(
      model="zai-org/GLM-5.3",
      tools=tools,
      messages=messages,
  )

  for choice in chat_w_info.choices:
      print(choice.message.content)
  ```

## Congratulations

Following the above instructions, we've experienced the whole process of defining and using a custom tool to generate an accurate and rich answer from LLM models!

Brainstorm creative ideas for your agent by reading our blog articles!

* [**Building an AI Agent for Google Calendar**](https://friendli.ai/blog/ai-agent-google-calendar)
* [**Building AI Agents Using Function Calling with LLMs**](https://friendli.ai/blog/ai-agents-function-calling)
* [**Function Calling: Connecting LLMs with Functions and APIs**](https://friendli.ai/blog/llm-function-calling)


# FriendliAI Docs
Source: https://friendli.ai/docs/index

FriendliAI provides fast, affordable, and reliable AI inference at scale. Explore guides, API reference, examples, and product quickstarts.

<div>
  <div>
    <section>
      <div>
        <h1>FriendliAI Docs</h1>

        <p>
          With FriendliAI, your team gets fast, affordable, and reliable AI inference at scale.
        </p>

        <p>
          Start with Model APIs, a curated set of popular, open-weight models that are ready for you today.
        </p>

        <p>
          If you want to run <em>any</em> model — including your own — on dedicated GPUs, try Dedicated Endpoints.
        </p>

        <LandingLink href="/docs/guides/introduction/overview">
          Start Building

          <svg>
            <path />

            <path />
          </svg>
        </LandingLink>
      </div>

      <div>
        <div />

        <div>
          <div>
            <div>
              <img alt="" />
            </div>
          </div>
        </div>
      </div>
    </section>

    <div>
      <LandingLink href="/docs/guides/introduction/overview">
        <div>
          <Icon icon="book-open" />
        </div>

        <h2>Guides</h2>
        <p>Learn about inference and FriendliAI by reading how-tos and best-practice articles.</p>

        <span>
          Read the Guides

          <svg>
            <path />

            <path />
          </svg>
        </span>
      </LandingLink>

      <LandingLink href="/docs/examples/models/overview">
        <div>
          <Icon icon="code" />
        </div>

        <h2>Examples</h2>
        <p>Explore real-world use cases and see what you can do with FriendliAI.</p>

        <span>
          Browse the Examples

          <svg>
            <path />

            <path />
          </svg>
        </span>
      </LandingLink>

      <LandingLink href="/docs/openapi/introduction">
        <div>
          <Icon icon="book" />
        </div>

        <h2>Reference</h2>
        <p>Look up the API's operations and parameters. For example, learn how to use the Chat Completions API.</p>

        <span>
          View the Reference

          <svg>
            <path />

            <path />
          </svg>
        </span>
      </LandingLink>
    </div>

    <section>
      <div>
        <h2>Featured Articles</h2>

        <LandingLink href="/docs/guides/introduction/overview">
          All Articles

          <svg>
            <path />

            <path />
          </svg>
        </LandingLink>
      </div>

      <div>
        <LandingLink href="/docs/guides/introduction/get-started">
          <span>Introduction</span>
          <h3>Get Started with FriendliAI</h3>
          <p>Create an API key and send your first request. Once you complete these steps, you're ready to use your agent or SDK with FriendliAI.</p>
        </LandingLink>

        <LandingLink href="/docs/guides/tool-calling">
          <span>Capabilities</span>
          <h3>Tool Calling</h3>
          <p>Use OpenAI-compatible tool calling with broad model support, strict schema enforcement, and parallel tool calls.</p>
        </LandingLink>

        <LandingLink href="/docs/guides/model-apis/pricing">
          <span>Model APIs</span>
          <h3>Models and Pricing</h3>
          <p>View pricing per model. Compare token-based and audio-based rates across text and audio models.</p>
        </LandingLink>
      </div>
    </section>

    <section>
      <div>
        <h2>Featured Examples</h2>

        <LandingLink href="/docs/examples/models/overview">
          All Examples

          <svg>
            <path />

            <path />
          </svg>
        </LandingLink>
      </div>

      <div>
        <LandingLink href="/docs/examples/agents/friendlilink">
          <span>Agents</span>
          <h3>Set Up Agents Automatically</h3>
          <p>Use the FriendliLink CLI to connect your agent to FriendliAI with one command. Once you complete these steps, your agent will send requests to FriendliAI.</p>
        </LandingLink>

        <LandingLink href="/docs/examples/models/zai-glm-5-3/overview">
          <span>Models</span>
          <h3>Use GLM-5.3</h3>
          <p>Use Z.ai's flagship open-weight model with FriendliAI. Review the model's properties and pricing, then choose a feature to start building.</p>
        </LandingLink>

        <LandingLink href="/docs/examples/agents/hermes-agent">
          <span>Agents</span>
          <h3>Use Hermes Agent</h3>
          <p>Set up Hermes Agent to connect to FriendliAI. Once you complete these steps, your agent will send requests to FriendliAI.</p>
        </LandingLink>
      </div>
    </section>
  </div>
</div>


# Cost
Source: https://friendli.ai/docs/openapi/administration/cost

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml get /v1/team/cost
Get cost details for the team.

Get cost details for the team.

If `X-Friendli-Team` is omitted, this endpoint uses the default team configured in Friendli Suite.

There may be a slight delay between usage and when it shows up in cost. If you need to call this endpoint repeatedly, wait at least 5 minutes between calls.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Usage
Source: https://friendli.ai/docs/openapi/administration/usage

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml get /v1/team/usage
Get usage details for the team.

Get usage details for the team.

If `X-Friendli-Team` is omitted, this endpoint uses the default team configured in Friendli Suite.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Container Audio Transcriptions
Source: https://friendli.ai/docs/openapi/container/audio-transcriptions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/audio/transcriptions
Transcribe an audio file into text.

Transcribe an audio file into text.


# Container Audio Transcriptions Chunk Object
Source: https://friendli.ai/docs/openapi/container/audio-transcriptions-chunk-object

Schema reference for the streamed chunk returned by Friendli Container when streaming the audio transcriptions API.

Represents a streamed chunk returned when streaming the audio transcriptions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "type": "transcript.text.delta",
    "delta": "The"
  }

  data: {
    "type": "transcript.text.delta",
    "delta": " quick"
  }

  ...

  data: {
    "type": "transcript.text.done",
    "text": "The quick brown fox jumps over the lazy dog.",
    "usage": {
      "type": "tokens",
      "input_tokens": 20,
      "output_tokens": 10,
      "total_tokens": 30,
      "input_audio_length_ms": 18000,
      "processed_audio_length_ms": 24000,
      "input_token_details": {
        "audio_tokens": 10,
        "text_tokens": 10
      }
    }
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="type" type="enum<string>">
  The event type.

  Available options: `transcript.text.delta`, `transcript.text.done`
</ResponseField>

<ResponseField name="delta" type="string | null">
  The incremental transcript text.
</ResponseField>

<ResponseField name="text" type="string | null">
  The transcribed text.
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="type" type="string">
      The type of the usage object. Always `tokens` for this variant.
    </ResponseField>

    <ResponseField name="input_tokens" type="integer">
      Number of input tokens billed for this request.
    </ResponseField>

    <ResponseField name="output_tokens" type="integer">
      Number of output tokens generated.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used (input + output tokens).
    </ResponseField>

    <ResponseField name="input_audio_length_ms" type="integer">
      The length of the input audio in milliseconds.
    </ResponseField>

    <ResponseField name="processed_audio_length_ms" type="integer">
      The length of the processed audio in milliseconds.
    </ResponseField>

    <ResponseField name="input_token_details" type="object | null">
      Details about the input tokens billed for this request.

      <Expandable title="child attributes">
        <ResponseField name="audio_tokens" type="integer">
          Number of audio tokens billed for this request.
        </ResponseField>

        <ResponseField name="text_tokens" type="integer">
          Number of text tokens billed for this request.
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>


# Container Chat Completions
Source: https://friendli.ai/docs/openapi/container/chat-completions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/chat/completions
Generate a model response from a list of messages comprising a conversation. Compatible with the OpenAI Chat Completions API, with support for streaming, tool calls, and structured outputs.

Generate a model response from a list of messages comprising a conversation. Compatible with the OpenAI Chat Completions API, with support for streaming, tool calls, and structured outputs.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/container/chat-completions-chunk-object).


# Container Chat Completions Chunk Object
Source: https://friendli.ai/docs/openapi/container/chat-completions-chunk-object

Schema reference for the streamed chunk returned by Friendli Container when streaming the chat completions API.

Represents a streamed chunk returned when streaming the chat completions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "role": "assistant", "content": "This" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294381
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "content": " is" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294381
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {},
        "finish_reason": "stop",
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294383
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [],
    "usage": {
      "prompt_tokens": 8,
      "completion_tokens": 4,
      "total_tokens": 12
    },
    "created": 1726294402
  }

  data: [DONE]
  ```

  ```json With tools theme={null}
  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "role": "assistant", "content": "This" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "id": "call_TARbemDG9CFdwuoaQBTRXiYK",
              "type": "function",
              "function": { "name": "func", "arguments": "{\"" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "type": "function",
              "function": { "arguments": "arg" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "type": "function",
              "function": { "arguments": "}" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {},
        "finish_reason": "tool_calls",
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "object": "chat.completion.chunk",
    "choices": [],
    "usage": {
      "prompt_tokens": 468,
      "completion_tokens": 59,
      "total_tokens": 527
    },
    "created": 1726294443
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="id" type="string">
  A unique ID of the chat completion.
</ResponseField>

<ResponseField name="object" type="string">
  The object type, which is always set to `chat.completion.chunk`.
</ResponseField>

<ResponseField name="model" type="string">
  The model to generate the completion.
</ResponseField>

<ResponseField name="choices" type="object[]">
  <Expandable title="child attributes">
    <ResponseField name="index" type="integer">
      The index of the choice in the list of generated choices.
    </ResponseField>

    <ResponseField name="delta" type="object">
      <Expandable title="child attributes">
        <ResponseField name="role" type="string | null">
          Role of the generated message author, in this case `assistant`.
        </ResponseField>

        <ResponseField name="content" type="string | null">
          The contents of the assistant message.
        </ResponseField>

        <ResponseField name="tool_calls" type="object[] | null">
          <Expandable title="child attributes">
            <ResponseField name="index" type="integer">
              The index of the tool call being generated.
            </ResponseField>

            <ResponseField name="id" type="string | null">
              The ID of the tool call.
            </ResponseField>

            <ResponseField name="type" type="string">
              The type of the tool, which is always set to `function`.
            </ResponseField>

            <ResponseField name="function" type="object">
              <Expandable title="child attributes">
                <ResponseField name="name" type="string | null">
                  The name of the function to call.
                </ResponseField>

                <ResponseField name="arguments" type="string">
                  The arguments for calling the function, generated by the model in JSON format.
                  Ensure to validate these arguments in your code before invoking the function since the model may not always produce valid JSON.
                </ResponseField>
              </Expandable>
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="finish_reason" type="enum<string> | null">
      Termination condition of the generation.
      `stop` means the API returned the full chat completions generated by the model without running into any limits.
      `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length.
      `tool_calls` means the API has generated tool calls.

      Available options: `stop`, `length`, `tool_calls`
    </ResponseField>

    <ResponseField name="logprobs" type="object | null">
      Log probability information for the choice.

      <Expandable title="child attributes">
        <ResponseField name="content" type="object[] | null">
          A list of message content tokens with log probability information.

          <Expandable title="child attributes">
            <ResponseField name="token" type="string">
              The token.
            </ResponseField>

            <ResponseField name="logprob" type="number">
              The log probability of this token.
            </ResponseField>

            <ResponseField name="bytes" type="integer[] | null">
              A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token.
            </ResponseField>

            <ResponseField name="top_logprobs" type="object[]">
              List of the most likely tokens and their log probability, at this token position.

              <Expandable title="child attributes">
                <ResponseField name="token" type="string">
                  The token.
                </ResponseField>

                <ResponseField name="logprob" type="number">
                  The log probability of this token.
                </ResponseField>

                <ResponseField name="bytes" type="integer[] | null">
                  A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token.
                </ResponseField>
              </Expandable>
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="prompt_tokens" type="integer">
      Number of tokens in the prompt.
    </ResponseField>

    <ResponseField name="completion_tokens" type="integer">
      Number of tokens in the generated chat completions.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`).
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="created" type="integer">
  The Unix timestamp (in seconds) for when the token is sampled.
</ResponseField>


# Container Completions
Source: https://friendli.ai/docs/openapi/container/completions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/completions
Generate a text completion from a prompt or token sequence. Supports streaming and configurable generation parameters.

Generate a text completion from a prompt or token sequence. Supports streaming and configurable generation parameters.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/container/completions-chunk-object).


# Container Completions Chunk Object
Source: https://friendli.ai/docs/openapi/container/completions-chunk-object

Schema reference for the streamed chunk returned by Friendli Container when streaming the completions API.

Represents a streamed chunk returned when streaming the completions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": " such",
        "token": 1778,
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": " as",
        "token": 439,
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  ...

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": "",
        "finish_reason": "length",
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "object": "text_completion",
    "choices": [],
    "usage": {
      "prompt_tokens": 5,
      "completion_tokens": 10,
      "total_tokens": 15
    },
    "created": 1733382157
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="id" type="string">
  A unique ID of the completion.
</ResponseField>

<ResponseField name="object" type="string">
  The object type, which is always set to `text_completion`.
</ResponseField>

<ResponseField name="model" type="string">
  The model to generate the completion.
</ResponseField>

<ResponseField name="choices" type="object[]">
  <Expandable title="child attributes">
    <ResponseField name="index" type="integer">
      The index of the choice in the list of generated choices.
    </ResponseField>

    <ResponseField name="text" type="string">
      The text.
    </ResponseField>

    <ResponseField name="token" type="integer | null">
      The token.
    </ResponseField>

    <ResponseField name="finish_reason" type="enum<string> | null">
      Termination condition of the generation.
      `stop` means the API returned the full completions generated by the model without running into any limits.
      `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length.

      Available options: `stop`, `length`
    </ResponseField>

    <ResponseField name="logprobs" type="object | null">
      Log probability information for the choice.

      <Expandable title="child attributes">
        <ResponseField name="text_offset" type="integer[]">
          The starting character position of each token in the generated text, useful for mapping tokens back to their exact location for detailed analysis.
        </ResponseField>

        <ResponseField name="token_logprobs" type="number[]">
          The log probabilities of each generated token, indicating the model's confidence in selecting each token.
        </ResponseField>

        <ResponseField name="tokens" type="string[]">
          A list of individual tokens generated in the completion, representing segments of text such as words or pieces of words.
        </ResponseField>

        <ResponseField name="top_logprobs" type="object[]">
          A list of dictionaries, where each dictionary represents the top alternative tokens considered by the model at a specific position in the generated text, along with their log probabilities. The number of items in each dictionary matches the value of `logprobs`.
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="prompt_tokens" type="integer">
      Number of tokens in the prompt.
    </ResponseField>

    <ResponseField name="completion_tokens" type="integer">
      Number of tokens in the generated completions.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`).
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="created" type="integer">
  The Unix timestamp (in seconds) for when the token is sampled.
</ResponseField>


# Container Detokenization
Source: https://friendli.ai/docs/openapi/container/detokenization

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/detokenize
Convert a list of token IDs back into text.

Convert a list of token IDs back into text.


# Container Image Edits
Source: https://friendli.ai/docs/openapi/container/image-edits

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/images/edits
Edit an image based on a text prompt.

Edit an image based on a text prompt.


# Container Image Generations
Source: https://friendli.ai/docs/openapi/container/image-generations

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/images/generations
Generate an image from a text prompt.

Generate an image from a text prompt.


# Container Messages
Source: https://friendli.ai/docs/openapi/container/messages

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/messages
Generate a model response from a conversation using the Anthropic Messages API format. Supports streaming, function tool calls, and extended thinking.

Generate a model response from a conversation using the Anthropic Messages API format. Supports streaming, function tool calls, and extended thinking.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/container/messages-chunk-object).

<Warning>Server-side tools are not supported in the Messages API. In `tools`, only custom/client function tools are used; non-`custom` tool types are ignored.</Warning>

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Container Messages Chunk Object
Source: https://friendli.ai/docs/openapi/container/messages-chunk-object

Schema reference for the streamed chunk returned by Friendli Container when streaming the messages API.

Represents a streamed chunk returned when streaming the messages API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "text", "text": "" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "text_delta", "text": "Hello" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "text_delta", "text": ", how can I help?" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "end_turn", "stop_sequence": null },
    "usage": { "output_tokens": 11 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```

  ```json Tool delta example theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "tool_use", "id": "toolu_01A", "name": "get_weather", "input": {} }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "input_json_delta", "partial_json": "{\"location\":\"Seoul\"" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "input_json_delta", "partial_json": ",\"date\":\"2026-03-12\"}" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "tool_use", "stop_sequence": null },
    "usage": { "output_tokens": 19 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```

  ```json Thinking delta example theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "thinking", "thinking": "" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "thinking_delta", "thinking": "I should answer briefly." }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "signature_delta", "signature": "sig_abc123" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "end_turn", "stop_sequence": null },
    "usage": { "output_tokens": 8 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```
</ResponseExample>

## Events

### `event: message_start` Chunk Object

<ResponseField name="type" type="string">
  Always `message_start`.
</ResponseField>

<ResponseField name="message" type="object">
  Initial response envelope for this streamed message.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique ID of the response message.
    </ResponseField>

    <ResponseField name="type" type="string">
      Object type, always `message`.
    </ResponseField>

    <ResponseField name="role" type="string">
      Author role of streamed output, always `assistant`.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Initialized as an empty array at start. Populated by subsequent block events.
    </ResponseField>

    <ResponseField name="stop_reason" type="enum<string> | null">
      Stop reason placeholder at start.

      Available options: `end_turn`, `max_tokens`, `tool_use`, `stop_sequence`
    </ResponseField>

    <ResponseField name="stop_sequence" type="string | null">
      Matched stop sequence placeholder at start.
    </ResponseField>

    <ResponseField name="usage" type="object">
      Running usage object.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          Number of billed input tokens.
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          Number of billed output tokens currently emitted.
        </ResponseField>

        <ResponseField name="cache_read_input_tokens" type="integer | null">
          Number of cached input tokens read, when applicable.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      Model identifier for this response, when available.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_start` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_start`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the content block in the response `content` array.
</ResponseField>

<ResponseField name="content_block" type="object">
  Initial content block payload.

  <Expandable title="child attributes">
    <ResponseField name="type" type="enum<string>">
      Content block type.

      Available options: `text`, `thinking`, `tool_use`
    </ResponseField>

    <ResponseField name="text" type="string">
      Text content when `type=text`.
    </ResponseField>

    <ResponseField name="thinking" type="string">
      Reasoning content when `type=thinking`.
    </ResponseField>

    <ResponseField name="signature" type="string | null">
      Optional signature when `type=thinking`.
    </ResponseField>

    <ResponseField name="id" type="string">
      Tool call ID when `type=tool_use`.
    </ResponseField>

    <ResponseField name="name" type="string">
      Tool name when `type=tool_use`.
    </ResponseField>

    <ResponseField name="input" type="object">
      Parsed tool input object when `type=tool_use`.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_delta` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_delta`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the content block being updated.
</ResponseField>

<ResponseField name="delta" type="object">
  Incremental delta payload.

  <Expandable title="child attributes">
    <ResponseField name="type" type="enum<string>">
      Delta type.

      Available options: `text_delta`, `thinking_delta`, `signature_delta`, `input_json_delta`
    </ResponseField>

    <ResponseField name="text" type="string">
      Text fragment when `type=text_delta`.
    </ResponseField>

    <ResponseField name="thinking" type="string">
      Reasoning fragment when `type=thinking_delta`.
    </ResponseField>

    <ResponseField name="signature" type="string">
      Signature fragment when `type=signature_delta`.
    </ResponseField>

    <ResponseField name="partial_json" type="string">
      Partial JSON fragment for tool arguments when `type=input_json_delta`.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_stop` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_stop`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the block that has finished streaming.
</ResponseField>

### `event: message_delta` Chunk Object

<ResponseField name="type" type="string">
  Always `message_delta`.
</ResponseField>

<ResponseField name="delta" type="object">
  Final message-level delta.

  <Expandable title="child attributes">
    <ResponseField name="stop_reason" type="enum<string> | null">
      Why generation stopped.

      Available options: `end_turn`, `max_tokens`, `tool_use`, `stop_sequence`
    </ResponseField>

    <ResponseField name="stop_sequence" type="string | null">
      Matched stop sequence when `stop_reason=stop_sequence`.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  Usage delta object that may be emitted near stream completion.

  <Expandable title="child attributes">
    <ResponseField name="input_tokens" type="integer | null">
      Input token count delta when included.
    </ResponseField>

    <ResponseField name="output_tokens" type="integer | null">
      Output token count delta when included.
    </ResponseField>

    <ResponseField name="cache_read_input_tokens" type="integer | null">
      Cached input token count delta when included.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: message_stop` Chunk Object

<ResponseField name="type" type="string">
  Always `message_stop`. Indicates stream completion.
</ResponseField>

### `event: error` Chunk Object

<ResponseField name="type" type="string">
  Always `error`.
</ResponseField>

<ResponseField name="error" type="object">
  Error payload object.

  <Expandable title="child attributes">
    <ResponseField name="type" type="string">
      Error category, such as `invalid_request_error`.
    </ResponseField>

    <ResponseField name="message" type="string">
      Human-readable error message.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="request_id" type="string | null">
  Request identifier for debugging and support.
</ResponseField>


# Container Overview
Source: https://friendli.ai/docs/openapi/container/overview

API reference for Friendli Container. Browse self-hosted inference endpoints for chat, responses, completions, messages, tokenization, images, and audio transcription.

OpenAPI reference of Friendli Container API.

## Inference

<CardGroup>
  <Card title="Chat completions" href="/docs/openapi/container/chat-completions">
    Discover how to generate text through interactive conversations.
  </Card>

  <Card title="Responses (Beta)" href="/docs/openapi/container/responses">
    Generate model responses from text or image inputs with tool calls, structured outputs, and reasoning.
  </Card>

  <Card title="Messages (Beta)" href="/docs/openapi/container/messages">
    Generate responses using Anthropic Messages-style payloads.
  </Card>

  <Card title="Completions" href="/docs/openapi/container/completions">
    Learn how to generate text.
  </Card>

  <Card title="Text classification" href="/docs/openapi/container/text-classification">
    Learn how to classify a given text input into categories.
  </Card>

  <Card title="Tokenization" href="/docs/openapi/container/tokenization">
    Explore the process of breaking down text into smaller tokens for machine processing.
  </Card>

  <Card title="Detokenization" href="/docs/openapi/container/detokenization">
    Learn how to reconstruct tokenized text back into its original, human-readable form.
  </Card>

  <Card title="Image generations" href="/docs/openapi/container/image-generations">
    Learn how to generate images.
  </Card>

  <Card title="Image edits" href="/docs/openapi/container/image-edits">
    Learn how to edit images.
  </Card>

  <Card title="Audio transcriptions" href="/docs/openapi/container/audio-transcriptions">
    Learn how to transcribe audio.
  </Card>
</CardGroup>


# Container Responses
Source: https://friendli.ai/docs/openapi/container/responses

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/responses
Generate a model response from text or image inputs. Follows the OpenAI Responses API format, with support for streaming, function and custom tool calls, structured outputs, and reasoning controls.

Generate a model response from text or image inputs. Follows the OpenAI Responses API format, with support for streaming, function and custom tool calls, structured outputs, and reasoning controls.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/container/responses-chunk-object).

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Container Responses Chunk Object
Source: https://friendli.ai/docs/openapi/container/responses-chunk-object

Schema reference for the streamed chunk returned by Friendli Container when streaming the responses API.

Represents a streamed chunk returned when streaming the responses API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  event: response.created
  data: {
    "type": "response.created",
    "sequence_number": 0,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "in_progress",
      "output": []
    }
  }

  event: response.in_progress
  data: {
    "type": "response.in_progress",
    "sequence_number": 1,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "in_progress",
      "output": []
    }
  }

  event: response.output_item.added
  data: {
    "type": "response.output_item.added",
    "sequence_number": 2,
    "output_index": 0,
    "item": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "status": "in_progress",
      "role": "assistant",
      "content": []
    }
  }

  event: response.content_part.added
  data: {
    "type": "response.content_part.added",
    "sequence_number": 3,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "part": { "type": "output_text", "text": "" }
  }

  event: response.output_text.delta
  data: {
    "type": "response.output_text.delta",
    "sequence_number": 4,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "delta": "Hello"
  }

  ...

  event: response.output_text.done
  data: {
    "type": "response.output_text.done",
    "sequence_number": 6,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "text": "Hello there, how may I assist you today?"
  }

  event: response.content_part.done
  data: {
    "type": "response.content_part.done",
    "sequence_number": 7,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "part": {
      "type": "output_text",
      "text": "Hello there, how may I assist you today?"
    }
  }

  event: response.output_item.done
  data: {
    "type": "response.output_item.done",
    "sequence_number": 8,
    "output_index": 0,
    "item": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "status": "completed",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Hello there, how may I assist you today?"
        }
      ]
    }
  }

  event: response.completed
  data: {
    "type": "response.completed",
    "sequence_number": 9,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "completed",
      "output": [
        {
          "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
          "type": "message",
          "status": "completed",
          "role": "assistant",
          "content": [
            {
              "type": "output_text",
              "text": "Hello there, how may I assist you today?"
            }
          ]
        }
      ],
      "usage": {
        "input_tokens": 9,
        "input_tokens_details": { "cached_tokens": 0 },
        "output_tokens": 11,
        "output_tokens_details": { "reasoning_tokens": 0 },
        "total_tokens": 20
      }
    }
  }
  ```
</ResponseExample>

## Events

### `event: response.created` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.created`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that was created.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.in_progress` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.in_progress`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that is in progress.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.completed` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.completed`.
</ResponseField>

<ResponseField name="response" type="object">
  Properties of the completed response.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.failed` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.failed`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that failed.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.incomplete` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.incomplete`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that was incomplete.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_item.added` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_item.added`.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that was added.
</ResponseField>

<ResponseField name="item" type="object">
  The output item that was added.

  <Expandable title="message">
    An output message from the model.

    <ResponseField name="id" type="string">
      The unique ID of the output message.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      The content of the output message.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the output text. Always `output_text`.

          Available options: `output_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The text output from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="role" type="enum<string>">
      The role of the output message. Always `assistant`.

      Available options: `assistant`
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the message input. One of `in_progress`, `completed`, or `incomplete`. Populated when input items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the output message. Always `message`.

      Available options: `message`
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning">
    A description of the chain of thought used by a reasoning model while generating a response. Be sure to include these items in your `input` to the Responses API for subsequent turns of a conversation if you are manually managing context.

    <ResponseField name="id" type="string">
      The unique identifier of the reasoning content.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Reasoning text content.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the reasoning text. Always `reasoning_text`.

          Available options: `reasoning_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The reasoning text from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the object. Always `reasoning`.

      Available options: `reasoning`
    </ResponseField>
  </Expandable>

  <Expandable title="function_call">
    A tool call to run a function.

    <ResponseField name="arguments" type="string">
      A JSON string of the arguments to pass to the function.
    </ResponseField>

    <ResponseField name="call_id" type="string">
      The unique ID of the function tool call generated by the model.
    </ResponseField>

    <ResponseField name="name" type="string">
      The name of the function to run.
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the function tool call. Always `function_call`.

      Available options: `function_call`
    </ResponseField>

    <ResponseField name="id" type="string">
      The unique ID of the function tool call.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_item.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_item.done`.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that was marked done.
</ResponseField>

<ResponseField name="item" type="object">
  The output item that was marked done.

  <Expandable title="message">
    An output message from the model.

    <ResponseField name="id" type="string">
      The unique ID of the output message.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      The content of the output message.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the output text. Always `output_text`.

          Available options: `output_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The text output from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="role" type="enum<string>">
      The role of the output message. Always `assistant`.

      Available options: `assistant`
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the message input. One of `in_progress`, `completed`, or `incomplete`. Populated when input items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the output message. Always `message`.

      Available options: `message`
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning">
    A description of the chain of thought used by a reasoning model while generating a response. Be sure to include these items in your `input` to the Responses API for subsequent turns of a conversation if you are manually managing context.

    <ResponseField name="id" type="string">
      The unique identifier of the reasoning content.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Reasoning text content.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the reasoning text. Always `reasoning_text`.

          Available options: `reasoning_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The reasoning text from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the object. Always `reasoning`.

      Available options: `reasoning`
    </ResponseField>
  </Expandable>

  <Expandable title="function_call">
    A tool call to run a function.

    <ResponseField name="arguments" type="string">
      A JSON string of the arguments to pass to the function.
    </ResponseField>

    <ResponseField name="call_id" type="string">
      The unique ID of the function tool call generated by the model.
    </ResponseField>

    <ResponseField name="name" type="string">
      The name of the function to run.
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the function tool call. Always `function_call`.

      Available options: `function_call`
    </ResponseField>

    <ResponseField name="id" type="string">
      The unique ID of the function tool call.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.content_part.added` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.content_part.added`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the content part was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the content part was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that was added.
</ResponseField>

<ResponseField name="part" type="object">
  The content part that was added.

  <Expandable title="output_text">
    A text output from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the output text. Always `output_text`.

      Available options: `output_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The text output from the model.
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning_text">
    Reasoning text from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the reasoning text. Always `reasoning_text`.

      Available options: `reasoning_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The reasoning text from the model.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.content_part.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.content_part.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the content part was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the content part was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that is done.
</ResponseField>

<ResponseField name="part" type="object">
  The content part that is done.

  <Expandable title="output_text">
    A text output from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the output text. Always `output_text`.

      Available options: `output_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The text output from the model.
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning_text">
    Reasoning text from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the reasoning text. Always `reasoning_text`.

      Available options: `reasoning_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The reasoning text from the model.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_text.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_text.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the text delta was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the text delta was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that the text delta was added to.
</ResponseField>

<ResponseField name="delta" type="string">
  The text delta that was added.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.output_text.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_text.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the text content is finalized.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the text content is finalized.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that the text content is finalized.
</ResponseField>

<ResponseField name="text" type="string">
  The text content that is finalized.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.function_call_arguments.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.function_call_arguments.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the function-call arguments delta is added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the function-call arguments delta is added to.
</ResponseField>

<ResponseField name="delta" type="string">
  The function-call arguments delta that is added.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.function_call_arguments.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.function_call_arguments.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item.
</ResponseField>

<ResponseField name="arguments" type="string">
  The function-call arguments.
</ResponseField>

<ResponseField name="name" type="string">
  The name of the function that was called.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.reasoning_text.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.reasoning_text.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item this reasoning text delta is associated with.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item this reasoning text delta is associated with.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the reasoning content part this delta is associated with.
</ResponseField>

<ResponseField name="delta" type="string">
  The text delta that was added to the reasoning content.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.reasoning_text.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.reasoning_text.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item this reasoning text is associated with.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item this reasoning text is associated with.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the reasoning content part.
</ResponseField>

<ResponseField name="text" type="string">
  The full text of the completed reasoning content.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>


# Container Text Classification
Source: https://friendli.ai/docs/openapi/container/text-classification

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /classify
Classify text input into categories with per-class probabilities.

Classify text input into categories with per-class probabilities.


# Container Tokenization
Source: https://friendli.ai/docs/openapi/container/tokenization

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /v1/tokenize
Convert text input into token IDs.

Convert text input into token IDs.


# Dedicated Create Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/create

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/beta/endpoint
Create a Friendli Dedicated Endpoint deployment for a Hugging Face model via the API. Specify GPU type, replica count, and model configuration.

Create a Dedicated Endpoint deployment for a Hugging Face model.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Delete Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/delete

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml delete /dedicated/beta/endpoint/{endpoint_id}
Permanently delete a Friendli Dedicated Endpoint deployment by ID. This stops the endpoint and releases all associated GPU resources immediately.

Delete an endpoint.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Get Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/get-spec

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml get /dedicated/beta/endpoint/{endpoint_id}
Retrieve the full specification of a Friendli Dedicated Endpoint by ID, including model config, GPU type, replica count, and deployment settings.

Given an endpoint ID, return its specification.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Get Endpoint Status
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/get-status

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml get /dedicated/beta/endpoint/{endpoint_id}/status
Check the current status of a Friendli Dedicated Endpoint by ID. Returns the lifecycle state such as running, sleeping, initializing, or terminated.

Given an endpoint ID, return its current status.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Get Endpoint Version
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/get-version

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml get /dedicated/beta/endpoint/{endpoint_id}/version
Retrieve the version history of a Friendli Dedicated Endpoint by ID. View past configurations and rollback points for deployment tracking.

Given an endpoint ID, return its version history.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated List Endpoints
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/list

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml get /dedicated/beta/endpoint
List all Friendli Dedicated Endpoint deployments in your project. Returns endpoint IDs, statuses, model names, and GPU configurations.

List Dedicated Endpoint deployments.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Restart Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/restart

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml put /dedicated/beta/endpoint/{endpoint_id}/restart
Restart a failed or terminated Friendli Dedicated Endpoint by ID. The endpoint re-initializes with the same model and GPU configuration.

Restart a failed or terminated Dedicated Endpoint.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Sleep Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/sleep

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml put /dedicated/beta/endpoint/{endpoint_id}/sleep
Put a Friendli Dedicated Endpoint into sleep mode by ID. The endpoint stops serving but retains its configuration for quick wake-up later.

Put a Dedicated Endpoint to sleep mode.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Terminate Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/terminate

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml put /dedicated/beta/endpoint/{endpoint_id}/terminate
Terminate a running Friendli Dedicated Endpoint by ID. Stops all inference and releases GPU resources while preserving the endpoint configuration.

Terminate an endpoint.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Update Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/update

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml put /dedicated/beta/endpoint/{endpoint_id}
Update a Friendli Dedicated Endpoint with a new model, GPU type, or replica count. Changes are applied as a new version in the deployment history.

Update a Dedicated Endpoint deployment with new configuration.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Wake Endpoint
Source: https://friendli.ai/docs/openapi/dedicated/endpoint/wake

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml put /dedicated/beta/endpoint/{endpoint_id}/wake
Wake up a sleeping Friendli Dedicated Endpoint by ID. The endpoint resumes serving with its previous model and GPU configuration intact.

Wake up a sleeping Dedicated Endpoint.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Audio Transcriptions
Source: https://friendli.ai/docs/openapi/dedicated/inference/audio-transcriptions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/audio/transcriptions
Transcribe an audio file into text.

Transcribe an audio file into text.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Audio Transcriptions Chunk Object
Source: https://friendli.ai/docs/openapi/dedicated/inference/audio-transcriptions-chunk-object

Schema reference for the streamed chunk returned by Friendli Dedicated Endpoints when streaming the audio transcriptions API.

Represents a streamed chunk returned when streaming the audio transcriptions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "type": "transcript.text.delta",
    "delta": "The"
  }

  data: {
    "type": "transcript.text.delta",
    "delta": " quick"
  }

  ...

  data: {
    "type": "transcript.text.done",
    "text": "The quick brown fox jumps over the lazy dog.",
    "usage": {
      "type": "tokens",
      "input_tokens": 20,
      "output_tokens": 10,
      "total_tokens": 30,
      "input_audio_length_ms": 18000,
      "processed_audio_length_ms": 24000,
      "input_token_details": {
        "audio_tokens": 10,
        "text_tokens": 10
      }
    }
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="type" type="enum<string>">
  The event type.

  Available options: `transcript.text.delta`, `transcript.text.done`
</ResponseField>

<ResponseField name="delta" type="string | null">
  The incremental transcript text.
</ResponseField>

<ResponseField name="text" type="string | null">
  The transcribed text.
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="type" type="string">
      The type of the usage object. Always `tokens` for this variant.
    </ResponseField>

    <ResponseField name="input_tokens" type="integer">
      Number of input tokens billed for this request.
    </ResponseField>

    <ResponseField name="output_tokens" type="integer">
      Number of output tokens generated.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used (input + output tokens).
    </ResponseField>

    <ResponseField name="input_audio_length_ms" type="integer">
      The length of the input audio in milliseconds.
    </ResponseField>

    <ResponseField name="processed_audio_length_ms" type="integer">
      The length of the processed audio in milliseconds.
    </ResponseField>

    <ResponseField name="input_token_details" type="object | null">
      Details about the input tokens billed for this request.

      <Expandable title="child attributes">
        <ResponseField name="audio_tokens" type="integer">
          Number of audio tokens billed for this request.
        </ResponseField>

        <ResponseField name="text_tokens" type="integer">
          Number of text tokens billed for this request.
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>


# Dedicated Chat Completions
Source: https://friendli.ai/docs/openapi/dedicated/inference/chat-completions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/chat/completions
Generate a model response from a list of messages comprising a conversation. Compatible with the OpenAI Chat Completions API, with support for streaming, tool calls, and structured outputs.

Generate a model response from a list of messages comprising a conversation. Compatible with the OpenAI Chat Completions API, with support for streaming, tool calls, and structured outputs.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/dedicated/inference/chat-completions-chunk-object).


# Dedicated Chat Completions Chunk Object
Source: https://friendli.ai/docs/openapi/dedicated/inference/chat-completions-chunk-object

Schema reference for the streamed chunk returned by Friendli Dedicated Endpoints when streaming the chat completions API.

Represents a streamed chunk returned when streaming the chat completions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "role": "assistant", "content": "This" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294381
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "content": " is" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294381
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {},
        "finish_reason": "stop",
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294383
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [],
    "usage": {
      "prompt_tokens": 8,
      "completion_tokens": 4,
      "total_tokens": 12
    },
    "created": 1726294402
  }

  data: [DONE]
  ```

  ```json With tools theme={null}
  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "role": "assistant", "content": "This" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "id": "call_TARbemDG9CFdwuoaQBTRXiYK",
              "type": "function",
              "function": { "name": "func", "arguments": "{\"" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "type": "function",
              "function": { "arguments": "arg" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "type": "function",
              "function": { "arguments": "}" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {},
        "finish_reason": "tool_calls",
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "(endpoint-id)",
    "object": "chat.completion.chunk",
    "choices": [],
    "usage": {
      "prompt_tokens": 468,
      "completion_tokens": 59,
      "total_tokens": 527
    },
    "created": 1726294443
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="id" type="string">
  A unique ID of the chat completion.
</ResponseField>

<ResponseField name="object" type="string">
  The object type, which is always set to `chat.completion.chunk`.
</ResponseField>

<ResponseField name="model" type="string">
  The model to generate the completion. For Dedicated Endpoints, it returns the endpoint ID.
</ResponseField>

<ResponseField name="choices" type="object[]">
  <Expandable title="child attributes">
    <ResponseField name="index" type="integer">
      The index of the choice in the list of generated choices.
    </ResponseField>

    <ResponseField name="delta" type="object">
      <Expandable title="child attributes">
        <ResponseField name="role" type="string | null">
          Role of the generated message author, in this case `assistant`.
        </ResponseField>

        <ResponseField name="content" type="string | null">
          The contents of the assistant message.
        </ResponseField>

        <ResponseField name="tool_calls" type="object[] | null">
          <Expandable title="child attributes">
            <ResponseField name="index" type="integer">
              The index of the tool call being generated.
            </ResponseField>

            <ResponseField name="id" type="string | null">
              The ID of the tool call.
            </ResponseField>

            <ResponseField name="type" type="string">
              The type of the tool, which is always set to `function`.
            </ResponseField>

            <ResponseField name="function" type="object">
              <Expandable title="child attributes">
                <ResponseField name="name" type="string | null">
                  The name of the function to call.
                </ResponseField>

                <ResponseField name="arguments" type="string">
                  The arguments for calling the function, generated by the model in JSON format.
                  Ensure to validate these arguments in your code before invoking the function since the model may not always produce valid JSON.
                </ResponseField>
              </Expandable>
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="finish_reason" type="enum<string> | null">
      Termination condition of the generation.
      `stop` means the API returned the full chat completions generated by the model without running into any limits.
      `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length.
      `tool_calls` means the API has generated tool calls.

      Available options: `stop`, `length`, `tool_calls`
    </ResponseField>

    <ResponseField name="logprobs" type="object | null">
      Log probability information for the choice.

      <Expandable title="child attributes">
        <ResponseField name="content" type="object[] | null">
          A list of message content tokens with log probability information.

          <Expandable title="child attributes">
            <ResponseField name="token" type="string">
              The token.
            </ResponseField>

            <ResponseField name="logprob" type="number">
              The log probability of this token.
            </ResponseField>

            <ResponseField name="bytes" type="integer[] | null">
              A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token.
            </ResponseField>

            <ResponseField name="top_logprobs" type="object[]">
              List of the most likely tokens and their log probability, at this token position.

              <Expandable title="child attributes">
                <ResponseField name="token" type="string">
                  The token.
                </ResponseField>

                <ResponseField name="logprob" type="number">
                  The log probability of this token.
                </ResponseField>

                <ResponseField name="bytes" type="integer[] | null">
                  A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token.
                </ResponseField>
              </Expandable>
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="prompt_tokens" type="integer">
      Number of tokens in the prompt.
    </ResponseField>

    <ResponseField name="completion_tokens" type="integer">
      Number of tokens in the generated chat completions.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`).
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="created" type="integer">
  The Unix timestamp (in seconds) for when the token is sampled.
</ResponseField>


# Dedicated Chat Render
Source: https://friendli.ai/docs/openapi/dedicated/inference/chat-render

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/chat/render
Render a list of chat messages into the prompt text sent to the model.

Render a list of chat messages into the prompt text sent to the model.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Completions
Source: https://friendli.ai/docs/openapi/dedicated/inference/completions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/completions
Generate a text completion from a prompt or token sequence. Supports streaming and configurable generation parameters.

Generate a text completion from a prompt or token sequence. Supports streaming and configurable generation parameters.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/dedicated/inference/completions-chunk-object).


# Dedicated Completions Chunk Object
Source: https://friendli.ai/docs/openapi/dedicated/inference/completions-chunk-object

Schema reference for the streamed chunk returned by Friendli Dedicated Endpoints when streaming the completions API.

Represents a streamed chunk returned when streaming the completions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "(endpoint-id)",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": " such",
        "token": 1778,
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "(endpoint-id)",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": " as",
        "token": 439,
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  ...

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "(endpoint-id)",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": "",
        "finish_reason": "length",
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "(endpoint-id)",
    "object": "text_completion",
    "choices": [],
    "usage": {
      "prompt_tokens": 5,
      "completion_tokens": 10,
      "total_tokens": 15
    },
    "created": 1733382157
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="id" type="string">
  A unique ID of the completion.
</ResponseField>

<ResponseField name="object" type="string">
  The object type, which is always set to `text_completion`.
</ResponseField>

<ResponseField name="model" type="string">
  The model to generate the completion. For Dedicated Endpoints, it returns the endpoint ID.
</ResponseField>

<ResponseField name="choices" type="object[]">
  <Expandable title="child attributes">
    <ResponseField name="index" type="integer">
      The index of the choice in the list of generated choices.
    </ResponseField>

    <ResponseField name="text" type="string">
      The text.
    </ResponseField>

    <ResponseField name="token" type="integer | null">
      The token.
    </ResponseField>

    <ResponseField name="finish_reason" type="enum<string> | null">
      Termination condition of the generation.
      `stop` means the API returned the full completions generated by the model without running into any limits.
      `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length.

      Available options: `stop`, `length`
    </ResponseField>

    <ResponseField name="logprobs" type="object | null">
      Log probability information for the choice.

      <Expandable title="child attributes">
        <ResponseField name="text_offset" type="integer[]">
          The starting character position of each token in the generated text, useful for mapping tokens back to their exact location for detailed analysis.
        </ResponseField>

        <ResponseField name="token_logprobs" type="number[]">
          The log probabilities of each generated token, indicating the model's confidence in selecting each token.
        </ResponseField>

        <ResponseField name="tokens" type="string[]">
          A list of individual tokens generated in the completion, representing segments of text such as words or pieces of words.
        </ResponseField>

        <ResponseField name="top_logprobs" type="object[]">
          A list of dictionaries, where each dictionary represents the top alternative tokens considered by the model at a specific position in the generated text, along with their log probabilities. The number of items in each dictionary matches the value of `logprobs`.
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="prompt_tokens" type="integer">
      Number of tokens in the prompt.
    </ResponseField>

    <ResponseField name="completion_tokens" type="integer">
      Number of tokens in the generated completions.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`).
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="created" type="integer">
  The Unix timestamp (in seconds) for when the token is sampled.
</ResponseField>


# Dedicated Detokenization
Source: https://friendli.ai/docs/openapi/dedicated/inference/detokenization

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/detokenize
Convert a list of token IDs back into text.

Convert a list of token IDs back into text.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Embeddings
Source: https://friendli.ai/docs/openapi/dedicated/inference/embeddings

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/embeddings
Generate an embedding vector from input text or token sequence.

Generate an embedding vector from input text or token sequence.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Image Edits
Source: https://friendli.ai/docs/openapi/dedicated/inference/image-edits

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/images/edits
Edit an image based on a text prompt.

Edit an image based on a text prompt.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Image Generations
Source: https://friendli.ai/docs/openapi/dedicated/inference/image-generations

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/images/generations
Generate an image from a text prompt.

Generate an image from a text prompt.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Messages
Source: https://friendli.ai/docs/openapi/dedicated/inference/messages

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/messages
Generate a model response from a conversation using the Anthropic Messages API format. Supports streaming, function tool calls, and extended thinking.

Generate a model response from a conversation using the Anthropic Messages API format. Supports streaming, function tool calls, and extended thinking.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/dedicated/inference/messages-chunk-object).

<Warning>Server-side tools are not supported in the Messages API. In `tools`, only custom/client function tools are used; non-`custom` tool types are ignored.</Warning>

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Messages Chunk Object
Source: https://friendli.ai/docs/openapi/dedicated/inference/messages-chunk-object

Schema reference for the streamed chunk returned by Friendli Dedicated Endpoints when streaming the messages API.

Represents a streamed chunk returned when streaming the messages API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "text", "text": "" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "text_delta", "text": "Hello" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "text_delta", "text": ", how can I help?" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "end_turn", "stop_sequence": null },
    "usage": { "output_tokens": 11 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```

  ```json Tool delta example theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "tool_use", "id": "toolu_01A", "name": "get_weather", "input": {} }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "input_json_delta", "partial_json": "{\"location\":\"Seoul\"" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "input_json_delta", "partial_json": ",\"date\":\"2026-03-12\"}" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "tool_use", "stop_sequence": null },
    "usage": { "output_tokens": 19 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```

  ```json Thinking delta example theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "thinking", "thinking": "" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "thinking_delta", "thinking": "I should answer briefly." }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "signature_delta", "signature": "sig_abc123" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "end_turn", "stop_sequence": null },
    "usage": { "output_tokens": 8 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```
</ResponseExample>

## Events

### `event: message_start` Chunk Object

<ResponseField name="type" type="string">
  Always `message_start`.
</ResponseField>

<ResponseField name="message" type="object">
  Initial response envelope for this streamed message.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique ID of the response message.
    </ResponseField>

    <ResponseField name="type" type="string">
      Object type, always `message`.
    </ResponseField>

    <ResponseField name="role" type="string">
      Author role of streamed output, always `assistant`.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Initialized as an empty array at start. Populated by subsequent block events.
    </ResponseField>

    <ResponseField name="stop_reason" type="enum<string> | null">
      Stop reason placeholder at start.

      Available options: `end_turn`, `max_tokens`, `tool_use`, `stop_sequence`
    </ResponseField>

    <ResponseField name="stop_sequence" type="string | null">
      Matched stop sequence placeholder at start.
    </ResponseField>

    <ResponseField name="usage" type="object">
      Running usage object.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          Number of billed input tokens.
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          Number of billed output tokens currently emitted.
        </ResponseField>

        <ResponseField name="cache_read_input_tokens" type="integer | null">
          Number of cached input tokens read, when applicable.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      Model identifier for this response, when available.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_start` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_start`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the content block in the response `content` array.
</ResponseField>

<ResponseField name="content_block" type="object">
  Initial content block payload.

  <Expandable title="child attributes">
    <ResponseField name="type" type="enum<string>">
      Content block type.

      Available options: `text`, `thinking`, `tool_use`
    </ResponseField>

    <ResponseField name="text" type="string">
      Text content when `type=text`.
    </ResponseField>

    <ResponseField name="thinking" type="string">
      Reasoning content when `type=thinking`.
    </ResponseField>

    <ResponseField name="signature" type="string | null">
      Optional signature when `type=thinking`.
    </ResponseField>

    <ResponseField name="id" type="string">
      Tool call ID when `type=tool_use`.
    </ResponseField>

    <ResponseField name="name" type="string">
      Tool name when `type=tool_use`.
    </ResponseField>

    <ResponseField name="input" type="object">
      Parsed tool input object when `type=tool_use`.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_delta` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_delta`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the content block being updated.
</ResponseField>

<ResponseField name="delta" type="object">
  Incremental delta payload.

  <Expandable title="child attributes">
    <ResponseField name="type" type="enum<string>">
      Delta type.

      Available options: `text_delta`, `thinking_delta`, `signature_delta`, `input_json_delta`
    </ResponseField>

    <ResponseField name="text" type="string">
      Text fragment when `type=text_delta`.
    </ResponseField>

    <ResponseField name="thinking" type="string">
      Reasoning fragment when `type=thinking_delta`.
    </ResponseField>

    <ResponseField name="signature" type="string">
      Signature fragment when `type=signature_delta`.
    </ResponseField>

    <ResponseField name="partial_json" type="string">
      Partial JSON fragment for tool arguments when `type=input_json_delta`.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_stop` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_stop`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the block that has finished streaming.
</ResponseField>

### `event: message_delta` Chunk Object

<ResponseField name="type" type="string">
  Always `message_delta`.
</ResponseField>

<ResponseField name="delta" type="object">
  Final message-level delta.

  <Expandable title="child attributes">
    <ResponseField name="stop_reason" type="enum<string> | null">
      Why generation stopped.

      Available options: `end_turn`, `max_tokens`, `tool_use`, `stop_sequence`
    </ResponseField>

    <ResponseField name="stop_sequence" type="string | null">
      Matched stop sequence when `stop_reason=stop_sequence`.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  Usage delta object that may be emitted near stream completion.

  <Expandable title="child attributes">
    <ResponseField name="input_tokens" type="integer | null">
      Input token count delta when included.
    </ResponseField>

    <ResponseField name="output_tokens" type="integer | null">
      Output token count delta when included.
    </ResponseField>

    <ResponseField name="cache_read_input_tokens" type="integer | null">
      Cached input token count delta when included.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: message_stop` Chunk Object

<ResponseField name="type" type="string">
  Always `message_stop`. Indicates stream completion.
</ResponseField>

### `event: error` Chunk Object

<ResponseField name="type" type="string">
  Always `error`.
</ResponseField>

<ResponseField name="error" type="object">
  Error payload object.

  <Expandable title="child attributes">
    <ResponseField name="type" type="string">
      Error category, such as `invalid_request_error`.
    </ResponseField>

    <ResponseField name="message" type="string">
      Human-readable error message.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="request_id" type="string | null">
  Request identifier for debugging and support.
</ResponseField>


# Dedicated Responses
Source: https://friendli.ai/docs/openapi/dedicated/inference/responses

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/responses
Generate a model response from text or image inputs. Follows the OpenAI Responses API format, with support for streaming, function and custom tool calls, structured outputs, and reasoning controls.

Generate a model response from text or image inputs. Follows the OpenAI Responses API format, with support for streaming, function and custom tool calls, structured outputs, and reasoning controls.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/dedicated/inference/responses-chunk-object).

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Dedicated Responses Chunk Object
Source: https://friendli.ai/docs/openapi/dedicated/inference/responses-chunk-object

Schema reference for the streamed chunk returned by Friendli Dedicated Endpoints when streaming the responses API.

Represents a streamed chunk returned when streaming the responses API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  event: response.created
  data: {
    "type": "response.created",
    "sequence_number": 0,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "in_progress",
      "output": []
    }
  }

  event: response.in_progress
  data: {
    "type": "response.in_progress",
    "sequence_number": 1,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "in_progress",
      "output": []
    }
  }

  event: response.output_item.added
  data: {
    "type": "response.output_item.added",
    "sequence_number": 2,
    "output_index": 0,
    "item": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "status": "in_progress",
      "role": "assistant",
      "content": []
    }
  }

  event: response.content_part.added
  data: {
    "type": "response.content_part.added",
    "sequence_number": 3,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "part": { "type": "output_text", "text": "" }
  }

  event: response.output_text.delta
  data: {
    "type": "response.output_text.delta",
    "sequence_number": 4,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "delta": "Hello"
  }

  ...

  event: response.output_text.done
  data: {
    "type": "response.output_text.done",
    "sequence_number": 6,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "text": "Hello there, how may I assist you today?"
  }

  event: response.content_part.done
  data: {
    "type": "response.content_part.done",
    "sequence_number": 7,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "part": {
      "type": "output_text",
      "text": "Hello there, how may I assist you today?"
    }
  }

  event: response.output_item.done
  data: {
    "type": "response.output_item.done",
    "sequence_number": 8,
    "output_index": 0,
    "item": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "status": "completed",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Hello there, how may I assist you today?"
        }
      ]
    }
  }

  event: response.completed
  data: {
    "type": "response.completed",
    "sequence_number": 9,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "completed",
      "output": [
        {
          "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
          "type": "message",
          "status": "completed",
          "role": "assistant",
          "content": [
            {
              "type": "output_text",
              "text": "Hello there, how may I assist you today?"
            }
          ]
        }
      ],
      "usage": {
        "input_tokens": 9,
        "input_tokens_details": { "cached_tokens": 0 },
        "output_tokens": 11,
        "output_tokens_details": { "reasoning_tokens": 0 },
        "total_tokens": 20
      },
      "model": "(endpoint-id)"
    }
  }
  ```
</ResponseExample>

## Events

### `event: response.created` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.created`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that was created.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.in_progress` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.in_progress`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that is in progress.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.completed` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.completed`.
</ResponseField>

<ResponseField name="response" type="object">
  Properties of the completed response.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.failed` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.failed`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that failed.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.incomplete` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.incomplete`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that was incomplete.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_item.added` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_item.added`.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that was added.
</ResponseField>

<ResponseField name="item" type="object">
  The output item that was added.

  <Expandable title="message">
    An output message from the model.

    <ResponseField name="id" type="string">
      The unique ID of the output message.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      The content of the output message.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the output text. Always `output_text`.

          Available options: `output_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The text output from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="role" type="enum<string>">
      The role of the output message. Always `assistant`.

      Available options: `assistant`
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the message input. One of `in_progress`, `completed`, or `incomplete`. Populated when input items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the output message. Always `message`.

      Available options: `message`
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning">
    A description of the chain of thought used by a reasoning model while generating a response. Be sure to include these items in your `input` to the Responses API for subsequent turns of a conversation if you are manually managing context.

    <ResponseField name="id" type="string">
      The unique identifier of the reasoning content.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Reasoning text content.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the reasoning text. Always `reasoning_text`.

          Available options: `reasoning_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The reasoning text from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the object. Always `reasoning`.

      Available options: `reasoning`
    </ResponseField>
  </Expandable>

  <Expandable title="function_call">
    A tool call to run a function.

    <ResponseField name="arguments" type="string">
      A JSON string of the arguments to pass to the function.
    </ResponseField>

    <ResponseField name="call_id" type="string">
      The unique ID of the function tool call generated by the model.
    </ResponseField>

    <ResponseField name="name" type="string">
      The name of the function to run.
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the function tool call. Always `function_call`.

      Available options: `function_call`
    </ResponseField>

    <ResponseField name="id" type="string">
      The unique ID of the function tool call.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_item.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_item.done`.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that was marked done.
</ResponseField>

<ResponseField name="item" type="object">
  The output item that was marked done.

  <Expandable title="message">
    An output message from the model.

    <ResponseField name="id" type="string">
      The unique ID of the output message.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      The content of the output message.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the output text. Always `output_text`.

          Available options: `output_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The text output from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="role" type="enum<string>">
      The role of the output message. Always `assistant`.

      Available options: `assistant`
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the message input. One of `in_progress`, `completed`, or `incomplete`. Populated when input items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the output message. Always `message`.

      Available options: `message`
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning">
    A description of the chain of thought used by a reasoning model while generating a response. Be sure to include these items in your `input` to the Responses API for subsequent turns of a conversation if you are manually managing context.

    <ResponseField name="id" type="string">
      The unique identifier of the reasoning content.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Reasoning text content.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the reasoning text. Always `reasoning_text`.

          Available options: `reasoning_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The reasoning text from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the object. Always `reasoning`.

      Available options: `reasoning`
    </ResponseField>
  </Expandable>

  <Expandable title="function_call">
    A tool call to run a function.

    <ResponseField name="arguments" type="string">
      A JSON string of the arguments to pass to the function.
    </ResponseField>

    <ResponseField name="call_id" type="string">
      The unique ID of the function tool call generated by the model.
    </ResponseField>

    <ResponseField name="name" type="string">
      The name of the function to run.
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the function tool call. Always `function_call`.

      Available options: `function_call`
    </ResponseField>

    <ResponseField name="id" type="string">
      The unique ID of the function tool call.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.content_part.added` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.content_part.added`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the content part was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the content part was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that was added.
</ResponseField>

<ResponseField name="part" type="object">
  The content part that was added.

  <Expandable title="output_text">
    A text output from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the output text. Always `output_text`.

      Available options: `output_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The text output from the model.
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning_text">
    Reasoning text from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the reasoning text. Always `reasoning_text`.

      Available options: `reasoning_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The reasoning text from the model.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.content_part.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.content_part.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the content part was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the content part was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that is done.
</ResponseField>

<ResponseField name="part" type="object">
  The content part that is done.

  <Expandable title="output_text">
    A text output from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the output text. Always `output_text`.

      Available options: `output_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The text output from the model.
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning_text">
    Reasoning text from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the reasoning text. Always `reasoning_text`.

      Available options: `reasoning_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The reasoning text from the model.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_text.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_text.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the text delta was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the text delta was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that the text delta was added to.
</ResponseField>

<ResponseField name="delta" type="string">
  The text delta that was added.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.output_text.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_text.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the text content is finalized.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the text content is finalized.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that the text content is finalized.
</ResponseField>

<ResponseField name="text" type="string">
  The text content that is finalized.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.function_call_arguments.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.function_call_arguments.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the function-call arguments delta is added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the function-call arguments delta is added to.
</ResponseField>

<ResponseField name="delta" type="string">
  The function-call arguments delta that is added.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.function_call_arguments.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.function_call_arguments.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item.
</ResponseField>

<ResponseField name="arguments" type="string">
  The function-call arguments.
</ResponseField>

<ResponseField name="name" type="string">
  The name of the function that was called.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.reasoning_text.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.reasoning_text.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item this reasoning text delta is associated with.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item this reasoning text delta is associated with.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the reasoning content part this delta is associated with.
</ResponseField>

<ResponseField name="delta" type="string">
  The text delta that was added to the reasoning content.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.reasoning_text.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.reasoning_text.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item this reasoning text is associated with.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item this reasoning text is associated with.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the reasoning content part.
</ResponseField>

<ResponseField name="text" type="string">
  The full text of the completed reasoning content.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>


# Dedicated Text Classification
Source: https://friendli.ai/docs/openapi/dedicated/inference/text-classification

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/classify
Classify text input into categories with per-class probabilities.

Classify text input into categories with per-class probabilities.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Tokenization
Source: https://friendli.ai/docs/openapi/dedicated/inference/tokenization

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /dedicated/v1/tokenize
Convert text input into token IDs.

Convert text input into token IDs.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Dedicated Overview
Source: https://friendli.ai/docs/openapi/dedicated/overview

API reference for Friendli Dedicated Endpoints. Browse inference, endpoint management, chat completions, responses, messages, completions, embeddings, and image generation endpoints.

OpenAPI reference of Friendli Dedicated Endpoints API.

## Inference

<CardGroup>
  <Card title="Chat completions" href="/docs/openapi/dedicated/inference/chat-completions">
    Discover how to generate text through interactive conversations.
  </Card>

  <Card title="Responses (Beta)" href="/docs/openapi/dedicated/inference/responses">
    Generate model responses from text or image inputs with tool calls, structured outputs, and reasoning.
  </Card>

  <Card title="Messages (Beta)" href="/docs/openapi/dedicated/inference/messages">
    Generate responses using Anthropic Messages-style payloads.
  </Card>

  <Card title="Chat render" href="/docs/openapi/dedicated/inference/chat-render">
    Learn how to render chat.
  </Card>

  <Card title="Completions" href="/docs/openapi/dedicated/inference/completions">
    Learn how to generate text.
  </Card>

  <Card title="Embeddings" href="/docs/openapi/dedicated/inference/embeddings">
    Learn how to create embeddings.
  </Card>

  <Card title="Text classification" href="/docs/openapi/dedicated/inference/text-classification">
    Learn how to classify a given text input into categories.
  </Card>

  <Card title="Tokenization" href="/docs/openapi/dedicated/inference/tokenization">
    Explore the process of breaking down text into smaller tokens for machine processing.
  </Card>

  <Card title="Detokenization" href="/docs/openapi/dedicated/inference/detokenization">
    Learn how to reconstruct tokenized text back into its original, human-readable form.
  </Card>

  <Card title="Image generations" href="/docs/openapi/dedicated/inference/image-generations">
    Learn how to generate images.
  </Card>

  <Card title="Image edits" href="/docs/openapi/dedicated/inference/image-edits">
    Learn how to edit images.
  </Card>

  <Card title="Audio transcriptions" href="/docs/openapi/dedicated/inference/audio-transcriptions">
    Learn how to transcribe audio.
  </Card>
</CardGroup>

## Endpoint (Beta)

<CardGroup>
  <Card title="List endpoints" href="/docs/openapi/dedicated/endpoint/list">
    List Dedicated Endpoint deployments.
  </Card>

  <Card title="Retrieve an endpoint specification" href="/docs/openapi/dedicated/endpoint/get-spec">
    Given an endpoint ID, return its specification.
  </Card>

  <Card title="Retrieve endpoint versions" href="/docs/openapi/dedicated/endpoint/get-version">
    Given an endpoint ID, return its version history.
  </Card>

  <Card title="Retrieve an endpoint status" href="/docs/openapi/dedicated/endpoint/get-status">
    Given an endpoint ID, return its current status.
  </Card>

  <Card title="Create an endpoint with Hugging Face model" href="/docs/openapi/dedicated/endpoint/create">
    Create a Dedicated Endpoint deployment for a Hugging Face model.
  </Card>

  <Card title="Update an endpoint with new configuration" href="/docs/openapi/dedicated/endpoint/update">
    Update a Dedicated Endpoint deployment with new configuration.
  </Card>

  <Card title="Terminate an endpoint" href="/docs/openapi/dedicated/endpoint/terminate">
    Terminate an endpoint.
  </Card>

  <Card title="Restart an endpoint" href="/docs/openapi/dedicated/endpoint/restart">
    Restart a failed or terminated Dedicated Endpoint.
  </Card>

  <Card title="Sleep an endpoint" href="/docs/openapi/dedicated/endpoint/sleep">
    Put a Dedicated Endpoint into sleep mode.
  </Card>

  <Card title="Wake up an endpoint" href="/docs/openapi/dedicated/endpoint/wake">
    Wake up a sleeping Dedicated Endpoint.
  </Card>

  <Card title="Delete an endpoint" href="/docs/openapi/dedicated/endpoint/delete">
    Delete an endpoint.
  </Card>
</CardGroup>


# API Reference
Source: https://friendli.ai/docs/openapi/introduction

Complete API reference for FriendliAI. Explore endpoints for Model APIs and Dedicated Endpoints with HTTP examples.

The FriendliAI API is a REST API you call over HTTP from any language. Most inference endpoints are OpenAI-compatible, so existing OpenAI clients work once you change the base URL and API key.

Requests to Friendli Model APIs and Friendli Dedicated Endpoints go to `https://api.friendli.ai`.

## Authentication

<Steps>
  <Step title="Create a Friendli Suite Account">
    If you haven't already, [sign up](https://auth.friendli.ai/sign-up) for an account.
  </Step>

  <Step title="Sign In">
    Then, [sign in](https://auth.friendli.ai). Friendli Suite opens your dashboard.
  </Step>

  <Step title="Create a FriendliAI API Key">
    1. In the left sidebar, click <Icon icon="gear" /> **Settings**. Then, click **[API Keys](https://friendli.ai/suite/~/setting/keys)**.
    2. In the upper-right corner, click <Icon icon="plus" /> **Create API Key**.
    3. (Optional) Name your API key and set an expiration date.
    4. Click **Create Key**.
    5. Click <Icon icon="copy" /> **Copy**.
  </Step>
</Steps>


# Model APIs Audio Transcriptions
Source: https://friendli.ai/docs/openapi/model-apis/audio-transcriptions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/audio/transcriptions
Transcribe an audio file into text.

Transcribe an audio file into text.

See available models at [this pricing table](/docs/guides/model-apis/pricing).

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/model-apis/audio-transcriptions-chunk-object).

<Tip>You can explore examples on the [Friendli Model APIs](https://friendli.ai/get-started/model-apis) playground and adjust settings with just a few clicks.</Tip>


# Model APIs Audio Transcriptions Chunk Object
Source: https://friendli.ai/docs/openapi/model-apis/audio-transcriptions-chunk-object

Schema reference for the streamed chunk returned by Friendli Model APIs when streaming the audio transcriptions API.

Represents a streamed chunk returned when streaming the audio transcriptions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "type": "transcript.text.delta",
    "delta": "The"
  }

  data: {
    "type": "transcript.text.delta",
    "delta": " quick"
  }

  ...

  data: {
    "type": "transcript.text.done",
    "text": "The quick brown fox jumps over the lazy dog.",
    "usage": {
      "type": "tokens",
      "input_tokens": 20,
      "output_tokens": 10,
      "total_tokens": 30,
      "input_audio_length_ms": 18000,
      "processed_audio_length_ms": 24000,
      "input_token_details": {
        "audio_tokens": 10,
        "text_tokens": 10
      }
    }
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="type" type="enum<string>">
  The event type.

  Available options: `transcript.text.delta`, `transcript.text.done`
</ResponseField>

<ResponseField name="delta" type="string | null">
  The incremental transcript text.
</ResponseField>

<ResponseField name="text" type="string | null">
  The transcribed text.
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="type" type="string">
      The type of the usage object. Always `tokens` for this variant.
    </ResponseField>

    <ResponseField name="input_tokens" type="integer">
      Number of input tokens billed for this request.
    </ResponseField>

    <ResponseField name="output_tokens" type="integer">
      Number of output tokens generated.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used (input + output tokens).
    </ResponseField>

    <ResponseField name="input_audio_length_ms" type="integer">
      The length of the input audio in milliseconds.
    </ResponseField>

    <ResponseField name="processed_audio_length_ms" type="integer">
      The length of the processed audio in milliseconds.
    </ResponseField>

    <ResponseField name="input_token_details" type="object | null">
      Details about the input tokens billed for this request.

      <Expandable title="child attributes">
        <ResponseField name="audio_tokens" type="integer">
          Number of audio tokens billed for this request.
        </ResponseField>

        <ResponseField name="text_tokens" type="integer">
          Number of text tokens billed for this request.
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>


# Model APIs Chat Completions
Source: https://friendli.ai/docs/openapi/model-apis/chat-completions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/chat/completions
Generate a model response from a list of messages comprising a conversation. Compatible with the OpenAI Chat Completions API, with support for streaming, tool calls, and structured outputs.

Generate a model response from a list of messages comprising a conversation. Compatible with the OpenAI Chat Completions API, with support for streaming, tool calls, and structured outputs.

See available models at [this pricing table](/docs/guides/model-apis/pricing).

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/model-apis/chat-completions-chunk-object).

<Tip>You can explore examples on the [Friendli Model APIs](https://friendli.ai/get-started/model-apis) playground and adjust settings with just a few clicks.</Tip>


# Model APIs Chat Completions Chunk Object
Source: https://friendli.ai/docs/openapi/model-apis/chat-completions-chunk-object

Schema reference for the streamed chunk returned by Friendli Model APIs when streaming the chat completions API.

Represents a streamed chunk returned when streaming the chat completions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "role": "assistant", "content": "This" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294381
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "content": " is" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294381
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {},
        "finish_reason": "stop",
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294383
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [],
    "usage": {
      "prompt_tokens": 8,
      "completion_tokens": 4,
      "total_tokens": 12
    },
    "created": 1726294402
  }

  data: [DONE]
  ```

  ```json With tools theme={null}
  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": { "role": "assistant", "content": "This" },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "id": "call_TARbemDG9CFdwuoaQBTRXiYK",
              "type": "function",
              "function": { "name": "func", "arguments": "{\"" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "type": "function",
              "function": { "arguments": "arg" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  ...

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {
          "role": "assistant",
          "tool_calls": [
            {
              "index": 0,
              "type": "function",
              "function": { "arguments": "}" }
            }
          ]
        },
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [
      {
        "index": 0,
        "delta": {},
        "finish_reason": "tool_calls",
        "logprobs": null
      }
    ],
    "usage": null,
    "created": 1726294442
  }

  data: {
    "id": "chatcmpl-4b71d12c86d94e719c7e3984a7bb7941",
    "model": "zai-org/GLM-5.3",
    "object": "chat.completion.chunk",
    "choices": [],
    "usage": {
      "prompt_tokens": 468,
      "completion_tokens": 59,
      "total_tokens": 527
    },
    "created": 1726294443
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="id" type="string">
  A unique ID of the chat completion.
</ResponseField>

<ResponseField name="object" type="string">
  The object type, which is always set to `chat.completion.chunk`.
</ResponseField>

<ResponseField name="model" type="string">
  The model to generate the completion.
</ResponseField>

<ResponseField name="choices" type="object[]">
  <Expandable title="child attributes">
    <ResponseField name="index" type="integer">
      The index of the choice in the list of generated choices.
    </ResponseField>

    <ResponseField name="delta" type="object">
      <Expandable title="child attributes">
        <ResponseField name="role" type="string | null">
          Role of the generated message author, in this case `assistant`.
        </ResponseField>

        <ResponseField name="content" type="string | null">
          The contents of the assistant message.
        </ResponseField>

        <ResponseField name="tool_calls" type="object[] | null">
          <Expandable title="child attributes">
            <ResponseField name="index" type="integer">
              The index of the tool call being generated.
            </ResponseField>

            <ResponseField name="id" type="string | null">
              The ID of the tool call.
            </ResponseField>

            <ResponseField name="type" type="string">
              The type of the tool, which is always set to `function`.
            </ResponseField>

            <ResponseField name="function" type="object">
              <Expandable title="child attributes">
                <ResponseField name="name" type="string | null">
                  The name of the function to call.
                </ResponseField>

                <ResponseField name="arguments" type="string">
                  The arguments for calling the function, generated by the model in JSON format.
                  Ensure to validate these arguments in your code before invoking the function since the model may not always produce valid JSON.
                </ResponseField>
              </Expandable>
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="finish_reason" type="enum<string> | null">
      Termination condition of the generation.
      `stop` means the API returned the full chat completions generated by the model without running into any limits.
      `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length.
      `tool_calls` means the API has generated tool calls.

      Available options: `stop`, `length`, `tool_calls`
    </ResponseField>

    <ResponseField name="logprobs" type="object | null">
      Log probability information for the choice.

      <Expandable title="child attributes">
        <ResponseField name="content" type="object[] | null">
          A list of message content tokens with log probability information.

          <Expandable title="child attributes">
            <ResponseField name="token" type="string">
              The token.
            </ResponseField>

            <ResponseField name="logprob" type="number">
              The log probability of this token.
            </ResponseField>

            <ResponseField name="bytes" type="integer[] | null">
              A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token.
            </ResponseField>

            <ResponseField name="top_logprobs" type="object[]">
              List of the most likely tokens and their log probability, at this token position.

              <Expandable title="child attributes">
                <ResponseField name="token" type="string">
                  The token.
                </ResponseField>

                <ResponseField name="logprob" type="number">
                  The log probability of this token.
                </ResponseField>

                <ResponseField name="bytes" type="integer[] | null">
                  A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token.
                </ResponseField>
              </Expandable>
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="prompt_tokens" type="integer">
      Number of tokens in the prompt.
    </ResponseField>

    <ResponseField name="completion_tokens" type="integer">
      Number of tokens in the generated chat completions.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`).
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="created" type="integer">
  The Unix timestamp (in seconds) for when the token is sampled.
</ResponseField>


# Model APIs Chat Render
Source: https://friendli.ai/docs/openapi/model-apis/chat-render

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/chat/render
Render a list of chat messages into the prompt text sent to the model.

Render a list of chat messages into the prompt text sent to the model.

See available models at [this pricing table](/docs/guides/model-apis/pricing).

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Model APIs Completions
Source: https://friendli.ai/docs/openapi/model-apis/completions

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/completions
Generate a text completion from a prompt or token sequence. Supports streaming and configurable generation parameters.

Generate a text completion from a prompt or token sequence. Supports streaming and configurable generation parameters.

See available models at [this pricing table](/docs/guides/model-apis/pricing).

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/model-apis/completions-chunk-object).

<Tip>You can explore examples on the [Friendli Model APIs](https://friendli.ai/get-started/model-apis) playground and adjust settings with just a few clicks.</Tip>


# Model APIs Completions Chunk Object
Source: https://friendli.ai/docs/openapi/model-apis/completions-chunk-object

Schema reference for the streamed chunk returned by Friendli Model APIs when streaming the completions API.

Represents a streamed chunk returned when streaming the completions API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "zai-org/GLM-5.3",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": " such",
        "token": 1778,
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "zai-org/GLM-5.3",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": " as",
        "token": 439,
        "finish_reason": null,
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  ...

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "zai-org/GLM-5.3",
    "object": "text_completion",
    "choices": [
      {
        "index": 0,
        "text": "",
        "finish_reason": "length",
        "logprobs": null
      }
    ],
    "created": 1733382157
  }

  data: {
    "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
    "model": "zai-org/GLM-5.3",
    "object": "text_completion",
    "choices": [],
    "usage": {
      "prompt_tokens": 5,
      "completion_tokens": 10,
      "total_tokens": 15
    },
    "created": 1733382157
  }

  data: [DONE]
  ```
</ResponseExample>

<ResponseField name="id" type="string">
  A unique ID of the completion.
</ResponseField>

<ResponseField name="object" type="string">
  The object type, which is always set to `text_completion`.
</ResponseField>

<ResponseField name="model" type="string">
  The model to generate the completion.
</ResponseField>

<ResponseField name="choices" type="object[]">
  <Expandable title="child attributes">
    <ResponseField name="index" type="integer">
      The index of the choice in the list of generated choices.
    </ResponseField>

    <ResponseField name="text" type="string">
      The text.
    </ResponseField>

    <ResponseField name="token" type="integer | null">
      The token.
    </ResponseField>

    <ResponseField name="finish_reason" type="enum<string> | null">
      Termination condition of the generation.
      `stop` means the API returned the full completions generated by the model without running into any limits.
      `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length.

      Available options: `stop`, `length`
    </ResponseField>

    <ResponseField name="logprobs" type="object | null">
      Log probability information for the choice.

      <Expandable title="child attributes">
        <ResponseField name="text_offset" type="integer[]">
          The starting character position of each token in the generated text, useful for mapping tokens back to their exact location for detailed analysis.
        </ResponseField>

        <ResponseField name="token_logprobs" type="number[]">
          The log probabilities of each generated token, indicating the model's confidence in selecting each token.
        </ResponseField>

        <ResponseField name="tokens" type="string[]">
          A list of individual tokens generated in the completion, representing segments of text such as words or pieces of words.
        </ResponseField>

        <ResponseField name="top_logprobs" type="object[]">
          A list of dictionaries, where each dictionary represents the top alternative tokens considered by the model at a specific position in the generated text, along with their log probabilities. The number of items in each dictionary matches the value of `logprobs`.
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  <Expandable title="child attributes">
    <ResponseField name="prompt_tokens" type="integer">
      Number of tokens in the prompt.
    </ResponseField>

    <ResponseField name="completion_tokens" type="integer">
      Number of tokens in the generated completions.
    </ResponseField>

    <ResponseField name="total_tokens" type="integer">
      Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`).
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="created" type="integer">
  The Unix timestamp (in seconds) for when the token is sampled.
</ResponseField>


# Model APIs Detokenization
Source: https://friendli.ai/docs/openapi/model-apis/detokenization

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/detokenize
Convert a list of token IDs back into text.

Convert a list of token IDs back into text.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.


# Model APIs Messages
Source: https://friendli.ai/docs/openapi/model-apis/messages

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/messages
Generate a model response from a conversation using the Anthropic Messages API format. Supports streaming, function tool calls, and extended thinking.

Generate a model response from a conversation using the Anthropic Messages API format. Supports streaming, function tool calls, and extended thinking.

See available models at [this pricing table](/docs/guides/model-apis/pricing).

<Note>
  The Messages API may not be supported by all models available on Model APIs.
</Note>

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/model-apis/messages-chunk-object).

<Warning>Server-side tools are not supported in the Messages API. In `tools`, only custom/client function tools are used; non-`custom` tool types are ignored.</Warning>

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Model APIs Messages Chunk Object
Source: https://friendli.ai/docs/openapi/model-apis/messages-chunk-object

Schema reference for the streamed chunk returned by Friendli Model APIs when streaming the messages API.

Represents a streamed chunk returned when streaming the messages API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "text", "text": "" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "text_delta", "text": "Hello" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "text_delta", "text": ", how can I help?" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "end_turn", "stop_sequence": null },
    "usage": { "output_tokens": 11 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```

  ```json Tool delta example theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "tool_use", "id": "toolu_01A", "name": "get_weather", "input": {} }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "input_json_delta", "partial_json": "{\"location\":\"Seoul\"" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "input_json_delta", "partial_json": ",\"date\":\"2026-03-12\"}" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "tool_use", "stop_sequence": null },
    "usage": { "output_tokens": 19 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```

  ```json Thinking delta example theme={null}
  event: message_start
  data: {
    "type": "message_start",
    "message": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "role": "assistant",
      "content": [],
      "stop_reason": null,
      "stop_sequence": null,
      "usage": { "input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0 }
    }
  }

  event: content_block_start
  data: {
    "type": "content_block_start",
    "index": 0,
    "content_block": { "type": "thinking", "thinking": "" }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "thinking_delta", "thinking": "I should answer briefly." }
  }

  event: content_block_delta
  data: {
    "type": "content_block_delta",
    "index": 0,
    "delta": { "type": "signature_delta", "signature": "sig_abc123" }
  }

  event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

  event: message_delta
  data: {
    "type": "message_delta",
    "delta": { "stop_reason": "end_turn", "stop_sequence": null },
    "usage": { "output_tokens": 8 }
  }

  event: message_stop
  data: { "type": "message_stop" }
  ```
</ResponseExample>

## Events

### `event: message_start` Chunk Object

<ResponseField name="type" type="string">
  Always `message_start`.
</ResponseField>

<ResponseField name="message" type="object">
  Initial response envelope for this streamed message.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique ID of the response message.
    </ResponseField>

    <ResponseField name="type" type="string">
      Object type, always `message`.
    </ResponseField>

    <ResponseField name="role" type="string">
      Author role of streamed output, always `assistant`.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Initialized as an empty array at start. Populated by subsequent block events.
    </ResponseField>

    <ResponseField name="stop_reason" type="enum<string> | null">
      Stop reason placeholder at start.

      Available options: `end_turn`, `max_tokens`, `tool_use`, `stop_sequence`
    </ResponseField>

    <ResponseField name="stop_sequence" type="string | null">
      Matched stop sequence placeholder at start.
    </ResponseField>

    <ResponseField name="usage" type="object">
      Running usage object.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          Number of billed input tokens.
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          Number of billed output tokens currently emitted.
        </ResponseField>

        <ResponseField name="cache_read_input_tokens" type="integer | null">
          Number of cached input tokens read, when applicable.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      Model identifier for this response, when available.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_start` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_start`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the content block in the response `content` array.
</ResponseField>

<ResponseField name="content_block" type="object">
  Initial content block payload.

  <Expandable title="child attributes">
    <ResponseField name="type" type="enum<string>">
      Content block type.

      Available options: `text`, `thinking`, `tool_use`
    </ResponseField>

    <ResponseField name="text" type="string">
      Text content when `type=text`.
    </ResponseField>

    <ResponseField name="thinking" type="string">
      Reasoning content when `type=thinking`.
    </ResponseField>

    <ResponseField name="signature" type="string | null">
      Optional signature when `type=thinking`.
    </ResponseField>

    <ResponseField name="id" type="string">
      Tool call ID when `type=tool_use`.
    </ResponseField>

    <ResponseField name="name" type="string">
      Tool name when `type=tool_use`.
    </ResponseField>

    <ResponseField name="input" type="object">
      Parsed tool input object when `type=tool_use`.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_delta` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_delta`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the content block being updated.
</ResponseField>

<ResponseField name="delta" type="object">
  Incremental delta payload.

  <Expandable title="child attributes">
    <ResponseField name="type" type="enum<string>">
      Delta type.

      Available options: `text_delta`, `thinking_delta`, `signature_delta`, `input_json_delta`
    </ResponseField>

    <ResponseField name="text" type="string">
      Text fragment when `type=text_delta`.
    </ResponseField>

    <ResponseField name="thinking" type="string">
      Reasoning fragment when `type=thinking_delta`.
    </ResponseField>

    <ResponseField name="signature" type="string">
      Signature fragment when `type=signature_delta`.
    </ResponseField>

    <ResponseField name="partial_json" type="string">
      Partial JSON fragment for tool arguments when `type=input_json_delta`.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: content_block_stop` Chunk Object

<ResponseField name="type" type="string">
  Always `content_block_stop`.
</ResponseField>

<ResponseField name="index" type="integer">
  Index of the block that has finished streaming.
</ResponseField>

### `event: message_delta` Chunk Object

<ResponseField name="type" type="string">
  Always `message_delta`.
</ResponseField>

<ResponseField name="delta" type="object">
  Final message-level delta.

  <Expandable title="child attributes">
    <ResponseField name="stop_reason" type="enum<string> | null">
      Why generation stopped.

      Available options: `end_turn`, `max_tokens`, `tool_use`, `stop_sequence`
    </ResponseField>

    <ResponseField name="stop_sequence" type="string | null">
      Matched stop sequence when `stop_reason=stop_sequence`.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="usage" type="object | null">
  Usage delta object that may be emitted near stream completion.

  <Expandable title="child attributes">
    <ResponseField name="input_tokens" type="integer | null">
      Input token count delta when included.
    </ResponseField>

    <ResponseField name="output_tokens" type="integer | null">
      Output token count delta when included.
    </ResponseField>

    <ResponseField name="cache_read_input_tokens" type="integer | null">
      Cached input token count delta when included.
    </ResponseField>
  </Expandable>
</ResponseField>

### `event: message_stop` Chunk Object

<ResponseField name="type" type="string">
  Always `message_stop`. Indicates stream completion.
</ResponseField>

### `event: error` Chunk Object

<ResponseField name="type" type="string">
  Always `error`.
</ResponseField>

<ResponseField name="error" type="object">
  Error payload object.

  <Expandable title="child attributes">
    <ResponseField name="type" type="string">
      Error category, such as `invalid_request_error`.
    </ResponseField>

    <ResponseField name="message" type="string">
      Human-readable error message.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="request_id" type="string | null">
  Request identifier for debugging and support.
</ResponseField>


# Model APIs Overview
Source: https://friendli.ai/docs/openapi/model-apis/overview

API reference for Friendli Model APIs. Browse chat, responses, messages, completions, tokenization, and audio transcription endpoints.

OpenAPI reference of Friendli Model APIs.

## Inference

<CardGroup>
  <Card title="Chat completions" href="/docs/openapi/model-apis/chat-completions">
    Discover how to generate text through interactive conversations.
  </Card>

  <Card title="Responses (Beta)" href="/docs/openapi/model-apis/responses">
    Generate model responses from text or image inputs with tool calls, structured outputs, and reasoning.
  </Card>

  <Card title="Messages (Beta)" href="/docs/openapi/model-apis/messages">
    Generate responses using Anthropic Messages-style payloads.
  </Card>

  <Card title="Chat render" href="/docs/openapi/model-apis/chat-render">
    Learn how to render chat.
  </Card>

  <Card title="Completions" href="/docs/openapi/model-apis/completions">
    Learn how to generate text.
  </Card>

  <Card title="Tokenization" href="/docs/openapi/model-apis/tokenization">
    Explore the process of breaking down text into smaller tokens for machine processing.
  </Card>

  <Card title="Detokenization" href="/docs/openapi/model-apis/detokenization">
    Learn how to reconstruct tokenized text back into its original, human-readable form.
  </Card>

  <Card title="Audio transcriptions" href="/docs/openapi/model-apis/audio-transcriptions">
    Learn how to transcribe audio.
  </Card>
</CardGroup>


# Model APIs Responses
Source: https://friendli.ai/docs/openapi/model-apis/responses

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/responses
Generate a model response from text or image inputs. Follows the OpenAI Responses API format, with support for streaming, function and custom tool calls, structured outputs, and reasoning controls.

Generate a model response from text or image inputs. Follows the OpenAI Responses API format, with support for streaming, function and custom tool calls, structured outputs, and reasoning controls.

See available models at [this pricing table](/docs/guides/model-apis/pricing).

<Note>
  The Responses API may not be supported by all models available on Model APIs.
</Note>

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.

When streaming mode is used (i.e., `stream` option is set to `true`), the response is in MIME type `text/event-stream`. Otherwise, the content type is `application/json`.
You can view the schema of the streamed sequence of chunk objects in streaming mode [here](/docs/openapi/model-apis/responses-chunk-object).

<Info>
  This API is currently in **Beta**.
  While we strive to provide a stable and reliable experience, this feature is still under active development.
  As a result, you may encounter unexpected behavior or limitations.
  We encourage you to provide feedback to help us improve the feature before its official release.

  * [Feature request & feedback](mailto:support@friendli.ai)
  * [Contact support](mailto:support@friendli.ai)
</Info>


# Model APIs Responses Chunk Object
Source: https://friendli.ai/docs/openapi/model-apis/responses-chunk-object

Schema reference for the streamed chunk returned by Friendli Model APIs when streaming the responses API.

Represents a streamed chunk returned when streaming the responses API, based on the provided input.

<ResponseExample>
  ```json Response theme={null}
  event: response.created
  data: {
    "type": "response.created",
    "sequence_number": 0,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "in_progress",
      "output": []
    }
  }

  event: response.in_progress
  data: {
    "type": "response.in_progress",
    "sequence_number": 1,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "in_progress",
      "output": []
    }
  }

  event: response.output_item.added
  data: {
    "type": "response.output_item.added",
    "sequence_number": 2,
    "output_index": 0,
    "item": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "status": "in_progress",
      "role": "assistant",
      "content": []
    }
  }

  event: response.content_part.added
  data: {
    "type": "response.content_part.added",
    "sequence_number": 3,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "part": { "type": "output_text", "text": "" }
  }

  event: response.output_text.delta
  data: {
    "type": "response.output_text.delta",
    "sequence_number": 4,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "delta": "Hello"
  }

  ...

  event: response.output_text.done
  data: {
    "type": "response.output_text.done",
    "sequence_number": 6,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "text": "Hello there, how may I assist you today?"
  }

  event: response.content_part.done
  data: {
    "type": "response.content_part.done",
    "sequence_number": 7,
    "item_id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
    "output_index": 0,
    "content_index": 0,
    "part": {
      "type": "output_text",
      "text": "Hello there, how may I assist you today?"
    }
  }

  event: response.output_item.done
  data: {
    "type": "response.output_item.done",
    "sequence_number": 8,
    "output_index": 0,
    "item": {
      "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
      "type": "message",
      "status": "completed",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Hello there, how may I assist you today?"
        }
      ]
    }
  }

  event: response.completed
  data: {
    "type": "response.completed",
    "sequence_number": 9,
    "response": {
      "id": "resp_4b71d12c86d94e719c7e3984a7bb7941",
      "object": "response",
      "created_at": 1735722153,
      "status": "completed",
      "output": [
        {
          "id": "msg_4b71d12c86d94e719c7e3984a7bb7941",
          "type": "message",
          "status": "completed",
          "role": "assistant",
          "content": [
            {
              "type": "output_text",
              "text": "Hello there, how may I assist you today?"
            }
          ]
        }
      ],
      "usage": {
        "input_tokens": 9,
        "input_tokens_details": { "cached_tokens": 0 },
        "output_tokens": 11,
        "output_tokens_details": { "reasoning_tokens": 0 },
        "total_tokens": 20
      },
      "model": "zai-org/GLM-5.3"
    }
  }
  ```
</ResponseExample>

## Events

### `event: response.created` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.created`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that was created.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.in_progress` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.in_progress`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that is in progress.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.completed` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.completed`.
</ResponseField>

<ResponseField name="response" type="object">
  Properties of the completed response.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.failed` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.failed`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that failed.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.incomplete` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.incomplete`.
</ResponseField>

<ResponseField name="response" type="object">
  The response that was incomplete.

  <Expandable title="child attributes">
    <ResponseField name="id" type="string">
      Unique identifier for this response.
    </ResponseField>

    <ResponseField name="object" type="string">
      The object type of this resource - always set to `response`.
    </ResponseField>

    <ResponseField name="created_at" type="integer">
      Unix timestamp (in seconds) of when this response was created.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the response generation. One of `completed`, `failed`, `in_progress`, or `incomplete`.

      Available options: `in_progress`, `completed`, `incomplete`, `failed`
    </ResponseField>

    <ResponseField name="output" type="object[]">
      An array of content items generated by the model.

      The length and order of items in the `output` array are dependent on the model's response.
    </ResponseField>

    <ResponseField name="usage" type="object | null">
      Token usage details including input tokens, output tokens, a breakdown of output tokens, and the total tokens used.

      <Expandable title="child attributes">
        <ResponseField name="input_tokens" type="integer">
          The number of input tokens.
        </ResponseField>

        <ResponseField name="input_tokens_details" type="object">
          A detailed breakdown of the input tokens.

          <Expandable title="child attributes">
            <ResponseField name="cached_tokens" type="integer">
              The number of tokens that were retrieved from the cache.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="output_tokens" type="integer">
          The number of output tokens.
        </ResponseField>

        <ResponseField name="output_tokens_details" type="object">
          A detailed breakdown of the output tokens.

          <Expandable title="child attributes">
            <ResponseField name="reasoning_tokens" type="integer">
              The number of reasoning tokens.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          The total number of tokens used.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="incomplete_details" type="object | null">
      Details about why the response is incomplete.

      <Expandable title="child attributes">
        <ResponseField name="reason" type="enum<string>">
          The reason why the response is incomplete.

          Available options: `max_output_tokens`, `content_filter`
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="model" type="string | null">
      The model used to generate the response.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_item.added` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_item.added`.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that was added.
</ResponseField>

<ResponseField name="item" type="object">
  The output item that was added.

  <Expandable title="message">
    An output message from the model.

    <ResponseField name="id" type="string">
      The unique ID of the output message.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      The content of the output message.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the output text. Always `output_text`.

          Available options: `output_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The text output from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="role" type="enum<string>">
      The role of the output message. Always `assistant`.

      Available options: `assistant`
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the message input. One of `in_progress`, `completed`, or `incomplete`. Populated when input items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the output message. Always `message`.

      Available options: `message`
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning">
    A description of the chain of thought used by a reasoning model while generating a response. Be sure to include these items in your `input` to the Responses API for subsequent turns of a conversation if you are manually managing context.

    <ResponseField name="id" type="string">
      The unique identifier of the reasoning content.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Reasoning text content.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the reasoning text. Always `reasoning_text`.

          Available options: `reasoning_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The reasoning text from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the object. Always `reasoning`.

      Available options: `reasoning`
    </ResponseField>
  </Expandable>

  <Expandable title="function_call">
    A tool call to run a function.

    <ResponseField name="arguments" type="string">
      A JSON string of the arguments to pass to the function.
    </ResponseField>

    <ResponseField name="call_id" type="string">
      The unique ID of the function tool call generated by the model.
    </ResponseField>

    <ResponseField name="name" type="string">
      The name of the function to run.
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the function tool call. Always `function_call`.

      Available options: `function_call`
    </ResponseField>

    <ResponseField name="id" type="string">
      The unique ID of the function tool call.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_item.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_item.done`.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that was marked done.
</ResponseField>

<ResponseField name="item" type="object">
  The output item that was marked done.

  <Expandable title="message">
    An output message from the model.

    <ResponseField name="id" type="string">
      The unique ID of the output message.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      The content of the output message.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the output text. Always `output_text`.

          Available options: `output_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The text output from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="role" type="enum<string>">
      The role of the output message. Always `assistant`.

      Available options: `assistant`
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the message input. One of `in_progress`, `completed`, or `incomplete`. Populated when input items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the output message. Always `message`.

      Available options: `message`
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning">
    A description of the chain of thought used by a reasoning model while generating a response. Be sure to include these items in your `input` to the Responses API for subsequent turns of a conversation if you are manually managing context.

    <ResponseField name="id" type="string">
      The unique identifier of the reasoning content.
    </ResponseField>

    <ResponseField name="content" type="object[]">
      Reasoning text content.

      <Expandable title="child attributes">
        <ResponseField name="type" type="enum<string>">
          The type of the reasoning text. Always `reasoning_text`.

          Available options: `reasoning_text`
        </ResponseField>

        <ResponseField name="text" type="string">
          The reasoning text from the model.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the object. Always `reasoning`.

      Available options: `reasoning`
    </ResponseField>
  </Expandable>

  <Expandable title="function_call">
    A tool call to run a function.

    <ResponseField name="arguments" type="string">
      A JSON string of the arguments to pass to the function.
    </ResponseField>

    <ResponseField name="call_id" type="string">
      The unique ID of the function tool call generated by the model.
    </ResponseField>

    <ResponseField name="name" type="string">
      The name of the function to run.
    </ResponseField>

    <ResponseField name="type" type="enum<string>">
      The type of the function tool call. Always `function_call`.

      Available options: `function_call`
    </ResponseField>

    <ResponseField name="id" type="string">
      The unique ID of the function tool call.
    </ResponseField>

    <ResponseField name="status" type="enum<string>">
      The status of the item. One of `in_progress`, `completed`, or `incomplete`. Populated when items are returned via API.

      Available options: `in_progress`, `completed`, `incomplete`
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.content_part.added` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.content_part.added`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the content part was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the content part was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that was added.
</ResponseField>

<ResponseField name="part" type="object">
  The content part that was added.

  <Expandable title="output_text">
    A text output from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the output text. Always `output_text`.

      Available options: `output_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The text output from the model.
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning_text">
    Reasoning text from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the reasoning text. Always `reasoning_text`.

      Available options: `reasoning_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The reasoning text from the model.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.content_part.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.content_part.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the content part was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the content part was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that is done.
</ResponseField>

<ResponseField name="part" type="object">
  The content part that is done.

  <Expandable title="output_text">
    A text output from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the output text. Always `output_text`.

      Available options: `output_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The text output from the model.
    </ResponseField>
  </Expandable>

  <Expandable title="reasoning_text">
    Reasoning text from the model.

    <ResponseField name="type" type="enum<string>">
      The type of the reasoning text. Always `reasoning_text`.

      Available options: `reasoning_text`
    </ResponseField>

    <ResponseField name="text" type="string">
      The reasoning text from the model.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.output_text.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_text.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the text delta was added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the text delta was added to.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that the text delta was added to.
</ResponseField>

<ResponseField name="delta" type="string">
  The text delta that was added.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.output_text.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.output_text.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the text content is finalized.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the text content is finalized.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the content part that the text content is finalized.
</ResponseField>

<ResponseField name="text" type="string">
  The text content that is finalized.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number for this event.
</ResponseField>

### `event: response.function_call_arguments.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.function_call_arguments.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the output item that the function-call arguments delta is added to.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item that the function-call arguments delta is added to.
</ResponseField>

<ResponseField name="delta" type="string">
  The function-call arguments delta that is added.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.function_call_arguments.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.function_call_arguments.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item.
</ResponseField>

<ResponseField name="arguments" type="string">
  The function-call arguments.
</ResponseField>

<ResponseField name="name" type="string">
  The name of the function that was called.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.reasoning_text.delta` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.reasoning_text.delta`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item this reasoning text delta is associated with.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item this reasoning text delta is associated with.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the reasoning content part this delta is associated with.
</ResponseField>

<ResponseField name="delta" type="string">
  The text delta that was added to the reasoning content.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>

### `event: response.reasoning_text.done` Chunk Object

<ResponseField name="type" type="string">
  The type of the event. Always `response.reasoning_text.done`.
</ResponseField>

<ResponseField name="item_id" type="string">
  The ID of the item this reasoning text is associated with.
</ResponseField>

<ResponseField name="output_index" type="integer">
  The index of the output item this reasoning text is associated with.
</ResponseField>

<ResponseField name="content_index" type="integer">
  The index of the reasoning content part.
</ResponseField>

<ResponseField name="text" type="string">
  The full text of the completed reasoning content.
</ResponseField>

<ResponseField name="sequence_number" type="integer">
  The sequence number of this event.
</ResponseField>


# Model APIs Tokenization
Source: https://friendli.ai/docs/openapi/model-apis/tokenization

https://github.com/friendliai/friendli-openapi/raw/refs/heads/main/openapi.yaml post /serverless/v1/tokenize
Convert text input into token IDs.

Convert text input into token IDs.

To request successfully, it is mandatory to enter a **Personal API Key** (e.g. flp\_XXX) value in the **Bearer Token** field.
Refer to the [authentication section](/docs/openapi/introduction#authentication) on our introduction page to learn how to acquire this variable and [visit here](https://friendli.ai/suite/~/setting/keys) to generate your API Key.