> ## Documentation Index
> Fetch the complete documentation index at: https://friendli.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Streaming

> Stream a response on FriendliAI as the model generates each token. Turn streaming on, include token usage, then read and cancel a stream.

Models generate responses one token at a time. Instead of waiting for the entire response, you can get each token as the model generates it.

This *streaming* improves the response-time experience--especially for longer responses, where someone would otherwise have to wait with no feedback from the model.

You can control whether and how Friendli Engine streams a response. Once you turn streaming on, you need to change only how you read and handle the response. You can also cancel a stream to stop the model from generating tokens.

## Streaming Parameters

With streaming parameters, you can control whether and how Friendli Engine streams a response. You pass them as request body parameters.

To learn more, see [Control a Capability](https://friendli.ai/docs/guides/introduction/get-started.md#control-a-capability).

You can control streaming with the following parameters:

| Parameter | Data Type | Pass As |
| - | - | - |
| `stream` | boolean | Request body parameter |
| `stream_options` | object | Request body parameter |

### `stream`

You can turn streaming on and off by setting the `stream` parameter to `true` or `false`, respectively. It controls whether Friendli Engine streams a response as the model generates it. By default, it's off.

If streaming is off, Friendli Engine waits for the model to generate the response before returning it. If streaming is on, Friendli Engine streams the response in chunks as the model generates each token, and sends each chunk as a [server-sent event](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) (SSE).

To turn streaming on, you can use the following code:

```python OpenAI Python SDK wrap highlight={16} theme={null}
import os

from openai import OpenAI

client = OpenAI(
  base_url="https://api.friendli.ai/serverless/v1",
  api_key=os.environ["FRIENDLIAI_API_KEY"],
)

stream = client.chat.completions.create(
  model="zai-org/GLM-5.3",
  messages=[
    {"role": "system", "content": "You are a friendly assistant."},
    {"role": "user", "content": "Describe FriendliAI in one sentence."},
  ],
  stream=True,
)

# Print the response.
content_started = False
for chunk in stream:
  delta = chunk.choices[0].delta
  if delta.content and not content_started: content_started = True; print("\n\n---\n\n", end="")
  print(getattr(delta, "reasoning_content", None) or delta.content or "", end="", flush=True)
```

```javascript OpenAI JavaScript SDK wrap highlight={14} theme={null}
import OpenAI from "openai"

const client = new OpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.FRIENDLIAI_API_KEY,
})

const stream = await client.chat.completions.create({
  model: "zai-org/GLM-5.3",
  messages: [
    { role: "system", content: "You are a friendly assistant." },
    { role: "user", content: "Describe FriendliAI in one sentence." },
  ],
  stream: true,
})

// Print the response.
let contentStarted = false
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta
  if (delta?.content && !contentStarted) { contentStarted = true; process.stdout.write("\n\n---\n\n") }
  process.stdout.write(delta?.reasoning_content || delta?.content || "")
}
```

```bash cURL wrap highlight={10} theme={null}
curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."}
    ],
    "stream": true
}'
```

### `stream_options`

With the `stream_options` parameter, you can control how Friendli Engine streams a response by setting its fields.

You can set the following fields:

| Field | Data Type |
| - | - |
| `include_usage` | boolean |

#### `include_usage`

If you turn streaming on, you can include or exclude token usage by setting the `include_usage` field to `true` or `false`, respectively. It controls whether Friendli Engine includes token usage in the stream. By default, Friendli Engine excludes it.

If you need token usage (for example, to track costs), include it. You get it in an additional chunk at the end of the stream. Note that your choice doesn't change the number of tokens the model uses to generate a response. You pay the same either way.

To include token usage, you can use the following code:

```python OpenAI Python SDK wrap highlight={17-19} theme={null}
import os

from openai import OpenAI

client = OpenAI(
  base_url="https://api.friendli.ai/serverless/v1",
  api_key=os.environ["FRIENDLIAI_API_KEY"],
)

stream = client.chat.completions.create(
  model="zai-org/GLM-5.3",
  messages=[
    {"role": "system", "content": "You are a friendly assistant."},
    {"role": "user", "content": "Describe FriendliAI in one sentence."},
  ],
  stream=True,
  stream_options={
    "include_usage": True,
  },
)

# Print the response.
content_started = False
for chunk in stream:
  if chunk.usage:
    print(f"\n\n---\n\n{chunk.usage.model_dump_json(indent=2, exclude_none=True)}")
  else:
    delta = chunk.choices[0].delta
    if delta.content and not content_started: content_started = True; print("\n\n---\n\n", end="")
    print(getattr(delta, "reasoning_content", None) or delta.content or "", end="", flush=True)
```

```javascript OpenAI JavaScript SDK wrap highlight={15-17} theme={null}
import OpenAI from "openai"

const client = new OpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.FRIENDLIAI_API_KEY,
})

const stream = await client.chat.completions.create({
  model: "zai-org/GLM-5.3",
  messages: [
    { role: "system", content: "You are a friendly assistant." },
    { role: "user", content: "Describe FriendliAI in one sentence." },
  ],
  stream: true,
  stream_options: {
    "include_usage": true,
  },
})

// Print the response.
let contentStarted = false
for await (const chunk of stream) {
  if (chunk.usage) {
    console.log(`\n\n---\n\n${JSON.stringify(chunk.usage, null, 2)}`)
  } else {
    const delta = chunk.choices[0].delta
    if (delta?.content && !contentStarted) { contentStarted = true; process.stdout.write("\n\n---\n\n") }
    process.stdout.write(delta?.reasoning_content || delta?.content || "")
  }
}
```

```bash cURL wrap highlight={11-13} theme={null}
curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."}
    ],
    "stream": true,
    "stream_options": {
      "include_usage": true
    }
}'
```

## Read a Stream

Friendli Engine streams incremental chunks in an event stream. For example, you get the following stream from a reasoning model:

```json wrap theme={null}
data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "role": "assistant", "reasoning": "Hmm", "reasoning_content": "Hmm" }, "finish_reason": null, "logprobs": null }],
  "created": 1790718391,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

...

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "reasoning": ".", "reasoning_content": "." }, "finish_reason": null, "logprobs": null }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

...

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "content": "FriendliAI" }, "finish_reason": null, "logprobs": null }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "content": "." }, "finish_reason": null, "logprobs": null }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": {}, "logprobs": null, "finish_reason": "stop" }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [],
  "usage": { "completion_tokens": 342, "prompt_tokens": 27, "prompt_tokens_details": { "cached_tokens": 0 }, "total_tokens": 369 },
  "created": 1790718392,
  "model": "zai-org/GLM-5.3"
}

data: [DONE]
```

The first chunk in the stream sets `role` to `assistant`. To learn how to read the rest of the stream, see the following:

* **Reasoning chunks:** If the model reasons, you get its chain of thought first. It comes in a stream of reasoning chunks, where each chunk usually contains one token in both `reasoning` and `reasoning_content`.

  To learn more, see [Reasoning](https://friendli.ai/docs/guides/capabilities/reasoning.md).

* **Content chunks:** You get the model's response next. It comes in a stream of content chunks, where each chunk usually contains one token in `content`.

* **Finish chunk:** After the last token, you get a finish chunk. Its `delta` object is empty, and its `finish_reason` is `stop`, `tool_calls` (if the model calls a tool), or `length` (if the model reaches a token limit, such as `max_tokens`).

* **Usage chunk:** If you include token usage, you get an additional chunk with token counts, such as `prompt_tokens` and `completion_tokens`. Its `choices` array is empty, and you should guard against it.

  To learn more, see [`include_usage`](#include_usage).

* **Done message:** The stream ends with `data: [DONE]`.

## Cancel a Stream

You can cancel a stream by closing its connection. When you close it, the model stops generating. If you pay per token, you pay only for the tokens you use, such as your prompt tokens and the completion tokens up to the point the model stops generating.

If, for example, a user wants to stop the model, you can close the connection for them. The stream ends midstream with no finish chunk, usage chunk, or done message.

To cancel a stream, you can use the following code:

```python OpenAI Python SDK wrap highlight={26-27} theme={null}
import os

from openai import OpenAI

client = OpenAI(
  base_url="https://api.friendli.ai/serverless/v1",
  api_key=os.environ["FRIENDLIAI_API_KEY"],
)

stream = client.chat.completions.create(
  model="zai-org/GLM-5.3",
  messages=[
    {"role": "system", "content": "You are a friendly assistant."},
    {"role": "user", "content": "Describe FriendliAI in one sentence."},
  ],
  stream=True,
)

# Print the response.
content_started = False
try:
  for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content and not content_started: content_started = True; print("\n\n---\n\n", end="")
    print(getattr(delta, "reasoning_content", None) or delta.content or "", end="", flush=True)
except KeyboardInterrupt:
  stream.close()
```

```javascript OpenAI JavaScript SDK wrap highlight={8-9,18} theme={null}
import OpenAI from "openai"

const client = new OpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.FRIENDLIAI_API_KEY,
})

const controller = new AbortController()
process.once("SIGINT", () => controller.abort())

const stream = await client.chat.completions.create({
  model: "zai-org/GLM-5.3",
  messages: [
    { role: "system", content: "You are a friendly assistant." },
    { role: "user", content: "Describe FriendliAI in one sentence." },
  ],
  stream: true,
}, { signal: controller.signal })

// Print the response.
let contentStarted = false
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta
  if (delta?.content && !contentStarted) { contentStarted = true; process.stdout.write("\n\n---\n\n") }
  process.stdout.write(delta?.reasoning_content || delta?.content || "")
}
```

## API Reference

Look up operations and parameters:

> [card] [Model APIs](https://friendli.ai/docs/openapi/model-apis/chat-completions.md)
>
> Learn to use the API for a curated set of models available at usage-based pricing.

> [card] [Dedicated Endpoints](https://friendli.ai/docs/openapi/dedicated/inference/chat-completions.md)
>
> Learn to use the API for any model you deploy on dedicated GPUs reserved for you.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.