Streaming

Models generate responses one token at a time. Instead of waiting for the entire response, you can get each token as the model generates it. This streaming improves the response-time experience—especially for longer responses, where someone would otherwise have to wait with no feedback from the model. You can control whether and how Friendli Engine streams a response. Once you turn streaming on, you need to change only how you read and handle the response. You can also cancel a stream to stop the model from generating tokens.

Streaming Parameters

With streaming parameters, you can control whether and how Friendli Engine streams a response. You pass them as request body parameters. To learn more, see Control a Capability. You can control streaming with the following parameters:

Parameter	Data Type	Pass As
`stream`	boolean	Request body parameter
`stream_options`	object	Request body parameter

`stream`

You can turn streaming on and off by setting the stream parameter to true or false, respectively. It controls whether Friendli Engine streams a response as the model generates it. By default, it’s off. If streaming is off, Friendli Engine waits for the model to generate the response before returning it. If streaming is on, Friendli Engine streams the response in chunks as the model generates each token, and sends each chunk as a server-sent event (SSE). To turn streaming on, you can use the following code:

import os

from openai import OpenAI

client = OpenAI(
  base_url="https://api.friendli.ai/serverless/v1",
  api_key=os.environ["FRIENDLIAI_API_KEY"],
)

stream = client.chat.completions.create(
  model="zai-org/GLM-5.3",
  messages=[
    {"role": "system", "content": "You are a friendly assistant."},
    {"role": "user", "content": "Describe FriendliAI in one sentence."},
  ],
  stream=True,
)

# Print the response.
content_started = False
for chunk in stream:
  delta = chunk.choices[0].delta
  if delta.content and not content_started: content_started = True; print("\n\n---\n\n", end="")
  print(getattr(delta, "reasoning_content", None) or delta.content or "", end="", flush=True)

import OpenAI from "openai"

const client = new OpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.FRIENDLIAI_API_KEY,
})

const stream = await client.chat.completions.create({
  model: "zai-org/GLM-5.3",
  messages: [
    { role: "system", content: "You are a friendly assistant." },
    { role: "user", content: "Describe FriendliAI in one sentence." },
  ],
  stream: true,
})

// Print the response.
let contentStarted = false
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta
  if (delta?.content && !contentStarted) { contentStarted = true; process.stdout.write("\n\n---\n\n") }
  process.stdout.write(delta?.reasoning_content || delta?.content || "")
}

curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."}
    ],
    "stream": true
}'

`stream_options`

With the stream_options parameter, you can control how Friendli Engine streams a response by setting its fields. You can set the following fields:

Field	Data Type
`include_usage`	boolean

`include_usage`

If you turn streaming on, you can include or exclude token usage by setting the include_usage field to true or false, respectively. It controls whether Friendli Engine includes token usage in the stream. By default, Friendli Engine excludes it. If you need token usage (for example, to track costs), include it. You get it in an additional chunk at the end of the stream. Note that your choice doesn’t change the number of tokens the model uses to generate a response. You pay the same either way. To include token usage, you can use the following code:

import os

from openai import OpenAI

client = OpenAI(
  base_url="https://api.friendli.ai/serverless/v1",
  api_key=os.environ["FRIENDLIAI_API_KEY"],
)

stream = client.chat.completions.create(
  model="zai-org/GLM-5.3",
  messages=[
    {"role": "system", "content": "You are a friendly assistant."},
    {"role": "user", "content": "Describe FriendliAI in one sentence."},
  ],
  stream=True,
  stream_options={
    "include_usage": True,
  },
)

# Print the response.
content_started = False
for chunk in stream:
  if chunk.usage:
    print(f"\n\n---\n\n{chunk.usage.model_dump_json(indent=2, exclude_none=True)}")
  else:
    delta = chunk.choices[0].delta
    if delta.content and not content_started: content_started = True; print("\n\n---\n\n", end="")
    print(getattr(delta, "reasoning_content", None) or delta.content or "", end="", flush=True)

import OpenAI from "openai"

const client = new OpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.FRIENDLIAI_API_KEY,
})

const stream = await client.chat.completions.create({
  model: "zai-org/GLM-5.3",
  messages: [
    { role: "system", content: "You are a friendly assistant." },
    { role: "user", content: "Describe FriendliAI in one sentence." },
  ],
  stream: true,
  stream_options: {
    "include_usage": true,
  },
})

// Print the response.
let contentStarted = false
for await (const chunk of stream) {
  if (chunk.usage) {
    console.log(`\n\n---\n\n${JSON.stringify(chunk.usage, null, 2)}`)
  } else {
    const delta = chunk.choices[0].delta
    if (delta?.content && !contentStarted) { contentStarted = true; process.stdout.write("\n\n---\n\n") }
    process.stdout.write(delta?.reasoning_content || delta?.content || "")
  }
}

curl -X POST https://api.friendli.ai/serverless/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $FRIENDLIAI_API_KEY" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [
      {"role": "system", "content": "You are a friendly assistant."},
      {"role": "user", "content": "Describe FriendliAI in one sentence."}
    ],
    "stream": true,
    "stream_options": {
      "include_usage": true
    }
}'

Read a Stream

Friendli Engine streams incremental chunks in an event stream. For example, you get the following stream from a reasoning model:

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "role": "assistant", "reasoning": "Hmm", "reasoning_content": "Hmm" }, "finish_reason": null, "logprobs": null }],
  "created": 1790718391,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

...

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "reasoning": ".", "reasoning_content": "." }, "finish_reason": null, "logprobs": null }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

...

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "content": "FriendliAI" }, "finish_reason": null, "logprobs": null }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": { "content": "." }, "finish_reason": null, "logprobs": null }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [{ "index": 0, "delta": {}, "logprobs": null, "finish_reason": "stop" }],
  "created": 1790718392,
  "model": "zai-org/GLM-5.3",
  "usage": null
}

data: {
  "id": "chatcmpl-d0c8285b-72ed-4d47-87e5-6fec649d74eb",
  "object": "chat.completion.chunk",
  "choices": [],
  "usage": { "completion_tokens": 342, "prompt_tokens": 27, "prompt_tokens_details": { "cached_tokens": 0 }, "total_tokens": 369 },
  "created": 1790718392,
  "model": "zai-org/GLM-5.3"
}

data: [DONE]

The first chunk in the stream sets role to assistant. To learn how to read the rest of the stream, see the following:

Reasoning chunks: If the model reasons, you get its chain of thought first. It comes in a stream of reasoning chunks, where each chunk usually contains one token in both reasoning and reasoning_content. To learn more, see Reasoning.
Content chunks: You get the model’s response next. It comes in a stream of content chunks, where each chunk usually contains one token in content.
Finish chunk: After the last token, you get a finish chunk. Its delta object is empty, and its finish_reason is stop, tool_calls (if the model calls a tool), or length (if the model reaches a token limit, such as max_tokens).
Usage chunk: If you include token usage, you get an additional chunk with token counts, such as prompt_tokens and completion_tokens. Its choices array is empty, and you should guard against it. To learn more, see include_usage.
Done message: The stream ends with data: [DONE].

Cancel a Stream

You can cancel a stream by closing its connection. When you close it, the model stops generating. If you pay per token, you pay only for the tokens you use, such as your prompt tokens and the completion tokens up to the point the model stops generating. If, for example, a user wants to stop the model, you can close the connection for them. The stream ends midstream with no finish chunk, usage chunk, or done message. To cancel a stream, you can use the following code:

import os

from openai import OpenAI

client = OpenAI(
  base_url="https://api.friendli.ai/serverless/v1",
  api_key=os.environ["FRIENDLIAI_API_KEY"],
)

stream = client.chat.completions.create(
  model="zai-org/GLM-5.3",
  messages=[
    {"role": "system", "content": "You are a friendly assistant."},
    {"role": "user", "content": "Describe FriendliAI in one sentence."},
  ],
  stream=True,
)

# Print the response.
content_started = False
try:
  for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content and not content_started: content_started = True; print("\n\n---\n\n", end="")
    print(getattr(delta, "reasoning_content", None) or delta.content or "", end="", flush=True)
except KeyboardInterrupt:
  stream.close()

import OpenAI from "openai"

const client = new OpenAI({
  baseURL: "https://api.friendli.ai/serverless/v1",
  apiKey: process.env.FRIENDLIAI_API_KEY,
})

const controller = new AbortController()
process.once("SIGINT", () => controller.abort())

const stream = await client.chat.completions.create({
  model: "zai-org/GLM-5.3",
  messages: [
    { role: "system", content: "You are a friendly assistant." },
    { role: "user", content: "Describe FriendliAI in one sentence." },
  ],
  stream: true,
}, { signal: controller.signal })

// Print the response.
let contentStarted = false
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta
  if (delta?.content && !contentStarted) { contentStarted = true; process.stdout.write("\n\n---\n\n") }
  process.stdout.write(delta?.reasoning_content || delta?.content || "")
}

API Reference

Look up operations and parameters:

Model APIs

Learn to use the API for a curated set of models available at usage-based pricing.

Dedicated Endpoints

Learn to use the API for any model you deploy on dedicated GPUs reserved for you.

Introduction

Capabilities

Friendli Model APIs

Friendli Dedicated Endpoints

Friendli Suite Guide

Streaming Parameters

`stream`

`stream_options`

`include_usage`

Read a Stream

Cancel a Stream

API Reference

Model APIs

Dedicated Endpoints

​Streaming Parameters

​stream

​stream_options

​include_usage

​Read a Stream

​Cancel a Stream

​API Reference

Model APIs

Dedicated Endpoints

Streaming Parameters

`stream`

`stream_options`

`include_usage`

Read a Stream

Cancel a Stream

API Reference