openapi: 3.1.0 info: title: Friendli Suite API Reference description: This is an OpenAPI reference of Friendli Suite API. termsOfService: https://friendli.ai/terms-of-service contact: name: FriendliAI Support Team email: support@friendli.ai version: 0.1.0 servers: - url: https://api.friendli.ai paths: /v1/chat/completions: post: tags: - Container.Chat summary: Chat completions description: >- Given a list of messages forming a conversation, the model generates a response. operationId: containerChatComplete requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerChatCompletionBody' required: true responses: '200': description: Successfully generated a chat response. content: application/json: schema: $ref: '#/components/schemas/ContainerChatCompleteSuccess' examples: Example: value: id: chatcmpl-4b71d12c86d94e719c7e3984a7bb7941 object: chat.completion choices: - index: 0 message: role: assistant content: Hello there, how may I assist you today? finish_reason: stop usage: prompt_tokens: 9 completion_tokens: 11 total_tokens: 20 created: 1735722153 '422': description: Unprocessable Entity x-speakeasy-name-override: complete servers: - url: http://localhost:8000 /v1/chat/completions#stream: post: tags: - Container.Chat summary: Stream chat completions description: >- Given a list of messages forming a conversation, the model generates a response. operationId: containerChatStream requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerChatCompletionStreamBody' required: true responses: '200': description: Successfully generated a chat response. content: text/event-stream: examples: Example: value: > data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"This"},"finish_reason":null,"logprobs":null}],"usage":null,"created":1726294381} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" is"},"finish_reason":null,"logprobs":null}],"usage":null,"created":1726294381} ... data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop","logprobs":null}],"usage":null,"created":1726294383} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":8,"completion_tokens":4,"total_tokens":12},"created":1726294402} data: [DONE] x-speakeasy-sse-sentinel: '[DONE]' schema: $ref: '#/components/schemas/ContainerChatCompletionStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream servers: - url: http://localhost:8000 /v1/messages: post: tags: - Container.Messages summary: Messages description: Generate responses using Anthropic Messages-style payloads. operationId: containerMessages requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerMessagesBody' required: true responses: '200': description: >- Successfully generated a Messages-style response. For streaming (`text/event-stream`) event and chunk details, see [Messages chunk object](/openapi/container/messages-chunk-object). content: application/json: schema: $ref: '#/components/schemas/ContainerMessagesSuccess' examples: Example: value: id: msg_4b71d12c86d94e719c7e3984a7bb7941 type: message role: assistant content: - type: text text: >- I can answer questions, generate text, and help with coding tasks. stop_reason: end_turn usage: input_tokens: 17 output_tokens: 14 cache_read_input_tokens: 0 model: meta-llama/Llama-3.1-8B-Instruct '422': description: Unprocessable Entity content: application/json: schema: $ref: '#/components/schemas/MessagesErrorResponse' examples: Invalid request error: value: type: error error: type: invalid_request_error message: ... request_id: req_... x-speakeasy-name-override: messages servers: - url: http://localhost:8000 /v1/completions: post: tags: - Container.Completions summary: Completions description: Generate text based on the given text prompt. operationId: containerCompletionsComplete requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerCompletionsBody' required: true responses: '200': description: Successfully generated completions. content: application/json: schema: $ref: '#/components/schemas/ContainerCompletionsSuccess' examples: Example: value: id: cmpl-26a1e10db8544bc3adb488d2d205288b object: text_completion choices: - index: 0 seed: 42 text: This is indeed a test tokens: - 128000 - 2028 - 374 - 13118 - 264 - 1296 finish_reason: stop usage: prompt_tokens: 7 completion_tokens: 6 total_tokens: 13 '422': description: Unprocessable Entity x-speakeasy-name-override: complete servers: - url: http://localhost:8000 /v1/completions#stream: post: tags: - Container.Completions summary: Stream completions description: Generate text based on the given text prompt. operationId: containerCompletionsStream requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerCompletionsStreamBody' required: true responses: '200': description: Successfully generated a chat response. content: text/event-stream: examples: Example: value: > data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":" such","token":1778,"finish_reason":null,"logprobs":null}],"created":1733382157} data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":" as","token":439,"finish_reason":null,"logprobs":null}],"created":1733382157} ... data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":"","finish_reason":"length","logprobs":null}],"created":1733382157} data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[],"usage":{"prompt_tokens":5,"completion_tokens":10,"total_tokens":15},"created":1733382157} data: [DONE] x-speakeasy-sse-sentinel: '[DONE]' schema: $ref: '#/components/schemas/ContainerCompletionsStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream servers: - url: http://localhost:8000 /v1/tokenize: post: tags: - Container.Token summary: Tokenization description: By giving a text input, generate a tokenized output of token IDs. operationId: containerTokenization requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerTokenizationBody' required: true responses: '200': description: Successfully tokenized the text. content: application/json: schema: $ref: '#/components/schemas/ContainerTokenizationSuccess' examples: Example: value: tokens: - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 '422': description: Unprocessable Entity x-speakeasy-name-override: tokenize servers: - url: http://localhost:8000 /v1/detokenize: post: tags: - Container.Token summary: Detokenization description: By giving a list of tokens, generate a detokenized output text string. operationId: containerDetokenization requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerDetokenizationBody' required: true responses: '200': description: Successfully detokenized the tokens. content: application/json: schema: $ref: '#/components/schemas/ContainerDetokenizationSuccess' examples: Example: value: text: What is generative AI? '422': description: Unprocessable Entity x-speakeasy-name-override: detokenize servers: - url: http://localhost:8000 /v1/images/generations: post: tags: - Container.Image summary: Image generations description: Given a description, the model generates image. operationId: containerImagesGenerate requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerImageGenerationBody' required: true responses: '200': description: Successfully generated image. content: application/json: schema: $ref: '#/components/schemas/ContainerImageGenerateSuccess' examples: Example: value: image: '422': description: Unprocessable Entity x-speakeasy-name-override: generate servers: - url: http://localhost:8000 /v1/images/edits: post: tags: - Container.Image summary: Image edits description: Given an image and a description, the model edits the image. operationId: containerImagesEdit requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerImageEditBody' required: true responses: '200': description: Successfully edited image. content: application/json: schema: $ref: '#/components/schemas/ContainerImageEditSuccess' examples: Example: value: image: '422': description: Unprocessable Entity x-speakeasy-name-override: edit servers: - url: http://localhost:8000 /v1/audio/transcriptions: post: tags: - Container.Audio summary: Audio transcriptions description: Given an audio file, the model transcribes it into text. operationId: containerAudioTranscriptions requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/ContainerAudioTranscriptionBody' required: true responses: '200': description: Successfully transcribed the audio file. content: application/json: schema: $ref: '#/components/schemas/ContainerAudioTranscriptionSuccess' examples: Example: value: text: Hello, how are you? usage: type: tokens input_tokens: 20 output_tokens: 10 total_tokens: 30 input_audio_length_ms: 18000 processed_audio_length_ms: 24000 input_token_details: audio_tokens: 10 text_tokens: 10 '422': description: Unprocessable Entity x-speakeasy-name-override: transcribe servers: - url: http://localhost:8000 /v1/audio/transcriptions#stream: post: tags: - Container.Audio summary: Stream audio transcriptions description: Given an audio file, the model transcribes it into text. operationId: containerAudioTranscriptionsStream requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/ContainerAudioTranscriptionStreamBody' required: true responses: '200': description: Successfully transcribed the audio file. content: text/event-stream: examples: Example: value: > data: {"type":"transcript.text.delta","delta":"The"} data: {"type":"transcript.text.delta","delta":" quick"} ... data: {"type":"transcript.text.done","text":"The quick brown fox jumps over the lazy dog.","usage":{"type":"tokens","input_tokens":20,"output_tokens":10,"total_tokens":30,"input_audio_length_ms":18000,"processed_audio_length_ms":24000,"input_token_details":{"audio_tokens":10,"text_tokens":10}}} data: [DONE] x-speakeasy-sse-sentinel: '[DONE]' schema: $ref: '#/components/schemas/ContainerAudioTranscriptionStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream servers: - url: http://localhost:8000 /classify: post: tags: - Container.Classification summary: Text classification description: Given a text input, the model classifies it into categories. operationId: containerTextClassification requestBody: content: application/json: schema: $ref: '#/components/schemas/ContainerTextClassificationBody' required: true responses: '200': description: Successfully classified the text input. content: application/json: schema: $ref: '#/components/schemas/ContainerTextClassificationSuccess' examples: Example: value: data: - index: 0 label: Positive num_classes: 2 probs: - 0.9 - 0.1 object: list usage: prompt_tokens: 5 total_tokens: 5 '422': description: Unprocessable Entity x-speakeasy-name-override: classify servers: - url: http://localhost:8000 /dedicated/v1/chat/completions: post: tags: - Dedicated.Chat summary: Chat completions description: >- Given a list of messages forming a conversation, the model generates a response. operationId: dedicatedChatComplete security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedChatCompletionBody' responses: '200': description: Successfully generated a chat response. content: application/json: schema: $ref: '#/components/schemas/DedicatedChatCompleteSuccess' examples: Example: value: id: chatcmpl-4b71d12c86d94e719c7e3984a7bb7941 model: (endpoint-id) object: chat.completion choices: - index: 0 message: role: assistant content: Hello there, how may I assist you today? finish_reason: stop usage: prompt_tokens: 9 completion_tokens: 11 total_tokens: 20 created: 1735722153 '422': description: Unprocessable Entity x-speakeasy-name-override: complete /dedicated/v1/chat/completions#stream: post: tags: - Dedicated.Chat summary: Stream chat completions description: >- Given a list of messages forming a conversation, the model generates a response. operationId: dedicatedChatStream security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedChatCompletionStreamBody' responses: '200': description: Successfully generated a chat response. content: text/event-stream: x-speakeasy-sse-sentinel: '[DONE]' examples: Example: value: > data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"This"},"finish_reason":null,"logprobs":null}],"usage":null,"created":1726294381} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" is"},"finish_reason":null,"logprobs":null}],"usage":null,"created":1726294381} ... data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop","logprobs":null}],"usage":null,"created":1726294383} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":8,"completion_tokens":4,"total_tokens":12},"created":1726294402} data: [DONE] schema: $ref: '#/components/schemas/DedicatedChatCompletionStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream x-speakeasy-usage-example: tags: - eventstream /dedicated/v1/chat/render: post: tags: - Dedicated.ChatRender summary: Chat render description: >- Given a list of messages forming a conversation, the API renders them into the final prompt text that will be sent to the model. operationId: dedicatedChatRender security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedChatRenderBody' responses: '200': description: Successfully rendered chat messages into prompt text. content: application/json: schema: $ref: '#/components/schemas/DedicatedChatRenderSuccess' examples: Example: value: text: > <|begin_of_text|><|start_header_id|>system<|end_header_id|> You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|> Hello!<|eot_id|><|start_header_id|>assistant<|end_header_id|> '422': description: Unprocessable Entity x-speakeasy-name-override: render /dedicated/v1/messages: post: tags: - Dedicated.Messages summary: Messages description: Generate responses using Anthropic Messages-style payloads. operationId: dedicatedMessages security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedMessagesBody' responses: '200': description: >- Successfully generated a Messages-style response. For streaming (`text/event-stream`) event and chunk details, see [Messages chunk object](/openapi/dedicated/inference/messages-chunk-object). content: application/json: schema: $ref: '#/components/schemas/DedicatedMessagesSuccess' examples: Example: value: id: msg_4b71d12c86d94e719c7e3984a7bb7941 type: message role: assistant content: - type: text text: >- I can answer questions, generate text, and help with coding tasks. stop_reason: end_turn usage: input_tokens: 17 output_tokens: 14 cache_read_input_tokens: 0 model: (endpoint-id) '422': description: Unprocessable Entity content: application/json: examples: Invalid request error: value: type: error error: type: invalid_request_error message: ... request_id: req_... schema: $ref: '#/components/schemas/MessagesErrorResponse' x-speakeasy-name-override: messages /dedicated/v1/completions: post: tags: - Dedicated.Completions summary: Completions description: Generate text based on the given text prompt. operationId: dedicatedCompletionsComplete security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedCompletionsBody' responses: '200': description: Successfully generated completions. content: application/json: schema: $ref: '#/components/schemas/DedicatedCompletionsSuccess' examples: Example: value: id: cmpl-26a1e10db8544bc3adb488d2d205288b model: (endpoint-id) object: text_completion choices: - index: 0 seed: 42 text: This is indeed a test tokens: - 128000 - 2028 - 374 - 13118 - 264 - 1296 finish_reason: stop usage: prompt_tokens: 7 completion_tokens: 6 total_tokens: 13 '422': description: Unprocessable Entity x-speakeasy-name-override: complete /dedicated/v1/completions#stream: post: tags: - Dedicated.Completions summary: Stream completions description: Generate text based on the given text prompt. operationId: dedicatedCompletionsStream security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedCompletionsStreamBody' responses: '200': description: Successfully generated a chat response. content: text/event-stream: x-speakeasy-sse-sentinel: '[DONE]' examples: Example: value: > data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":" such","token":1778,"finish_reason":null,"logprobs":null}],"created":1733382157} data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":" as","token":439,"finish_reason":null,"logprobs":null}],"created":1733382157} ... data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":"","finish_reason":"length","logprobs":null}],"created":1733382157} data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[],"usage":{"prompt_tokens":5,"completion_tokens":10,"total_tokens":15},"created":1733382157} data: [DONE] schema: $ref: '#/components/schemas/DedicatedCompletionsStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream x-speakeasy-usage-example: tags: - eventstream /dedicated/v1/embeddings: post: tags: - Dedicated.Embeddings summary: Embeddings description: Creates an embedding vector representing the input text. operationId: dedicatedEmbeddings security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedEmbeddingsBody' responses: '200': description: Successfully generated embeddings. content: application/json: schema: $ref: '#/components/schemas/DedicatedEmbeddingsSuccess' examples: Example: value: id: embd-26a1e10db1311bc2adb488d2d205288b model: (endpoint-id) object: list data: - index: 0 object: embedding embedding: - 0.0023064255 - -0.009327292 - -0.0028842222 usage: prompt_tokens: 26 completion_tokens: 0 total_tokens: 26 created: 1735722153 '422': description: Unprocessable Entity x-speakeasy-name-override: embeddings /dedicated/v1/tokenize: post: tags: - Dedicated.Token summary: Tokenization description: By giving a text input, generate a tokenized output of token IDs. operationId: dedicatedTokenization security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedTokenizationBody' responses: '200': description: Successfully tokenized the text. content: application/json: schema: $ref: '#/components/schemas/DedicatedTokenizationSuccess' examples: Example: value: tokens: - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 '422': description: Unprocessable Entity x-speakeasy-name-override: tokenize /dedicated/v1/detokenize: post: tags: - Dedicated.Token summary: Detokenization description: By giving a list of tokens, generate a detokenized output text string. operationId: dedicatedDetokenization security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedDetokenizationBody' responses: '200': description: Successfully detokenized the tokens. content: application/json: schema: $ref: '#/components/schemas/DedicatedDetokenizationSuccess' examples: Example: value: text: What is generative AI? '422': description: Unprocessable Entity x-speakeasy-name-override: detokenize /dedicated/v1/images/generations: post: tags: - Dedicated.Image summary: Image generations description: Given a description, the model generates image(s). operationId: dedicatedImagesGenerate security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedImageGenerationBody' responses: '200': description: Successfully generated image(s). content: application/json: schema: $ref: '#/components/schemas/DedicatedImageGenerateSuccess' examples: Example: value: data: - url: (url-to-generated-image) seed: 123 response_format: url '422': description: Unprocessable Entity x-speakeasy-name-override: generate /dedicated/v1/images/edits: post: tags: - Dedicated.Image summary: Image edits description: Given an image and a description, the model edits the image. operationId: dedicatedImagesEdit security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedImageEditBody' responses: '200': description: Successfully edited image(s). content: application/json: schema: $ref: '#/components/schemas/DedicatedImageEditSuccess' examples: Example: value: data: - url: (url-to-edited-image) seed: 789 response_format: url '422': description: Unprocessable Entity x-speakeasy-name-override: edit /dedicated/v1/audio/transcriptions: post: tags: - Dedicated.Audio summary: Audio transcriptions description: Given an audio file, the model transcribes it into text. operationId: dedicatedAudioTranscriptions security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: multipart/form-data: schema: $ref: '#/components/schemas/DedicatedAudioTranscriptionBody' responses: '200': description: Successfully transcribed the audio file. content: application/json: schema: $ref: '#/components/schemas/DedicatedAudioTranscriptionSuccess' examples: Example: value: text: Hello, how are you? usage: type: tokens input_tokens: 20 output_tokens: 10 total_tokens: 30 input_audio_length_ms: 18000 processed_audio_length_ms: 24000 input_token_details: audio_tokens: 10 text_tokens: 10 '422': description: Unprocessable Entity x-speakeasy-name-override: transcribe /dedicated/v1/audio/transcriptions#stream: post: tags: - Dedicated.Audio summary: Stream audio transcriptions description: Given an audio file, the model transcribes it into text. operationId: dedicatedAudioTranscriptionsStream security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: multipart/form-data: schema: $ref: '#/components/schemas/DedicatedAudioTranscriptionStreamBody' responses: '200': description: Successfully transcribed the audio file. content: text/event-stream: x-speakeasy-sse-sentinel: '[DONE]' examples: Example: value: > data: {"type":"transcript.text.delta","delta":"The"} data: {"type":"transcript.text.delta","delta":" quick"} ... data: {"type":"transcript.text.done","text":"The quick brown fox jumps over the lazy dog.","usage":{"type":"tokens","input_tokens":20,"output_tokens":10,"total_tokens":30,"input_audio_length_ms":18000,"processed_audio_length_ms":24000,"input_token_details":{"audio_tokens":10,"text_tokens":10}}} data: [DONE] schema: $ref: '#/components/schemas/DedicatedAudioTranscriptionStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream /dedicated/classify: post: tags: - Dedicated.Classification summary: Text classification description: Given a text input, the model classifies it into categories. operationId: dedicatedTextClassification security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedTextClassificationBody' responses: '200': description: Successfully classified the text input. content: application/json: schema: $ref: '#/components/schemas/DedicatedTextClassificationSuccess' examples: Example: value: data: - index: 0 label: Positive num_classes: 2 probs: - 0.9 - 0.1 object: list usage: prompt_tokens: 5 total_tokens: 5 '422': description: Unprocessable Entity x-speakeasy-name-override: classify /serverless/v1/chat/completions: post: tags: - Serverless.Chat summary: Chat completions description: >- Given a list of messages forming a conversation, the model generates a response. operationId: serverlessChatComplete security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessChatCompletionBody' responses: '200': description: Successfully generated a chat response. content: application/json: schema: $ref: '#/components/schemas/ServerlessChatCompleteSuccess' examples: Example: value: id: chatcmpl-4b71d12c86d94e719c7e3984a7bb7941 model: meta-llama/Llama-3.1-8B-Instruct object: chat.completion choices: - index: 0 message: role: assistant content: Hello there, how may I assist you today? finish_reason: stop usage: prompt_tokens: 9 completion_tokens: 11 total_tokens: 20 created: 1735722153 '422': description: Unprocessable Entity x-speakeasy-name-override: complete x-speakeasy-usage-example: title: Chat completions description: >- Given a list of messages forming a conversation, the model generates a response. position: 1 /serverless/v1/chat/completions#stream: post: tags: - Serverless.Chat summary: Stream chat completions description: >- Given a list of messages forming a conversation, the model generates a response. operationId: serverlessChatStream security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessChatCompletionStreamBody' responses: '200': description: Successfully generated a chat response. content: text/event-stream: x-speakeasy-sse-sentinel: '[DONE]' examples: Example: value: > data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"This"},"finish_reason":null,"logprobs":null}],"usage":null,"created":1726294381} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" is"},"finish_reason":null,"logprobs":null}],"usage":null,"created":1726294381} ... data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop","logprobs":null}],"usage":null,"created":1726294383} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":8,"completion_tokens":4,"total_tokens":12},"created":1726294402} data: [DONE] schema: $ref: '#/components/schemas/ServerlessChatCompletionStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream x-speakeasy-usage-example: tags: - eventstream /serverless/v1/messages: post: tags: - Serverless.Messages summary: Messages description: Generate responses using Anthropic Messages-style payloads. operationId: serverlessMessages security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessMessagesBody' responses: '200': description: >- Successfully generated a Messages-style response. For streaming (`text/event-stream`) event and chunk details, see [Messages chunk object](/openapi/model-apis/messages-chunk-object). content: application/json: schema: $ref: '#/components/schemas/ServerlessMessagesSuccess' examples: Example: value: id: msg_4b71d12c86d94e719c7e3984a7bb7941 type: message role: assistant content: - type: text text: >- I can answer questions, generate text, and help with coding tasks. stop_reason: end_turn usage: input_tokens: 17 output_tokens: 14 cache_read_input_tokens: 0 model: meta-llama/Llama-3.1-8B-Instruct '422': description: Unprocessable Entity content: application/json: examples: Invalid request error: value: type: error error: type: invalid_request_error message: ... request_id: req_... schema: $ref: '#/components/schemas/MessagesErrorResponse' x-speakeasy-name-override: messages /serverless/v1/chat/render: post: tags: - Serverless.ChatRender summary: Chat render description: >- Given a list of messages forming a conversation, the API renders them into the final prompt text that will be sent to the model. operationId: serverlessChatRender security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessChatRenderBody' responses: '200': description: Successfully rendered chat messages into prompt text. content: application/json: schema: $ref: '#/components/schemas/ServerlessChatRenderSuccess' examples: Example: value: text: > <|begin_of_text|><|start_header_id|>system<|end_header_id|> You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|> Hello!<|eot_id|><|start_header_id|>assistant<|end_header_id|> '422': description: Unprocessable Entity x-speakeasy-name-override: render /serverless/v1/completions: post: tags: - Serverless.Completions summary: Completions description: Generate text based on the given text prompt. operationId: serverlessCompletionsComplete security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessCompletionsBody' responses: '200': description: Successfully generated completions. content: application/json: schema: $ref: '#/components/schemas/ServerlessCompletionsSuccess' examples: Example: value: id: cmpl-26a1e10db8544bc3adb488d2d205288b model: meta-llama/Llama-3.1-8B-Instruct object: text_completion choices: - index: 0 seed: 42 text: This is indeed a test tokens: - 128000 - 2028 - 374 - 13118 - 264 - 1296 finish_reason: stop usage: prompt_tokens: 7 completion_tokens: 6 total_tokens: 13 '422': description: Unprocessable Entity x-speakeasy-name-override: complete /serverless/v1/completions#stream: post: tags: - Serverless.Completions summary: Stream completions description: Generate text based on the given text prompt. operationId: serverlessCompletionsStream security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessCompletionsStreamBody' responses: '200': description: Successfully generated a chat response. content: text/event-stream: x-speakeasy-sse-sentinel: '[DONE]' examples: Example: value: > data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":" such","token":1778,"finish_reason":null,"logprobs":null}],"created":1733382157} data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":" as","token":439,"finish_reason":null,"logprobs":null}],"created":1733382157} ... data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[{"index":0,"text":"","finish_reason":"length","logprobs":null}],"created":1733382157} data: {"id":"cmpl-26a1e10db8544bc3adb488d2d205288b","object":"text_completion","choices":[],"usage":{"prompt_tokens":5,"completion_tokens":10,"total_tokens":15},"created":1733382157} data: [DONE] schema: $ref: '#/components/schemas/ServerlessCompletionsStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream x-speakeasy-usage-example: tags: - eventstream /serverless/v1/audio/transcriptions: post: tags: - Serverless.Audio summary: Audio transcriptions description: Given an audio file, the model transcribes it into text. operationId: serverlessAudioTranscriptions security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: multipart/form-data: schema: $ref: '#/components/schemas/ServerlessAudioTranscriptionBody' responses: '200': description: Successfully transcribed the audio file. content: application/json: schema: $ref: '#/components/schemas/ServerlessAudioTranscriptionSuccess' examples: Example: value: text: Hello, how are you? usage: type: tokens input_tokens: 20 output_tokens: 10 total_tokens: 30 input_audio_length_ms: 18000 processed_audio_length_ms: 24000 input_token_details: audio_tokens: 10 text_tokens: 10 '422': description: Unprocessable Entity x-speakeasy-name-override: transcribe /serverless/v1/audio/transcriptions#stream: post: tags: - Serverless.Audio summary: Stream audio transcriptions description: Given an audio file, the model transcribes it into text. operationId: serverlessAudioTranscriptionsStream security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: multipart/form-data: schema: $ref: '#/components/schemas/ServerlessAudioTranscriptionStreamBody' responses: '200': description: Successfully transcribed the audio file. content: text/event-stream: x-speakeasy-sse-sentinel: '[DONE]' examples: Example: value: > data: {"type":"transcript.text.delta","delta":"The"} data: {"type":"transcript.text.delta","delta":" quick"} ... data: {"type":"transcript.text.done","text":"The quick brown fox jumps over the lazy dog.","usage":{"type":"tokens","input_tokens":20,"output_tokens":10,"total_tokens":30,"input_audio_length_ms":18000,"processed_audio_length_ms":24000,"input_token_details":{"audio_tokens":10,"text_tokens":10}}} data: [DONE] schema: $ref: '#/components/schemas/ServerlessAudioTranscriptionStreamSuccess' '422': description: Unprocessable Entity x-speakeasy-name-override: stream /serverless/v1/tokenize: post: tags: - Serverless.Token summary: Tokenization description: By giving a text input, generate a tokenized output of token IDs. operationId: serverlessTokenization security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessTokenizationBody' responses: '200': description: Successfully tokenized the text. content: application/json: schema: $ref: '#/components/schemas/ServerlessTokenizationSuccess' examples: Example: value: tokens: - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 '422': description: Unprocessable Entity x-speakeasy-name-override: tokenize /serverless/v1/detokenize: post: tags: - Serverless.Token summary: Detokenization description: By giving a list of tokens, generate a detokenized output text string. operationId: serverlessDetokenization security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessDetokenizationBody' responses: '200': description: Successfully detokenized the tokens. content: application/json: schema: $ref: '#/components/schemas/ServerlessDetokenizationSuccess' examples: Example: value: text: What is generative AI? '422': description: Unprocessable Entity x-speakeasy-name-override: detokenize /serverless/tools/v1/chat/completions: post: tags: - Serverless.ToolAssistedChat summary: Tool assisted chat completions description: >- Given a list of messages forming a conversation, the model generates a response. Additionally, the model can utilize built-in tools for tool calls, enhancing its capability to provide more comprehensive and actionable responses. operationId: serverlessToolAssistedChatComplete security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessToolAssistedChatCompletionBody' responses: '200': description: Successfully generated a tool assisted chat response. content: application/json: schema: $ref: '#/components/schemas/ServerlessToolAssistedChatCompleteSuccess' examples: Example: value: id: chatcmpl-4b71d12c86d94e719c7e3984a7bb7941 model: meta-llama/Llama-3.1-8B-Instruct object: chat.completion choices: - index: 0 message: role: assistant content: The result is 9. finish_reason: stop usage: prompt_tokens: 9 completion_tokens: 11 total_tokens: 20 created: 1735722153 '422': description: Unprocessable Entity x-speakeasy-name-override: complete x-speakeasy-usage-example: title: Tool assisted chat completions description: >- Given a list of messages forming a conversation, the model generates a response. Additionally, the model can utilize built-in tools for tool calls, enhancing its capability to provide more comprehensive and actionable responses. position: 2 /serverless/tools/v1/chat/completions#stream: post: tags: - Serverless.ToolAssistedChat summary: Stream tool assisted chat completions description: >- Given a list of messages forming a conversation, the model generates a response. Additionally, the model can utilize built-in tools for tool calls, enhancing its capability to provide more comprehensive and actionable responses. operationId: serverlessToolAssistedChatStream security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: >- #/components/schemas/ServerlessToolAssistedChatCompletionStreamBody responses: '200': description: Successfully generated a tool assisted chat response. content: text/event-stream: x-speakeasy-sse-sentinel: '[DONE]' examples: Example: value: > event: tool_status data: {"tool_call_id":"call_3QrfStXSU6fGdOGPcETocIAq","name":"math:calculator","status":"STARTED","parameters":[{"name":"expression","value":"150 * 1.60934"}],"result":null,"files":null,"message":null,"error":null,"usage":null,"timestamp":1726277121} event: tool_status data: {"tool_call_id":"call_3QrfStXSU6fGdOGPcETocIAq","name":"math:calculator","status":"ENDED","parameters":[{"name":"expression","value":"150 * 1.60934"}],"result":""{\"result\": \"150 * 1.60934=241.401000000000\"}"","files":null,"message":null,"error":null,"usage":null,"timestamp":1726277121} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"To"},"finish_reason":null,"logprobs":null}],"created":1726277121} ... data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"."},"finish_reason":null,"logprobs":null}],"created":1726277121} data: {"id":"chatcmpl-4b71d12c86d94e719c7e3984a7bb7941","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop","logprobs":null}],"created":1726277121} data: [DONE] schema: $ref: >- #/components/schemas/ServerlessToolAssistedChatCompletionStreamSuccess '422': description: Unprocessable Entity x-speakeasy-name-override: stream /dedicated/endpoint/wandb-artifact-create: post: tags: - Dedicated.Endpoint summary: Create endpoint from W&B artifact description: >- Create an endpoint from Weights & Biases artifact. If the idempotency key is provided, the API will check if the endpoint already exists, and rollout the existing endpoint if it does. In such cases, the project id must be provided. operationId: dedicatedEndpointWandbArtifactCreate security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointWandbArtifactCreateBody' responses: '200': description: Successfully created the endpoint from W&B artifact. content: application/json: schema: $ref: >- #/components/schemas/DedicatedEndpointWandbArtifactCreateResponse examples: Example: value: endpointId: endpoint-id endpointName: endpoint-name projectId: project-id projectName: project-name teamId: team-id teamName: team-name '400': description: Bad Request '422': description: Unprocessable Entity x-speakeasy-name-override: wandbArtifactCreate /dedicated/beta/endpoint: post: tags: - Dedicated.Endpoint summary: Create a new endpoint description: Create a new endpoint and return its status operationId: dedicatedCreateEndpoint security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointCreateBody' responses: '200': description: Successfully created the endpoint. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointStatus' examples: Example: value: status: INITIALIZING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' phase: DOWNLOADING_MODEL '400': description: Bad Request '422': description: Unprocessable Entity '500': description: Internal Server Error x-speakeasy-name-override: create get: tags: - Dedicated.Endpoint summary: List all endpoints description: List all endpoint statuses operationId: dedicatedListEndpoints security: - token: [] parameters: - name: project_id in: query required: false schema: anyOf: - type: string - type: 'null' description: >- The ID of the project. If omitted, query all endpoints under the team. default: '' title: Project Id description: >- The ID of the project. If omitted, query all endpoints under the team. - name: cursor in: query required: false schema: anyOf: - type: string format: binary - type: 'null' description: Cursor for pagination title: Cursor description: Cursor for pagination - name: limit in: query required: false schema: anyOf: - type: integer maximum: 100 minimum: 1 - type: 'null' description: Limit of items per page default: 20 title: Limit description: Limit of items per page - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully listed endpoints. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointListResponse' examples: Example: value: data: endpoint-id-1: status: INITIALIZING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' phase: DOWNLOADING_MODEL endpoint-id-2: status: RUNNING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' '400': description: Bad Request '422': description: Unprocessable Entity x-speakeasy-name-override: list /dedicated/beta/endpoint/{endpoint_id}: get: tags: - Dedicated.Endpoint summary: Get endpoint specification description: Get the specification of an endpoint operationId: dedicatedGetEndpoint security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully retrieved the endpoint specification. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointSpec' examples: Example: value: name: endpoint-name gpuType: NVIDIA H100 numGpu: 1 instanceId: instance-id projectId: project-id creatorId: creator-id teamId: team-id autoscalingMin: 0 autoscalingMax: 1 autoscalingCooldown: 300 maxBatchSize: 10 maxInputLength: 1024 tokenizerSkipSpecialTokens: true tokenizerAddSpecialTokens: true currReplicaCnt: 1 desiredReplicaCnt: 1 updatedReplicaCnt: 1 '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity x-speakeasy-name-override: getSpec put: tags: - Dedicated.Endpoint summary: Update endpoint spec description: Update the specification of a specific endpoint operationId: dedicatedUpdateEndpoint security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointUpdateBody' responses: '200': description: Successfully updated the endpoint specification. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointSpec' examples: Example: value: name: endpoint-name gpuType: NVIDIA H100 numGpu: 1 instanceId: instance-id projectId: project-id creatorId: creator-id teamId: team-id autoscalingMin: 0 autoscalingMax: 1 autoscalingCooldown: 300 maxBatchSize: 10 maxInputLength: 1024 tokenizerSkipSpecialTokens: true tokenizerAddSpecialTokens: true currReplicaCnt: 1 desiredReplicaCnt: 1 updatedReplicaCnt: 1 '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity x-speakeasy-name-override: update delete: tags: - Dedicated.Endpoint summary: Delete endpoint description: Delete a specific endpoint operationId: dedicatedDeleteEndpoint security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successful Response content: application/json: schema: {} '400': description: Bad Request '404': description: Not Found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: delete /dedicated/beta/endpoint/{endpoint_id}/version: get: tags: - Dedicated.Endpoint summary: Get endpoint version history description: Get version history of a specific endpoint operationId: dedicatedGetEndpointVersionHistory security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: cursor in: query required: false schema: anyOf: - type: string format: binary - type: 'null' description: Cursor for pagination title: Cursor description: Cursor for pagination - name: limit in: query required: false schema: anyOf: - type: integer maximum: 100 minimum: 1 - type: 'null' description: Limit of items per page default: 20 title: Limit description: Limit of items per page - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully retrieved the endpoint version history. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointVersionHistoryResponse' examples: Example: value: data: '0': name: endpoint-name gpuType: NVIDIA H100 numGpu: 1 instanceId: instance-id projectId: project-id creatorId: creator-id teamId: team-id autoscalingMin: 0 autoscalingMax: 1 autoscalingCooldown: 300 maxBatchSize: 10 maxInputLength: 1024 tokenizerSkipSpecialTokens: true tokenizerAddSpecialTokens: true currReplicaCnt: 1 desiredReplicaCnt: 1 updatedReplicaCnt: 1 '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity x-speakeasy-name-override: getVersionHistory /dedicated/beta/endpoint/{endpoint_id}/status: get: tags: - Dedicated.Endpoint summary: Get endpoint status description: Get the status of a specific endpoint operationId: dedicatedGetEndpointStatus security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully retrieved the endpoint status. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointStatus' examples: Example: value: status: INITIALIZING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' phase: DOWNLOADING_MODEL '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity x-speakeasy-name-override: getStatus /dedicated/beta/endpoint/{endpoint_id}/sleep: put: tags: - Dedicated.Endpoint summary: Sleep endpoint description: Put a specific endpoint to sleep operationId: dedicatedSleepEndpoint security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully requested to put the endpoint to sleep. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointStatus' examples: Example: value: status: SLEEPING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity x-speakeasy-name-override: sleep /dedicated/beta/endpoint/{endpoint_id}/wake: put: tags: - Dedicated.Endpoint summary: Wake endpoint description: Wake up a specific endpoint operationId: dedicatedWakeEndpoint security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully requested to wake up the endpoint. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointStatus' examples: Example: value: status: AWAKING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' phase: ENGINE_INITIALIZING '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity '500': description: Internal Server Error x-speakeasy-name-override: wake /dedicated/beta/endpoint/{endpoint_id}/terminate: put: tags: - Dedicated.Endpoint summary: Terminate endpoint description: Terminate a specific endpoint operationId: dedicatedTerminateEndpoint security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully requested to terminate the endpoint. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointStatus' examples: Example: value: status: TERMINATING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity x-speakeasy-name-override: terminate /dedicated/beta/endpoint/{endpoint_id}/restart: put: tags: - Dedicated.Endpoint summary: Restart endpoint description: Restart a FAILED or TERMINATED endpoint operationId: dedicatedRestartEndpoint security: - token: [] parameters: - name: endpoint_id in: path required: true schema: type: string description: The ID of the endpoint title: Endpoint Id description: The ID of the endpoint - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successfully requested to restart the endpoint. content: application/json: schema: $ref: '#/components/schemas/DedicatedEndpointStatus' examples: Example: value: status: INITIALIZING createdAt: '2025-01-01T00:00:00Z' updatedAt: '2025-01-01T00:00:00Z' phase: DOWNLOADING_MODEL '400': description: Bad Request '404': description: Not Found '422': description: Unprocessable Entity '500': description: Internal Server Error x-speakeasy-name-override: restart /serverless/v1/knowledge/retrieve: post: tags: - Serverless.Knowledge summary: Retrieve contexts from chosen knowledge base description: Retrieve related documents from knowledge base by similarity. operationId: serverlessKnowledgeRetrieve security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ServerlessKnowledgeRetrievalBody' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ServerlessKnowledgeRetrievalSuccess' '400': description: Bad Request '422': description: Unprocessable Entity x-speakeasy-name-override: retrieve /serverless/v1/models: get: tags: - Serverless.Model summary: Retrieve serverless models description: Retrieve list of serverless models. operationId: serverlessModelList responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ServerlessModelListSuccess' '400': description: Bad Request '422': description: Unprocessable Entity x-speakeasy-name-override: list /beta/dataset: post: tags: - Dataset summary: Create a new dataset description: Create a new dataset. operationId: create_dataset security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CreateDatasetRequest' responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/DatasetInfo' examples: Example: value: id: 123 name: dataset-name created_at: 1705244400 updated_at: 1765724399 '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: createDataset get: tags: - Dataset summary: List datasets description: List datasets accessible to the user. operationId: list_datasets security: - token: [] parameters: - name: cursor in: query required: false schema: anyOf: - type: string format: binary - type: 'null' title: Cursor - name: limit in: query required: false schema: anyOf: - type: integer maximum: 100 minimum: 1 - type: 'null' default: 20 title: Limit - name: direction in: query required: false schema: anyOf: - enum: - asc - desc type: string - type: 'null' title: Direction - name: projectId in: query required: true schema: type: string title: Projectid - name: nameSearch in: query required: false schema: anyOf: - type: string - type: 'null' title: Namesearch - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/ListDatasetsResponse' examples: Example: value: data: - id: 123 name: dataset-name created_at: 1705244400 updated_at: 1765724399 '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: listDatasets /beta/dataset/{dataset_id}: get: tags: - Dataset summary: Get dataset info description: Get information about a specific dataset. operationId: get_dataset security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset to retrieve. title: Dataset Id description: ID of the dataset to retrieve. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/DatasetInfo' examples: Example: value: id: 123 name: dataset-name created_at: 1705244400 updated_at: 1765724399 '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: getDataset delete: tags: - Dataset summary: Delete dataset description: Delete a specific dataset. operationId: delete_dataset security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset to delete. title: Dataset Id description: ID of the dataset to delete. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: deleteDataset /beta/dataset/{dataset_id}/version: post: tags: - Dataset summary: Create a version description: Create a version for the current state of the split. operationId: create_version security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: comment in: query required: true schema: type: string title: Comment - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/VersionInfo' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: createVersion get: tags: - Dataset summary: List versions description: List versions for a dataset. operationId: list_versions security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/ListVersionsResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: listVersions /beta/dataset/{dataset_id}/version/{version_id}: get: tags: - Dataset summary: Get version info description: Get a dataset version. operationId: get_version security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: version_id in: path required: true schema: type: string description: ID of the version to get. title: Version Id description: ID of the version to get. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/VersionInfo' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: getVersion delete: tags: - Dataset summary: Delete a version description: Delete a version from the dataset. operationId: delete_version security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: version_id in: path required: true schema: type: string description: ID of the version to delete. title: Version Id description: ID of the version to delete. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: deleteVersion /beta/dataset/{dataset_id}/split: post: tags: - Dataset summary: Create a split description: Create a new split in the dataset. operationId: create_split security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: name in: query required: true schema: type: string title: Name - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/SplitInfo' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: createSplit get: tags: - Dataset summary: List splits description: List splits in the dataset. operationId: list_splits security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: cursor in: query required: false schema: anyOf: - type: string format: binary - type: 'null' title: Cursor - name: limit in: query required: false schema: anyOf: - type: integer maximum: 100 minimum: 1 - type: 'null' default: 20 title: Limit - name: direction in: query required: false schema: anyOf: - enum: - asc - desc type: string - type: 'null' title: Direction - name: versionId in: query required: false schema: anyOf: - type: string - type: 'null' title: Versionid - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/ListSplitsResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: listSplits /beta/dataset/{dataset_id}/split/{split_id}: get: tags: - Dataset summary: Get split info description: Get information about a specific split. operationId: get_split security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: split_id in: path required: true schema: type: string description: ID of the split. title: Split Id description: ID of the split. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/SplitInfo' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: getSplit delete: tags: - Dataset summary: Delete split description: Delete a specific split. operationId: delete_split security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: split_id in: path required: true schema: type: string description: ID of the split. title: Split Id description: ID of the split. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: deleteSplit /beta/dataset/{dataset_id}/split/{split_id}/sample: post: tags: - Dataset summary: Add samples description: Add samples to the split. operationId: add_samples security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: split_id in: path required: true schema: type: string description: ID of the split. title: Split Id description: ID of the split. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: type: array items: type: string title: Samples responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/AddSamplesResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: addSamples get: tags: - Dataset summary: List samples description: List samples from the split. operationId: list_samples security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: split_id in: path required: true schema: type: string description: ID of the split. title: Split Id description: ID of the split. - name: cursor in: query required: false schema: anyOf: - type: string format: binary - type: 'null' title: Cursor - name: limit in: query required: false schema: anyOf: - type: integer maximum: 100 minimum: 1 - type: 'null' default: 20 title: Limit - name: direction in: query required: false schema: anyOf: - enum: - asc - desc type: string - type: 'null' title: Direction - name: versionId in: query required: false schema: anyOf: - type: string - type: 'null' title: Versionid - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/ListSamplesResponse' '404': description: Not Found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: listSamples put: tags: - Dataset summary: Update samples description: Update samples as raw file. operationId: upload_raw_samples security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: split_id in: path required: true schema: type: string description: ID of the split. title: Split Id description: ID of the split. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: multipart/form-data: schema: $ref: '#/components/schemas/Body_upload_raw_samples' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/AddSamplesResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: updateSamples /beta/dataset/{dataset_id}/split/{split_id}/sample/delete: post: tags: - Dataset summary: Delete samples description: Delete samples from the split. operationId: delete_samples security: - token: [] parameters: - name: dataset_id in: path required: true schema: type: string description: ID of the dataset. title: Dataset Id description: ID of the dataset. - name: split_id in: path required: true schema: type: string description: ID of the split. title: Split Id description: ID of the split. - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: type: array items: anyOf: - type: string - type: integer title: Sampleids responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/DeleteSamplesResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: deleteSamples /beta/file: post: tags: - File summary: Initiate file upload description: Initiate file upload. operationId: init_upload security: - token: [] parameters: - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/FileInitUploadRequest' responses: '200': description: OK - File is already uploaded. content: application/json: schema: $ref: '#/components/schemas/FileInitUploadResponse' '202': description: Accepted - Return upload URL. content: application/json: schema: $ref: '#/components/schemas/FileInitUploadResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: initUpload /beta/file/{file_id}: patch: tags: - File summary: Complete file upload description: Complete file upload. operationId: complete_upload security: - token: [] parameters: - name: file_id in: path required: true schema: type: string description: File ID title: File Id description: File ID - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: completeUpload get: tags: - File summary: Get file info description: Get file info. operationId: get_info security: - token: [] parameters: - name: file_id in: path required: true schema: type: string description: File ID title: File Id description: File ID - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/FileInfo' examples: Example: value: id: file-id name: file-name size: 1024 digest: file-digest project_id: project-id creator_id: creator-id created_at: 1705244400 expires_at: 1765724399 status: UPLOADED '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: getInfo /beta/file/{file_id}/download_url: get: tags: - File summary: Get file download URL description: Get file download URL. operationId: get_download_url security: - token: [] parameters: - name: file_id in: path required: true schema: type: string description: File ID title: File Id description: File ID - name: X-Friendli-Team in: header required: false schema: anyOf: - type: string - type: 'null' description: ID of team to run requests as (optional parameter). title: X-Friendli-Team description: ID of team to run requests as (optional parameter). responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/FileGetDownloadUrlResponse' examples: Example: value: id: file-id s3_uri: s3-uri download_url: download-url '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' x-speakeasy-name-override: getDownloadUrl components: schemas: AcceleratorRequirement: properties: type: type: string title: Type description: Specifies the instance type. count: type: integer title: Count description: Specifies the number of instances. type: object required: - type - count title: AcceleratorRequirement description: Accelerator requirement. AccountFileStatus: type: string enum: - UPLOADING - UPLOADED - FAILED - DELETED title: AccountFileStatus description: AccountFile status. AddSamplesResponse: properties: samples: items: prefixItems: - type: string - type: string type: array maxItems: 2 minItems: 2 type: array title: Samples description: List of sample IDs that were added to the dataset. type: object required: - samples title: AddSamplesResponse description: Add samples response. AssistantMessage: properties: role: type: string const: assistant title: Role description: The role of the messages author. content: anyOf: - type: string - type: 'null' title: Content description: >- The content of assistant message. Required unless `tool_calls` is specified. reasoning_content: anyOf: - type: string - type: 'null' title: Reasoning Content description: The intermediate reasoning content of assistant message. reasoning: anyOf: - type: string - type: 'null' title: Reasoning description: >- The intermediate reasoning content of assistant message. This field is a compatible option for the 'reasoning_content' field. name: anyOf: - type: string - type: 'null' title: Name description: >- The name for the participant to distinguish between participants with the same role. tool_calls: anyOf: - items: $ref: '#/components/schemas/AssistantMessageToolCall' type: array - type: 'null' title: Tool Calls type: object required: - role title: AssistantMessage AssistantMessageToolCall: properties: id: type: string title: Id description: The ID of tool call. type: type: string const: function title: Type description: The type of tool call. function: $ref: '#/components/schemas/AssistantMessageToolCallFunction' description: The function specification type: object required: - id - type - function title: AssistantMessageToolCall AssistantMessageToolCallFunction: properties: name: type: string title: Name description: The name of function arguments: type: string title: Arguments description: >- The arguments of function in JSON schema format to call the function. type: object required: - name - arguments title: AssistantMessageToolCallFunction AudioContent: properties: type: type: string const: audio_url title: Type description: The type of the message content. audio_url: $ref: '#/components/schemas/AudioData' description: The audio URL data. type: object required: - type - audio_url title: AudioContent AudioData: properties: url: type: string title: Url description: The URL of the audio. type: object required: - url title: AudioData AudioTranscriptionInputTokenDetails: properties: audio_tokens: type: integer title: Audio Tokens description: Number of audio tokens billed for this request. text_tokens: type: integer title: Text Tokens description: Number of text tokens billed for this request. type: object required: - audio_tokens - text_tokens title: AudioTranscriptionInputTokenDetails AudioTranscriptionResult: properties: text: type: string title: Text description: The transcribed text. usage: $ref: '#/components/schemas/AudioTranscriptionUsage' type: object required: - text - usage title: AudioTranscriptionResult AudioTranscriptionUsage: properties: type: type: string const: tokens title: Type description: The type of the usage object. Always `tokens` for this variant. input_tokens: type: integer title: Input Tokens description: Number of input tokens billed for this request. output_tokens: type: integer title: Output Tokens description: Number of output tokens generated. total_tokens: type: integer title: Total Tokens description: Total number of tokens used (input + output). input_audio_length_ms: type: integer title: Input Audio Length Ms description: The length of the input audio in milliseconds. processed_audio_length_ms: type: integer title: Processed Audio Length Ms description: The length of the processed audio in milliseconds. input_token_details: anyOf: - $ref: '#/components/schemas/AudioTranscriptionInputTokenDetails' - type: 'null' description: Details about the input tokens billed for this request. type: object required: - type - input_tokens - output_tokens - total_tokens - input_audio_length_ms - processed_audio_length_ms title: AudioTranscriptionUsage AutoscalingPolicy: properties: minReplica: type: integer minimum: 0 title: Minimum Replica description: >- Setting `minReplica` to 0 allows the endpoint to sleep when idle, reducing costs. The minimum value is 0. default: 0 maxReplica: type: integer maximum: 10 title: Maximum Replica description: >- The maximum replicas that the endpoint can scale up to. The maximum value is 10. default: 1 cooldownPeriod: type: integer title: Cooldown Period description: >- Determines how long the endpoint waits before scaling down after the last request. default: 300 type: object title: AutoscalingPolicy description: Autoscaling policy. B64ImageItem: properties: seed: type: integer title: Seed description: The seed used during image generation. response_format: type: string enum: - url - raw - png - jpeg - jpg title: Response Format description: >- The format of the generated image. One of `url(default)`, `raw`, `png`, `jpeg`, and `jpg`. b64_json: type: string format: binary title: B64 Json description: The base64-encoded JSON of the generated image. type: object required: - seed - response_format - b64_json title: B64ImageItem BaseClassificationData: properties: index: type: integer title: Index description: The index of the input in the list of inputs. examples: - 0 label: type: string title: Label description: The predicted label for the input text. examples: - Positive num_classes: type: integer title: Num Classes description: The number of possible labels the model can predict. examples: - 2 probs: items: type: number type: array title: Probs description: A list of logits for each possible label. examples: - - 0.1 - 0.9 type: object required: - index - label - num_classes - probs title: BaseClassificationData Body_upload_raw_samples: properties: file: type: string format: binary title: File description: File to update samples. type: object required: - file title: Body_upload_raw_samples ChatChoice: properties: index: type: integer title: Index description: The index of the choice in the list of generated choices. examples: - 0 logprobs: anyOf: - $ref: '#/components/schemas/ChatLogprobs' - type: 'null' message: $ref: '#/components/schemas/ChatChoiceMessage' finish_reason: type: string enum: - stop - length - tool_calls title: Finish Reason description: >- Termination condition of the generation. `stop` means the API returned the full chat completions generated by the model without running into any limits. `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length. `tool_calls` means the API has generated tool calls. type: object required: - index - message - finish_reason title: ChatChoice ChatChoiceMessage: properties: content: anyOf: - type: string - type: 'null' title: Content description: The contents of the assistant message. role: type: string title: Role description: Role of the generated message author, in this case `assistant`. tool_calls: anyOf: - items: $ref: '#/components/schemas/ToolCallResult' type: array - type: 'null' title: Tool Calls type: object required: - role title: ChatChoiceMessage ChatCompleteBodyToolChoice: properties: type: type: string const: function title: Type description: The type of the tool. Currently, only `function` is supported. function: $ref: '#/components/schemas/ChatCompleteBodyToolChoiceFunction' type: object required: - type - function title: ChatCompleteBodyToolChoice ChatCompleteBodyToolChoiceFunction: properties: name: type: string title: Name description: >- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. type: object required: - name title: ChatCompleteBodyToolChoiceFunction ChatLogprobs: properties: content: anyOf: - items: $ref: '#/components/schemas/ChatLogprobsContent' type: array - type: 'null' title: Content description: A list of message content tokens with log probability information. type: object title: ChatLogprobs ChatLogprobsContent: properties: token: type: string title: Token description: The token. logprob: type: number title: Logprob description: The log probability of this token. bytes: anyOf: - items: type: integer type: array - type: 'null' title: Bytes description: >- A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token. top_logprobs: items: $ref: '#/components/schemas/ChatLogprobsContentTopLogprob' type: array title: Top Logprobs description: >- List of the most likely tokens and their log probability, at this token position. type: object required: - token - logprob - top_logprobs title: ChatLogprobsContent ChatLogprobsContentTopLogprob: properties: token: type: string title: Token description: The token. logprob: type: number title: Logprob description: The log probability of this token. bytes: anyOf: - items: type: integer type: array - type: 'null' title: Bytes description: >- A list of integers representing the UTF-8 bytes representation of the token. Useful in instances where characters are represented by multiple tokens and their byte representations must be combined to generate the correct text representation. Can be `null` if there is no bytes representation for the token. type: object required: - token - logprob title: ChatLogprobsContentTopLogprob ChatResult: properties: id: type: string title: Id description: A unique ID of the chat completion. choices: items: $ref: '#/components/schemas/ChatChoice' type: array title: Choices usage: $ref: '#/components/schemas/ChatUsage' object: type: string const: chat.completion title: Object description: The object type, which is always set to `chat.completion`. created: type: integer title: Created description: The Unix timestamp (in seconds) for when the generation completed. model: anyOf: - type: string - type: 'null' title: Model description: >- The model to generate the completion. For dedicated endpoints, it returns the endpoint id. type: object required: - id - choices - usage - object - created title: ChatResult ChatUsage: properties: prompt_tokens: type: integer title: Prompt Tokens description: Number of tokens in the prompt. examples: - 5 completion_tokens: type: integer title: Completion Tokens description: Number of tokens in the generated chat completions. examples: - 7 total_tokens: type: integer title: Total Tokens description: >- Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`). examples: - 12 prompt_tokens_details: anyOf: - $ref: '#/components/schemas/PromptTokensDetails' - type: 'null' description: Breakdown of tokens used in the prompt. type: object required: - prompt_tokens - completion_tokens - total_tokens title: ChatUsage CompletionsBodyWithPrompt: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. prompt: anyOf: - type: string - items: type: string type: array title: Prompt description: >- The prompt (i.e., input text) to generate completions for. Either `prompt` or `tokens` field is required. examples: - Say this is a test! type: object required: - prompt title: CompletionsBodyWithPrompt CompletionsBodyWithTokens: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tokens: items: type: integer type: array title: Tokens description: >- The tokenized prompt (i.e., input tokens). Either `prompt` or `tokens` field is required. type: object required: - tokens title: CompletionsBodyWithTokens CompletionsChoice: properties: index: type: integer title: Index description: The index of the choice in the list of generated choices. examples: - 0 text: type: string title: Text description: Generated text output. examples: - This is indeed a test logprobs: anyOf: - $ref: '#/components/schemas/CompletionsLogprobs' - type: 'null' seed: type: integer title: Seed description: Random seed used for the generation. examples: - 42 finish_reason: type: string enum: - stop - length title: Finish Reason description: >- Termination condition of the generation. `stop` means the API returned the full completions generated by the model without running into any limits. `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length. tokens: items: type: integer type: array title: Tokens description: Generated output tokens. examples: - - 128000 - 2028 - 374 - 13118 - 264 - 1296 type: object required: - index - text - seed - finish_reason - tokens title: CompletionsChoice CompletionsDedicatedBodyWithPrompt: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. prompt: anyOf: - type: string - items: type: string type: array title: Prompt description: >- The prompt (i.e., input text) to generate completions for. Either `prompt` or `tokens` field is required. examples: - Say this is a test! type: object required: - model - prompt title: CompletionsDedicatedBodyWithPrompt CompletionsDedicatedBodyWithTokens: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tokens: items: type: integer type: array title: Tokens description: >- The tokenized prompt (i.e., input tokens). Either `prompt` or `tokens` field is required. type: object required: - model - tokens title: CompletionsDedicatedBodyWithTokens CompletionsLogprobs: properties: text_offset: items: type: integer type: array title: Text Offset description: >- The starting character position of each token in the generated text, useful for mapping tokens back to their exact location for detailed analysis. token_logprobs: items: type: number type: array title: Token Logprobs description: >- The log probabilities of each generated token, indicating the model's confidence in selecting each token. tokens: items: type: string type: array title: Tokens description: >- A list of individual tokens generated in the completion, representing segments of text such as words or pieces of words. top_logprobs: items: additionalProperties: true type: object type: array title: Top Logprobs description: >- A list of dictionaries, where each dictionary represents the top alternative tokens considered by the model at a specific position in the generated text, along with their log probabilities. The number of items in each dictionary matches the value of `logprobs`. type: object required: - text_offset - token_logprobs - tokens - top_logprobs title: CompletionsLogprobs CompletionsResult: properties: id: type: string title: Id description: A unique ID of the completion. object: type: string const: text_completion title: Object description: The object type, which is always set to `text_completion`. usage: $ref: '#/components/schemas/TextUsage' choices: items: $ref: '#/components/schemas/CompletionsChoice' type: array title: Choices model: anyOf: - type: string - type: 'null' title: Model description: >- The model to generate the completion. For dedicated endpoints, it returns the endpoint id. type: object required: - id - object - usage - choices title: CompletionsResult CompletionsServerlessBodyWithPrompt: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. prompt: anyOf: - type: string - items: type: string type: array title: Prompt description: >- The prompt (i.e., input text) to generate completions for. Either `prompt` or `tokens` field is required. examples: - Say this is a test! type: object required: - model - prompt title: CompletionsServerlessBodyWithPrompt CompletionsServerlessBodyWithTokens: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tokens: items: type: integer type: array title: Tokens description: >- The tokenized prompt (i.e., input tokens). Either `prompt` or `tokens` field is required. type: object required: - model - tokens title: CompletionsServerlessBodyWithTokens CompletionsStreamBodyWithPrompt: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. prompt: anyOf: - type: string - items: type: string type: array title: Prompt description: >- The prompt (i.e., input text) to generate completions for. Either `prompt` or `tokens` field is required. examples: - Say this is a test! type: object required: - prompt title: CompletionsStreamBodyWithPrompt CompletionsStreamBodyWithTokens: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tokens: items: type: integer type: array title: Tokens description: >- The tokenized prompt (i.e., input tokens). Either `prompt` or `tokens` field is required. type: object required: - tokens title: CompletionsStreamBodyWithTokens CompletionsStreamDedicatedBodyWithPrompt: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. prompt: anyOf: - type: string - items: type: string type: array title: Prompt description: >- The prompt (i.e., input text) to generate completions for. Either `prompt` or `tokens` field is required. examples: - Say this is a test! type: object required: - model - prompt title: CompletionsStreamDedicatedBodyWithPrompt CompletionsStreamDedicatedBodyWithTokens: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tokens: items: type: integer type: array title: Tokens description: >- The tokenized prompt (i.e., input tokens). Either `prompt` or `tokens` field is required. type: object required: - model - tokens title: CompletionsStreamDedicatedBodyWithTokens CompletionsStreamServerlessBodyWithPrompt: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. prompt: anyOf: - type: string - items: type: string type: array title: Prompt description: >- The prompt (i.e., input text) to generate completions for. Either `prompt` or `tokens` field is required. examples: - Say this is a test! type: object required: - model - prompt title: CompletionsStreamServerlessBodyWithPrompt CompletionsStreamServerlessBodyWithTokens: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct bad_word_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Bad Word Tokens description: >- Same as the above `bad_words` field, but receives token sequences instead of text phrases. This is similar to Hugging Face's [`bad_word_ids`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.bad_words_ids) argument. bad_words: anyOf: - items: type: string type: array - type: 'null' title: Bad Words description: > Text phrases that should not be generated. For a bad word phrase that contains N tokens, if the first N-1 tokens appears at the last of the generated result, the logit for the last token of the phrase is set to -inf. Before checking whether a bard word is included in the result, the word is converted into tokens. We recommend using `bad_word_tokens` because it is clearer. For example, after tokenization, phrases "clear" and " clear" can result in different token sequences due to the prepended space character. Defaults to empty list. embedding_to_replace: anyOf: - items: type: number type: array - type: 'null' title: Embedding To Replace description: >- A list of flattened embedding vectors used for replacing the tokens at the specified indices provided via `token_index_to_replace`. encoder_no_repeat_ngram: anyOf: - type: integer - type: 'null' title: Encoder No Repeat Ngram description: >- If this exceeds 1, every ngram of that size occurring in the input token sequence cannot appear in the generated result. 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Only allowed for encoder-decoder models. Defaults to 1. This is similar to Hugging Face's [`encoder_no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_no_repeat_ngram_size) argument. encoder_repetition_penalty: anyOf: - type: number - type: 'null' title: Encoder Repetition Penalty description: >- Penalizes tokens that have already appeared in the input tokens. Should be positive value. 1.0 means no penalty. Only allowed for encoder-decoder models. See [Keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`encoder_repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.encoder_repetition_penalty) argument. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. forced_output_tokens: anyOf: - items: type: integer type: array - type: 'null' title: Forced Output Tokens description: >- A token sequence that is enforced as a generation output. This option can be used when evaluating the model for the datasets with multi-choice problems (e.g., [HellaSwag](https://huggingface.co/datasets/hellaswag), [MMLU](https://huggingface.co/datasets/cais/mmlu)). Use this option with `logprobs` to get logprobs for the evaluation. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logprobs: anyOf: - type: integer - type: 'null' title: Logprobs description: >- Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 max_total_tokens: anyOf: - type: integer - type: 'null' title: Max Total Tokens description: >- The maximum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `max_tokens` and `max_total_tokens` is allowed. Default value is the model's maximum length. This is similar to Hugging Face's [`max_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_length) argument. min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `response_format` is specified.** min_total_tokens: anyOf: - type: integer - type: 'null' title: Min Total Tokens description: >- The minimum number of tokens including both the generated result and the input tokens. Only allowed for decoder-only models. Only one argument between `min_tokens` and `min_total_tokens` is allowed. This is similar to Hugging Face's [`min_length`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.min_length) argument. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. no_repeat_ngram: anyOf: - type: integer - type: 'null' title: No Repeat Ngram description: >- If this exceeds 1, every ngram of that size can only occur once among the generated result (plus the input tokens for decoder-only models). 1 means that this mechanism is disabled (i.e., you cannot prevent 1-gram from being generated repeatedly). Defaults to 1. This is similar to Hugging Face's [`no_repeat_ngram_size`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.no_repeat_ngram_size) argument. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, the API generate the seed randomly, use it for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stop_tokens: anyOf: - items: $ref: '#/components/schemas/TokenSequence' type: array - type: 'null' title: Stop Tokens description: >- Stop generating further tokens when generated token corresponds to any of the tokens in the sequence. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. token_index_to_replace: anyOf: - items: type: integer type: array - type: 'null' title: Token Index To Replace description: >- A list of token indices where to replace the embeddings of input tokens provided via either `tokens` or `prompt`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. examples: - 1 top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tokens: items: type: integer type: array title: Tokens description: >- The tokenized prompt (i.e., input tokens). Either `prompt` or `tokens` field is required. type: object required: - model - tokens title: CompletionsStreamServerlessBodyWithTokens ContainerAudioTranscriptionBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) file: type: string format: binary title: File description: >- The audio file object (not file name) to transcribe, in one of these formats: mp3, wav, flac, ogg, and many other standard audio formats. chunking_strategy: anyOf: - type: string const: auto - $ref: '#/components/schemas/ServerVadChunkingStrategy' - type: 'null' title: Chunking Strategy description: >- Controls how the audio is cut into chunks. When set to `"auto"`, the server first normalizes loudness and then uses voice activity detection (VAD) to choose boundaries. `server_vad` object can be provided to tweak VAD detection parameters manually. If unset, the audio is transcribed as a single block. language: anyOf: - type: string - type: 'null' title: Language description: >- The language of the input audio. Supplying the input language in [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format will improve accuracy and latency. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the transcription result. When set to `true`, the transcription result will be streamed as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. type: object required: - file title: ContainerAudioTranscriptionBody example: file: '@/path/to/file/audio.mp3' ContainerAudioTranscriptionStreamBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) file: type: string format: binary title: File description: >- The audio file object (not file name) to transcribe, in one of these formats: mp3, wav, flac, ogg, and many other standard audio formats. chunking_strategy: anyOf: - type: string const: auto - $ref: '#/components/schemas/ServerVadChunkingStrategy' - type: 'null' title: Chunking Strategy description: >- Controls how the audio is cut into chunks. When set to `"auto"`, the server first normalizes loudness and then uses voice activity detection (VAD) to choose boundaries. `server_vad` object can be provided to tweak VAD detection parameters manually. If unset, the audio is transcribed as a single block. language: anyOf: - type: string - type: 'null' title: Language description: >- The language of the input audio. Supplying the input language in [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format will improve accuracy and latency. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the transcription result. When set to `true`, the transcription result will be streamed as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. type: object required: - file title: ContainerAudioTranscriptionStreamBody example: file: '@/path/to/file/audio.mp3' ContainerAudioTranscriptionStreamSuccess: $ref: '#/components/schemas/StreamedAudioTranscriptionResult' title: ContainerAudioTranscriptionStreamSuccess ContainerAudioTranscriptionSuccess: $ref: '#/components/schemas/AudioTranscriptionResult' title: ContainerAudioTranscriptionSuccess ContainerChatCompleteSuccess: $ref: '#/components/schemas/ChatResult' title: ContainerChatCompleteSuccess ContainerChatCompletionBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `tools` or `response_format` is specified.** response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' description: >- The enforced format of the model's output. Note that the content of the output message may be truncated if it exceeds the `max_tokens`. You can check this by verifying that the `finish_reason` of the output message is `length`. For more detailed information, please refer [here](https://friendli.ai/docs/guides/structured-outputs). ***Important*** You must explicitly instruct the model to produce the desired output format using a system prompt or user message (e.g., `You are an API generating a valid JSON as output.`). Otherwise, the model may result in an unending stream of whitespace or other characters. **This field is unsupported when `tools` is specified.** **When `response_format` is specified, `min_tokens` field is unsupported.** type: object required: - messages title: ContainerChatCompletionBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user ContainerChatCompletionStreamBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `tools` or `response_format` is specified.** response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' description: >- The enforced format of the model's output. Note that the content of the output message may be truncated if it exceeds the `max_tokens`. You can check this by verifying that the `finish_reason` of the output message is `length`. For more detailed information, please refer [here](https://friendli.ai/docs/guides/structured-outputs). ***Important*** You must explicitly instruct the model to produce the desired output format using a system prompt or user message (e.g., `You are an API generating a valid JSON as output.`). Otherwise, the model may result in an unending stream of whitespace or other characters. **This field is unsupported when `tools` is specified.** **When `response_format` is specified, `min_tokens` field is unsupported.** type: object required: - messages title: ContainerChatCompletionStreamBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user ContainerChatCompletionStreamSuccess: $ref: '#/components/schemas/StreamedChatResult' title: ContainerChatCompletionStreamSuccess ContainerCompletionsBody: anyOf: - $ref: '#/components/schemas/CompletionsBodyWithPrompt' - $ref: '#/components/schemas/CompletionsBodyWithTokens' title: ContainerCompletionsBody ContainerCompletionsStreamBody: anyOf: - $ref: '#/components/schemas/CompletionsStreamBodyWithPrompt' - $ref: '#/components/schemas/CompletionsStreamBodyWithTokens' title: ContainerCompletionsStreamBody ContainerCompletionsStreamSuccess: $ref: '#/components/schemas/StreamedCompletionsResult' title: ContainerCompletionsStreamSuccess ContainerCompletionsSuccess: $ref: '#/components/schemas/CompletionsResult' title: ContainerCompletionsSuccess ContainerDetokenizationBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) tokens: items: type: integer type: array title: Tokens description: A token sequence to detokenize. examples: - - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 type: object required: - tokens title: ContainerDetokenizationBody example: tokens: - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 ContainerDetokenizationSuccess: properties: text: type: string title: Text description: Detokenized text output. type: object required: - text title: ContainerDetokenizationSuccess ContainerImageEditBody: properties: image: anyOf: - $ref: '#/components/schemas/ImageInput' - items: $ref: '#/components/schemas/ImageInput' type: array title: Image description: The image(s) to edit. Must be in a supported image format. prompt: type: string title: Prompt description: A text description of the desired image. model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) num_inference_steps: type: integer title: Num Inference Steps description: >- The number of inference steps to use during image generation. Defaults to 20. Supported range: [1, 50]. default: 20 guidance_scale: type: number title: Guidance Scale description: >- Adjusts the alignment of the generated image with the input prompt. Higher values (e.g., 8-10) make the output more faithful to the prompt, while lower values (e.g., 1-5) encourage more creative freedom. Defaults to 0. This parameter may be irrelevant for certain models, such as `FLUX.Schnell`. default: 0 seed: anyOf: - type: integer - type: 'null' title: Seed description: The seed to use for image generation. response_format: anyOf: - type: string enum: - raw - jpeg - type: 'null' title: Response Format description: >- The format in which the generated image will be returned. One of `raw` and `jpeg`. default: jpeg type: object required: - image - prompt title: ContainerImageEditBody example: guidance_scale: 0 image: '@/path/to/file/image.png' num_inference_steps: 20 prompt: Add a red sports car in the foreground. ContainerImageEditSuccess: $ref: '#/components/schemas/ImageGenerationContainerResult' title: ContainerImageEditSuccess ContainerImageGenerateSuccess: $ref: '#/components/schemas/ImageGenerationContainerResult' title: ContainerImageGenerateSuccess ContainerImageGenerationBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) prompt: type: string title: Prompt description: A text description of the desired image. num_inference_steps: type: integer title: Num Inference Steps description: >- The number of inference steps to use during image generation. Defaults to 20. Supported range: [1, 50]. default: 20 guidance_scale: type: number title: Guidance Scale description: >- Adjusts the alignment of the generated image with the input prompt. Higher values (e.g., 8-10) make the output more faithful to the prompt, while lower values (e.g., 1-5) encourage more creative freedom. Defaults to 0. This parameter may be irrelevant for certain models, such as `FLUX.Schnell`. default: 0 seed: anyOf: - type: integer - type: 'null' title: Seed description: The seed to use for image generation. response_format: anyOf: - type: string enum: - raw - jpeg - type: 'null' title: Response Format description: >- The format in which the generated image will be returned. One of `raw` and `jpeg`. default: jpeg control_images: anyOf: - items: $ref: '#/components/schemas/ImageInput' type: array - type: 'null' title: Control Images description: >- Optional input images used to condition or guide the generation process (e.g., for ControlNet or image editing models). This field is only applicable when using ControlNet or image editing models. controlnet_weights: anyOf: - items: type: number type: array - type: 'null' title: Controlnet Weights description: >- A list of weights that determine the influence of each ControlNet model in the generation process. Each value must be within [0, 1], where 0 disables the corresponding ControlNet and 1 applies it fully. When multiple ControlNet models are used, the list length must match the number of control images. If omitted, all ControlNet models default to full influence (1.0). This field is only applicable when using ControlNet models. type: object required: - prompt title: ContainerImageGenerationBody example: guidance_scale: 3.5 num_inference_steps: 10 prompt: >- An orange Lamborghini driving down a hill road at night with a beautiful ocean view in the background. ContainerMessagesBody: properties: messages: items: $ref: '#/components/schemas/MessagesInputMessage' type: array minItems: 1 title: Messages description: >- A list of conversation messages ordered from oldest to newest. Must contain at least one item. examples: - - content: Explain top_p in one sentence. role: user max_tokens: anyOf: - type: integer minimum: 1 - type: 'null' title: Max Tokens description: >- Maximum number of tokens to generate for the assistant response. Must be greater than 0 when provided. model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) system: anyOf: - anyOf: - type: string - items: $ref: '#/components/schemas/MessagesSystemTextBlock' type: array description: Payload format for the top-level `system` instruction. - type: 'null' title: System description: >- Optional top-level system instruction applied before conversation turns. Supports a plain string or an array of text blocks. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream output as server-sent events (`text/event-stream`). When false or omitted, returns a single JSON response. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Lower values make outputs more deterministic; higher values increase diversity. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Nucleus sampling parameter. The model samples from the smallest token set whose cumulative probability reaches `top_p`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: Limits sampling to the `k` most likely tokens at each decoding step. stop_sequences: anyOf: - items: type: string type: array - type: 'null' title: Stop Sequences description: >- Stop strings that terminate generation when matched in output. The matched value is returned in `stop_sequence` when applicable. tools: anyOf: - items: $ref: '#/components/schemas/MessagesToolDefinition' type: array - type: 'null' title: Tools description: >- Tool definitions available to the model. Use this to allow tool calls with structured arguments. tool_choice: anyOf: - $ref: '#/components/schemas/MessagesToolChoice' - type: 'null' description: >- Controls tool-calling behavior (`auto`, `any`, `tool`, `none`) and optional parallel-call behavior. thinking: anyOf: - $ref: '#/components/schemas/MessagesThinkingConfig' - type: 'null' description: >- Controls reasoning behavior with mode (`enabled`, `disabled`, `adaptive`). `enabled` requires `budget_tokens`; `disabled` and `adaptive` must not include it. output_config: anyOf: - $ref: '#/components/schemas/MessagesOutputConfig' - type: 'null' description: >- Output generation options including effort level and structured output format settings. cache_control: anyOf: - additionalProperties: true type: object - type: 'null' title: Cache Control description: >- Compatibility field accepted for request portability. Parsed but not used for generation. container: anyOf: - additionalProperties: true type: object - type: 'null' title: Container description: >- Compatibility field accepted for request portability. Parsed but not used for generation. context_manager: anyOf: - additionalProperties: true type: object - type: 'null' title: Context Manager description: >- Compatibility field accepted for request portability. Parsed but not used for generation. inference_geo: anyOf: - additionalProperties: true type: object - type: 'null' title: Inference Geo description: >- Compatibility field accepted for request portability. Parsed but not used for generation. metadata: anyOf: - additionalProperties: true type: object - type: 'null' title: Metadata description: >- Compatibility field accepted for request portability. Parsed but not used for generation. service_tier: anyOf: - type: string - additionalProperties: true type: object - type: 'null' title: Service Tier description: >- Compatibility field accepted for request portability. Parsed but not used for generation. additionalProperties: true type: object required: - messages title: ContainerMessagesBody example: max_tokens: 128 messages: - content: Hello, summarize what you can do in one sentence. role: user model: meta-llama/Llama-3.1-8B-Instruct ContainerMessagesSuccess: properties: id: type: string title: Id description: Unique identifier for this message response. type: type: string const: message title: Type description: Response object type (`message`). role: type: string const: assistant title: Role description: Role of the output message author (`assistant`). content: items: $ref: '#/components/schemas/MessagesResponseContentBlock' type: array title: Content description: Assistant output blocks in generation order. stop_reason: anyOf: - type: string enum: - end_turn - max_tokens - tool_use - stop_sequence - type: 'null' title: Stop Reason description: >- Why generation stopped (`end_turn`, `max_tokens`, `tool_use`, `stop_sequence`). stop_sequence: anyOf: - type: string - type: 'null' title: Stop Sequence description: >- Matched stop string when `stop_reason` is `stop_sequence`; otherwise null. usage: $ref: '#/components/schemas/MessagesUsage' description: Token usage details for this response. model: anyOf: - type: string - type: 'null' title: Model description: Model used to generate the response. additionalProperties: true type: object required: - id - type - role - content - usage title: ContainerMessagesSuccess ContainerTextClassificationBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) input: anyOf: - type: string - items: type: string type: array - type: 'null' title: Input description: >- Input text to classify, encoded as a string or array of strings. To classify multiple inputs in a single request, pass an array of strings. Either `input` or `tokens` field is required. examples: - I love programming. tokens: anyOf: - items: type: integer type: array - type: 'null' title: Tokens description: |- The tokenized prompt (i.e., input tokens). Either `input` or `tokens` field is required. examples: - 72 - 1563 - 2335 - 13 type: object title: ContainerTextClassificationBody example: input: I love programming. ContainerTextClassificationSuccess: $ref: '#/components/schemas/TextClassificationResult' title: ContainerTextClassificationSuccess ContainerTokenizationBody: properties: model: anyOf: - type: string - type: 'null' title: Model description: Routes the request to a specific adapter. examples: - (adapter-route) prompt: type: string title: Prompt description: Input text prompt to tokenize. examples: - What is generative AI? type: object required: - prompt title: ContainerTokenizationBody example: prompt: What is generative AI? ContainerTokenizationSuccess: properties: tokens: items: type: integer type: array title: Tokens description: A list of token IDs. type: object required: - tokens title: ContainerTokenizationSuccess CreateDatasetRequest: properties: name: type: string title: Name description: Name of the dataset. projectId: type: string title: Projectid description: ID of the project. modality: $ref: '#/components/schemas/DedicatedDatasetModality' description: Modality of the dataset. type: object required: - name - projectId - modality title: CreateDatasetRequest description: Create dataset request. DatasetInfo: properties: id: type: string title: Id description: ID of the dataset. name: type: string title: Name description: Name of the dataset. createdAt: type: integer title: Createdat description: Unix timestamp (in seconds) of when the dataset was created. updatedAt: type: integer title: Updatedat description: Unix timestamp (in seconds) of when the dataset was last modified. modality: $ref: '#/components/schemas/DedicatedDatasetModality' description: Modality of the dataset. type: object required: - id - name - createdAt - updatedAt - modality title: DatasetInfo description: Dataset info. DedicatedAudioTranscriptionBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) file: type: string format: binary title: File description: >- The audio file object (not file name) to transcribe, in one of these formats: mp3, wav, flac, ogg, and many other standard audio formats. chunking_strategy: anyOf: - type: string const: auto - $ref: '#/components/schemas/ServerVadChunkingStrategy' - type: 'null' title: Chunking Strategy description: >- Controls how the audio is cut into chunks. When set to `"auto"`, the server first normalizes loudness and then uses voice activity detection (VAD) to choose boundaries. `server_vad` object can be provided to tweak VAD detection parameters manually. If unset, the audio is transcribed as a single block. language: anyOf: - type: string - type: 'null' title: Language description: >- The language of the input audio. Supplying the input language in [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format will improve accuracy and latency. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the transcription result. When set to `true`, the transcription result will be streamed as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. type: object required: - model - file title: DedicatedAudioTranscriptionBody example: file: '@/path/to/file/audio.mp3' model: (endpoint-id) DedicatedAudioTranscriptionStreamBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) file: type: string format: binary title: File description: >- The audio file object (not file name) to transcribe, in one of these formats: mp3, wav, flac, ogg, and many other standard audio formats. chunking_strategy: anyOf: - type: string const: auto - $ref: '#/components/schemas/ServerVadChunkingStrategy' - type: 'null' title: Chunking Strategy description: >- Controls how the audio is cut into chunks. When set to `"auto"`, the server first normalizes loudness and then uses voice activity detection (VAD) to choose boundaries. `server_vad` object can be provided to tweak VAD detection parameters manually. If unset, the audio is transcribed as a single block. language: anyOf: - type: string - type: 'null' title: Language description: >- The language of the input audio. Supplying the input language in [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format will improve accuracy and latency. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the transcription result. When set to `true`, the transcription result will be streamed as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. type: object required: - model - file title: DedicatedAudioTranscriptionStreamBody example: file: '@/path/to/file/audio.mp3' model: (endpoint-id) DedicatedAudioTranscriptionStreamSuccess: $ref: '#/components/schemas/StreamedAudioTranscriptionResult' title: DedicatedAudioTranscriptionStreamSuccess DedicatedAudioTranscriptionSuccess: $ref: '#/components/schemas/AudioTranscriptionResult' title: DedicatedAudioTranscriptionSuccess DedicatedChatCompleteSuccess: $ref: '#/components/schemas/ChatResult' title: DedicatedChatCompleteSuccess DedicatedChatCompletionBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `tools` or `response_format` is specified.** response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' description: >- The enforced format of the model's output. Note that the content of the output message may be truncated if it exceeds the `max_tokens`. You can check this by verifying that the `finish_reason` of the output message is `length`. For more detailed information, please refer [here](https://friendli.ai/docs/guides/structured-outputs). ***Important*** You must explicitly instruct the model to produce the desired output format using a system prompt or user message (e.g., `You are an API generating a valid JSON as output.`). Otherwise, the model may result in an unending stream of whitespace or other characters. **This field is unsupported when `tools` is specified.** **When `response_format` is specified, `min_tokens` field is unsupported.** type: object required: - model - messages title: DedicatedChatCompletionBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user model: (endpoint-id) DedicatedChatCompletionStreamBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `tools` or `response_format` is specified.** response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' description: >- The enforced format of the model's output. Note that the content of the output message may be truncated if it exceeds the `max_tokens`. You can check this by verifying that the `finish_reason` of the output message is `length`. For more detailed information, please refer [here](https://friendli.ai/docs/guides/structured-outputs). ***Important*** You must explicitly instruct the model to produce the desired output format using a system prompt or user message (e.g., `You are an API generating a valid JSON as output.`). Otherwise, the model may result in an unending stream of whitespace or other characters. **This field is unsupported when `tools` is specified.** **When `response_format` is specified, `min_tokens` field is unsupported.** type: object required: - model - messages title: DedicatedChatCompletionStreamBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user model: (endpoint-id) DedicatedChatCompletionStreamSuccess: $ref: '#/components/schemas/StreamedChatResult' title: DedicatedChatCompletionStreamSuccess DedicatedChatRenderBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** type: object required: - model - messages title: DedicatedChatRenderBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user model: (endpoint-id) DedicatedChatRenderSuccess: properties: text: type: string title: Text description: The rendered text. type: object required: - text title: DedicatedChatRenderSuccess DedicatedCompletionsBody: anyOf: - $ref: '#/components/schemas/CompletionsDedicatedBodyWithPrompt' - $ref: '#/components/schemas/CompletionsDedicatedBodyWithTokens' title: DedicatedCompletionsBody DedicatedCompletionsStreamBody: anyOf: - $ref: '#/components/schemas/CompletionsStreamDedicatedBodyWithPrompt' - $ref: '#/components/schemas/CompletionsStreamDedicatedBodyWithTokens' title: DedicatedCompletionsStreamBody DedicatedCompletionsStreamSuccess: $ref: '#/components/schemas/StreamedCompletionsResult' title: DedicatedCompletionsStreamSuccess DedicatedCompletionsSuccess: $ref: '#/components/schemas/CompletionsResult' title: DedicatedCompletionsSuccess DedicatedDatasetModality: properties: input_modals: items: $ref: '#/components/schemas/DedicatedDatasetModalityType' type: array title: Input Modals default: [] output_modals: items: $ref: '#/components/schemas/DedicatedDatasetModalityType' type: array title: Output Modals default: [] type: object title: DedicatedDatasetModality description: Dataset modality. DedicatedDatasetModalityType: type: string enum: - VIDEO - IMAGE - AUDIO - TEXT title: DedicatedDatasetModalityType description: Modality type. DedicatedDetokenizationBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) tokens: items: type: integer type: array title: Tokens description: A token sequence to detokenize. examples: - - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 type: object required: - model - tokens title: DedicatedDetokenizationBody example: model: (endpoint-id) tokens: - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 DedicatedDetokenizationSuccess: properties: text: type: string title: Text description: Detokenized text output. type: object required: - text title: DedicatedDetokenizationSuccess DedicatedEmbeddingsBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) input: anyOf: - type: string - items: type: string type: array - type: 'null' title: Input description: >- Input text to embed, encoded as a string or array of tokens. To embed multiple inputs in a single request, pass an array of strings or array of token arrays. Either `input` or `tokens` field is required. examples: - The food was delicious and the waiter... tokens: anyOf: - items: type: integer type: array - type: 'null' title: Tokens description: |- The tokenized prompt (i.e., input tokens). Either `input` or `tokens` field is required. encoding_format: anyOf: - type: string enum: - float - base64 - type: 'null' title: Encoding Format description: >- The format to return the embeddings in. Can be either `float` or [`base64`](https://pypi.org/project/pybase64/). default: float type: object required: - model title: DedicatedEmbeddingsBody example: encoding_format: float input: The food was delicious and the waiter... model: (endpoint-id) DedicatedEmbeddingsSuccess: properties: id: type: string title: Id description: A unique ID of the embeddings. model: anyOf: - type: string - type: 'null' title: Model description: >- The model to generate the embeddings. For dedicated endpoints, it returns the endpoint id. object: type: string const: list title: Object description: The object type, which is always set to `list`. data: items: $ref: '#/components/schemas/EmbeddingObject' type: array title: Data description: A list of embedding objects. usage: $ref: '#/components/schemas/TextUsage' created: type: integer title: Created description: >- The Unix timestamp (in seconds) for when the embeddings were created. type: object required: - id - object - data - usage - created title: DedicatedEmbeddingsSuccess DedicatedEndpointCreateBody: properties: projectId: type: string title: Project ID description: The ID of the project that owns the endpoint. name: type: string title: Name description: The name of the endpoint. instanceOptionId: type: string title: Instance Option ID description: |- The ID of the instance option. Available options: - 1x NVIDIA A100 80GB: `ShbPuOs4tfGb` - 2x NVIDIA A100 80GB: `mrAHuYt7T40o` - 4x NVIDIA A100 80GB: `JkNob0NMdoF3` - 8x NVIDIA A100 80GB: `sYH4kHmAcA5P` - 1x NVIDIA H100: `TwD5AqnBSVN0` - 2x NVIDIA H100: `zfTutSiLn0Hq` - 4x NVIDIA H100: `lfkRz5G48REc` - 8x NVIDIA H100: `GUA4qYFmsYz8` - 1x NVIDIA H200: `LnK1wTaKc7WO` - 2x NVIDIA H200: `Tu6GjBnfHPe4` - 4x NVIDIA H200: `OhTzYtZuomzI` - 8x NVIDIA H200: `ahBzWtOuomsI` - 1x NVIDIA B200: `8GiQTLKfJNOr` - 2x NVIDIA B200: `brTZGIuYgVrs` - 4x NVIDIA B200: `AFoZMFXZnAdD` - 8x NVIDIA B200: `drbc6G9FxJWZ` advanced: $ref: '#/components/schemas/EndpointAdvancedConfig' title: Advanced description: The advanced configuration of the endpoint. simplescale: anyOf: - $ref: '#/components/schemas/EndpointSimplescaleConfig' - type: 'null' title: Simple Scale description: The simple scaling configuration of the endpoint. autoscalingPolicy: anyOf: - $ref: '#/components/schemas/AutoscalingPolicy' - type: 'null' title: Auto Scale Policy description: The auto scaling configuration of the endpoint. hfModelRepo: type: string title: HF Model Repo description: HF ID of the model. hfModelRepoRevision: anyOf: - type: string - type: 'null' title: HF Model Repo Revision description: HF commit hash of the model. initialVersionComment: anyOf: - type: string - type: 'null' title: Initial Version Comment description: The comment for the initial version. type: object required: - projectId - name - instanceOptionId - advanced - hfModelRepo title: DedicatedEndpointCreateBody description: Dedicated endpoint create request. DedicatedEndpointListResponse: properties: data: additionalProperties: $ref: '#/components/schemas/DedicatedEndpointStatus' type: object title: Data description: The response data containing endpoint statuses. default: {} nextCursor: anyOf: - type: string format: binary - type: 'null' title: Next Cursor description: The next cursor for pagination. type: object title: DedicatedEndpointListResponse description: Dedicated endpoint list response. DedicatedEndpointSpec: properties: name: type: string title: Name description: The name of the endpoint. gpuType: type: string title: GPU Type description: The type of GPU to use for the endpoint. numGpu: type: integer title: Number of GPUs description: The number of GPUs to use per replica. instanceId: anyOf: - type: string - type: 'null' title: Instance ID description: The ID of the instance. projectId: type: string title: Project ID description: The ID of the project that owns the endpoint. creatorId: type: string title: Creator ID description: The ID of the user who created the endpoint. teamId: type: string title: Team ID description: The ID of the team that owns the endpoint. autoscalingMin: type: integer title: Minimum Replicas description: The minimum number of replicas to maintain. autoscalingMax: type: integer title: Maximum Replicas description: The maximum number of replicas allowed. autoscalingCooldown: type: integer title: Autoscaling Cooldown description: The cooldown period in seconds between scaling operations. maxBatchSize: type: integer title: Maximum Batch Size description: The maximum batch size for inference requests. maxInputLength: anyOf: - type: integer - type: 'null' title: Maximum Input Length description: The maximum allowed input length. tokenizerSkipSpecialTokens: type: boolean title: Skip Special Tokens description: Whether to skip special tokens in tokenizer output. tokenizerAddSpecialTokens: type: boolean title: Add Special Tokens description: Whether to add special tokens in tokenizer input. currReplicaCnt: anyOf: - type: integer - type: 'null' title: Current Replica Count description: The current number of replicas. desiredReplicaCnt: anyOf: - type: integer - type: 'null' title: Desired Replica Count description: The desired number of replicas. updatedReplicaCnt: anyOf: - type: integer - type: 'null' title: Updated Replica Count description: The updated number of replicas. type: object required: - name - gpuType - numGpu - projectId - creatorId - teamId - autoscalingMin - autoscalingMax - autoscalingCooldown - maxBatchSize - tokenizerSkipSpecialTokens - tokenizerAddSpecialTokens title: DedicatedEndpointSpec description: Dedicated endpoint specification. DedicatedEndpointStatus: properties: status: $ref: '#/components/schemas/InferenceDeploymentStatus' title: Status description: The current status of the endpoint deployment. errorCode: anyOf: - $ref: '#/components/schemas/InferenceDeploymentErrorCode' - type: 'null' title: Error Code description: Error code if deployment failed. createdAt: type: string format: date-time title: Created At description: When the endpoint was created. updatedAt: anyOf: - type: string format: date-time - type: 'null' title: Updated At description: When the endpoint was last updated. phase: anyOf: - type: string enum: - REQUESTING_VIRTUAL_MACHINE - DOWNLOADING_MODEL - ENGINE_INITIALIZING - type: 'null' title: Phase description: The current phase of the endpoint. type: object required: - status - createdAt title: DedicatedEndpointStatus description: Dedicated endpoint status. DedicatedEndpointUpdateBody: properties: name: anyOf: - type: string - type: 'null' title: Name description: The name of the endpoint. advanced: anyOf: - $ref: '#/components/schemas/EndpointAdvancedConfig' - type: 'null' title: Advanced description: The advanced configuration of the endpoint. simplescale: anyOf: - $ref: '#/components/schemas/EndpointSimplescaleConfig' - type: 'null' title: Simple Scale description: The simple scaling configuration of the endpoint. autoscalingPolicy: anyOf: - $ref: '#/components/schemas/AutoscalingPolicy' - type: 'null' title: Auto Scale Policy description: The auto scaling configuration of the endpoint. hfModelRepo: anyOf: - type: string - type: 'null' title: HF Model Repo description: HF ID of the model. hfModelRepoRevision: anyOf: - type: string - type: 'null' title: HF Model Repo Revision description: HF commit hash of the model. newVersionComment: anyOf: - type: string - type: 'null' title: New Version Comment description: Comment for the new version. instanceOptionId: anyOf: - type: string - type: 'null' title: Instance Option ID description: |- The ID of the instance option. Available options: - 1x NVIDIA A100 80GB: `ShbPuOs4tfGb` - 2x NVIDIA A100 80GB: `mrAHuYt7T40o` - 4x NVIDIA A100 80GB: `JkNob0NMdoF3` - 8x NVIDIA A100 80GB: `sYH4kHmAcA5P` - 1x NVIDIA H100: `TwD5AqnBSVN0` - 2x NVIDIA H100: `zfTutSiLn0Hq` - 4x NVIDIA H100: `lfkRz5G48REc` - 8x NVIDIA H100: `GUA4qYFmsYz8` - 1x NVIDIA H200: `LnK1wTaKc7WO` - 2x NVIDIA H200: `Tu6GjBnfHPe4` - 4x NVIDIA H200: `OhTzYtZuomzI` - 8x NVIDIA H200: `ahBzWtOuomsI` - 1x NVIDIA B200: `8GiQTLKfJNOr` - 2x NVIDIA B200: `brTZGIuYgVrs` - 4x NVIDIA B200: `AFoZMFXZnAdD` - 8x NVIDIA B200: `drbc6G9FxJWZ` type: object title: DedicatedEndpointUpdateBody description: Dedicated endpoint update request. DedicatedEndpointVersionHistoryResponse: properties: data: additionalProperties: $ref: '#/components/schemas/DedicatedEndpointSpec' type: object title: Data description: The response data containing endpoint versions. default: {} nextCursor: anyOf: - type: string format: binary - type: 'null' title: Next Cursor description: The next cursor for pagination. type: object title: DedicatedEndpointVersionHistoryResponse description: Dedicated endpoint version history response. DedicatedEndpointWandbArtifactCreateBody: properties: wandbArtifactVersionName: type: string title: Wandb Artifact Version Name description: >- The specific model artifact version from Weights & Biases. The referred artifact will be used to create a new endpoint in Friendli Dedicated Endpoints or rollout an existing one. accelerator: anyOf: - $ref: '#/components/schemas/AcceleratorRequirement' - type: 'null' title: Accelerator description: Specifies the instance type for the endpoint. name: anyOf: - type: string - type: 'null' title: Name description: >- Specifies the name of your endpoint. If not provided, a name will be automatically generated for you. projectId: anyOf: - type: string - type: 'null' title: Project Id description: >- Specifies where endpoint will be created in your Friendli Dedicated Endpoints. If not provided, a new project will be created within your default team. idempotencyKey: anyOf: - type: string - type: 'null' title: Idempotency Key description: >- Used by Friendli Dedicated Endpoints to track which webhook automation triggered an endpoint rollout. If the `idempotencyKey` is provided, the API will check if the endpoint already exists, and rollout the existing endpoint if it does. In such cases, the `projectId` must be provided. Any unique value can be used. autoscalingPolicy: anyOf: - $ref: '#/components/schemas/AutoscalingPolicy' - type: 'null' title: Autoscaling Policy description: Defines autoscaling settings for the endpoint. type: object required: - wandbArtifactVersionName title: DedicatedEndpointWandbArtifactCreateBody description: Endpoint create request. example: wandbArtifactVersionName: org/registry/name:v0 DedicatedEndpointWandbArtifactCreateResponse: properties: endpointId: type: string title: Endpoint Id description: The ID of the created endpoint. endpointName: type: string title: Endpoint Name description: The name of the created endpoint. projectId: type: string title: Project Id description: The ID of the project where the endpoint was created. projectName: type: string title: Project Name description: The name of the project where the endpoint was created. teamId: type: string title: Team Id description: The ID of the team where the endpoint was created. teamName: anyOf: - type: string - type: 'null' title: Team Name description: The name of the team where the endpoint was created. type: object required: - endpointId - endpointName - projectId - projectName - teamId - teamName title: DedicatedEndpointWandbArtifactCreateResponse description: Endpoint create response. DedicatedImageEditBody: properties: image: anyOf: - $ref: '#/components/schemas/ImageInput' - items: $ref: '#/components/schemas/ImageInput' type: array title: Image description: The image(s) to edit. Must be in a supported image format. prompt: type: string title: Prompt description: A text description of the desired image(s). model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) num_inference_steps: type: integer title: Num Inference Steps description: >- The number of inference steps to use during image generation. Defaults to 20. Supported range: [1, 50]. default: 20 guidance_scale: anyOf: - type: number - type: 'null' title: Guidance Scale description: >- Adjusts the alignment of the generated image with the input prompt. Higher values (e.g., 8-10) make the output more faithful to the prompt, while lower values (e.g., 1-5) encourage more creative freedom. This parameter may be irrelevant for certain models, such as `FLUX.Schnell`. seed: anyOf: - type: integer - type: 'null' title: Seed description: The seed to use for image generation. response_format: anyOf: - type: string enum: - url - raw - png - jpeg - jpg - type: 'null' title: Response Format description: >- The format in which the generated image(s) will be returned. One of `url(default)`, `raw`, `png`, `jpeg`, and `jpg`. default: url type: object required: - image - prompt - model title: DedicatedImageEditBody example: guidance_scale: 0 image: '@/path/to/file/image.png' model: (endpoint-id) num_inference_steps: 20 prompt: Add a red sports car in the foreground. DedicatedImageEditSuccess: $ref: '#/components/schemas/ImageGenerationDedicatedResult' title: DedicatedImageEditSuccess DedicatedImageGenerateSuccess: $ref: '#/components/schemas/ImageGenerationDedicatedResult' title: DedicatedImageGenerateSuccess DedicatedImageGenerationBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) prompt: type: string title: Prompt description: A text description of the desired image(s). num_inference_steps: type: integer title: Num Inference Steps description: >- The number of inference steps to use during image generation. Defaults to 20. Supported range: [1, 50]. default: 20 guidance_scale: anyOf: - type: number - type: 'null' title: Guidance Scale description: >- Adjusts the alignment of the generated image with the input prompt. Higher values (e.g., 8-10) make the output more faithful to the prompt, while lower values (e.g., 1-5) encourage more creative freedom. This parameter may be irrelevant for certain models, such as `FLUX.Schnell`. seed: anyOf: - type: integer - type: 'null' title: Seed description: The seed to use for image generation. response_format: anyOf: - type: string enum: - url - raw - png - jpeg - jpg - type: 'null' title: Response Format description: >- The format in which the generated image(s) will be returned. One of `url(default)`, `raw`, `png`, `jpeg`, and `jpg`. default: url control_images: anyOf: - items: $ref: '#/components/schemas/ImageInput' type: array - type: 'null' title: Control Images description: >- Optional input images used to condition or guide the generation process (e.g., for ControlNet or image editing models). This field is only applicable when using ControlNet or image editing models. controlnet_weights: anyOf: - items: type: number type: array - type: 'null' title: Controlnet Weights description: >- A list of weights that determine the influence of each ControlNet model in the generation process. Each value must be within [0, 1], where 0 disables the corresponding ControlNet and 1 applies it fully. When multiple ControlNet models are used, the list length must match the number of control images. If omitted, all ControlNet models default to full influence (1.0). This field is only applicable when using ControlNet models. type: object required: - model - prompt title: DedicatedImageGenerationBody example: model: (endpoint-id) num_inference_steps: 10 prompt: >- An orange Lamborghini driving down a hill road at night with a beautiful ocean view in the background. DedicatedMessagesBody: properties: messages: items: $ref: '#/components/schemas/MessagesInputMessage' type: array minItems: 1 title: Messages description: >- A list of conversation messages ordered from oldest to newest. Must contain at least one item. examples: - - content: Explain top_p in one sentence. role: user max_tokens: anyOf: - type: integer minimum: 1 - type: 'null' title: Max Tokens description: >- Maximum number of tokens to generate for the assistant response. Must be greater than 0 when provided. model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) system: anyOf: - anyOf: - type: string - items: $ref: '#/components/schemas/MessagesSystemTextBlock' type: array description: Payload format for the top-level `system` instruction. - type: 'null' title: System description: >- Optional top-level system instruction applied before conversation turns. Supports a plain string or an array of text blocks. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream output as server-sent events (`text/event-stream`). When false or omitted, returns a single JSON response. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Lower values make outputs more deterministic; higher values increase diversity. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Nucleus sampling parameter. The model samples from the smallest token set whose cumulative probability reaches `top_p`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: Limits sampling to the `k` most likely tokens at each decoding step. stop_sequences: anyOf: - items: type: string type: array - type: 'null' title: Stop Sequences description: >- Stop strings that terminate generation when matched in output. The matched value is returned in `stop_sequence` when applicable. tools: anyOf: - items: $ref: '#/components/schemas/MessagesToolDefinition' type: array - type: 'null' title: Tools description: >- Tool definitions available to the model. Use this to allow tool calls with structured arguments. tool_choice: anyOf: - $ref: '#/components/schemas/MessagesToolChoice' - type: 'null' description: >- Controls tool-calling behavior (`auto`, `any`, `tool`, `none`) and optional parallel-call behavior. thinking: anyOf: - $ref: '#/components/schemas/MessagesThinkingConfig' - type: 'null' description: >- Controls reasoning behavior with mode (`enabled`, `disabled`, `adaptive`). `enabled` requires `budget_tokens`; `disabled` and `adaptive` must not include it. output_config: anyOf: - $ref: '#/components/schemas/MessagesOutputConfig' - type: 'null' description: >- Output generation options including effort level and structured output format settings. cache_control: anyOf: - additionalProperties: true type: object - type: 'null' title: Cache Control description: >- Compatibility field accepted for request portability. Parsed but not used for generation. container: anyOf: - additionalProperties: true type: object - type: 'null' title: Container description: >- Compatibility field accepted for request portability. Parsed but not used for generation. context_manager: anyOf: - additionalProperties: true type: object - type: 'null' title: Context Manager description: >- Compatibility field accepted for request portability. Parsed but not used for generation. inference_geo: anyOf: - additionalProperties: true type: object - type: 'null' title: Inference Geo description: >- Compatibility field accepted for request portability. Parsed but not used for generation. metadata: anyOf: - additionalProperties: true type: object - type: 'null' title: Metadata description: >- Compatibility field accepted for request portability. Parsed but not used for generation. service_tier: anyOf: - type: string - additionalProperties: true type: object - type: 'null' title: Service Tier description: >- Compatibility field accepted for request portability. Parsed but not used for generation. additionalProperties: true type: object required: - messages - model title: DedicatedMessagesBody example: max_tokens: 128 messages: - content: Hello, summarize what you can do in one sentence. role: user model: (endpoint-id) DedicatedMessagesSuccess: properties: id: type: string title: Id description: Unique identifier for this message response. type: type: string const: message title: Type description: Response object type (`message`). role: type: string const: assistant title: Role description: Role of the output message author (`assistant`). content: items: $ref: '#/components/schemas/MessagesResponseContentBlock' type: array title: Content description: Assistant output blocks in generation order. stop_reason: anyOf: - type: string enum: - end_turn - max_tokens - tool_use - stop_sequence - type: 'null' title: Stop Reason description: >- Why generation stopped (`end_turn`, `max_tokens`, `tool_use`, `stop_sequence`). stop_sequence: anyOf: - type: string - type: 'null' title: Stop Sequence description: >- Matched stop string when `stop_reason` is `stop_sequence`; otherwise null. usage: $ref: '#/components/schemas/MessagesUsage' description: Token usage details for this response. model: anyOf: - type: string - type: 'null' title: Model description: Model used to generate the response. additionalProperties: true type: object required: - id - type - role - content - usage title: DedicatedMessagesSuccess DedicatedTextClassificationBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) input: anyOf: - type: string - items: type: string type: array - type: 'null' title: Input description: >- Input text to classify, encoded as a string or array of strings. To classify multiple inputs in a single request, pass an array of strings. Either `input` or `tokens` field is required. examples: - I love programming. tokens: anyOf: - items: type: integer type: array - type: 'null' title: Tokens description: |- The tokenized prompt (i.e., input tokens). Either `input` or `tokens` field is required. examples: - 72 - 1563 - 2335 - 13 type: object required: - model title: DedicatedTextClassificationBody example: input: I love programming. model: (endpoint-id) DedicatedTextClassificationSuccess: $ref: '#/components/schemas/TextClassificationResult' title: DedicatedTextClassificationSuccess DedicatedTokenizationBody: properties: model: type: string title: Model description: >- ID of target endpoint. If you want to send request to specific adapter, use the format "YOUR_ENDPOINT_ID:YOUR_ADAPTER_ROUTE". Otherwise, you can just use "YOUR_ENDPOINT_ID" alone. examples: - (endpoint-id) prompt: type: string title: Prompt description: Input text prompt to tokenize. examples: - What is generative AI? type: object required: - model - prompt title: DedicatedTokenizationBody example: model: (endpoint-id) prompt: What is generative AI? DedicatedTokenizationSuccess: properties: tokens: items: type: integer type: array title: Tokens description: A list of token IDs. type: object required: - tokens title: DedicatedTokenizationSuccess DeleteSamplesResponse: properties: sampleIds: items: type: string type: array title: Sampleids description: List of sample IDs that were deleted from the dataset. type: object required: - sampleIds title: DeleteSamplesResponse description: Delete samples response. EmbeddingObject: properties: index: type: integer title: Index description: The index of the embedding in the list of embeddings. object: type: string const: embedding title: Object description: The object type, which is always set to `embedding`. embedding: anyOf: - items: type: number type: array - type: string format: binary title: Embedding description: >- The embedding vector, which is a list of floats or a base64-encoded string. The length of vector depends on the model. type: object required: - index - object - embedding title: EmbeddingObject EndpointAdvancedConfig: properties: max_batch_size: anyOf: - type: integer - type: 'null' title: Max Batch Size tokenizer_skip_special_tokens: type: boolean title: Tokenizer Skip Special Tokens tokenizer_add_special_tokens: type: boolean title: Tokenizer Add Special Tokens max_token_count: type: integer title: Max Token Count default: 2560 enable_content_logging: anyOf: - type: boolean - type: 'null' title: Enable Content Logging max_input_length: anyOf: - type: integer - type: 'null' title: Max Input Length type: object required: - tokenizer_skip_special_tokens - tokenizer_add_special_tokens title: EndpointAdvancedConfig description: Endpoint advanced config. EndpointSimplescaleConfig: properties: replicas: type: integer minimum: 1 title: Replicas type: object required: - replicas title: EndpointSimplescaleConfig description: Simple scaling options. FileBuiltInTool: properties: type: type: string const: file:text title: Type description: >- The type of the file parser tool. Only .txt and .pdf files are supported. files: items: type: string type: array title: Files description: A List of file IDs. For now, only one file is supported. type: object required: - type - files title: FileBuiltInTool FileGetDownloadUrlResponse: properties: fileId: type: string title: Fileid description: ID of the file. s3Uri: type: string title: S3Uri description: S3 URI of the file. downloadUrl: type: string title: Downloadurl description: Download URL of the file. type: object required: - fileId - s3Uri - downloadUrl title: FileGetDownloadUrlResponse description: Get download URL response. FileInfo: properties: id: type: string title: Id description: ID of the file. name: type: string title: Name description: Name of the file. size: type: integer title: Size description: Size of the file in bytes. digest: type: string title: Digest description: SHA256 hash of the file. projectId: type: string title: Projectid description: ID of the project the file belongs to. creatorId: type: string title: Creatorid description: ID of the creator of the file. createdAt: type: string format: date-time title: Createdat description: Unix timestamp (in seconds) of when the file was created. expiresAt: anyOf: - type: string format: date-time - type: 'null' title: Expiresat description: Unix timestamp (in seconds) of when the file will expire. status: $ref: '#/components/schemas/AccountFileStatus' description: Status of the file upload. type: object required: - id - name - size - digest - projectId - creatorId - createdAt - expiresAt - status title: FileInfo description: File information. FileInitUploadRequest: properties: name: type: string title: Name description: Name of the file. size: type: integer title: Size description: Size of the file in bytes. digest: type: string pattern: ^[a-zA-Z0-9_+.-]+:[a-fA-F0-9]+$ title: Digest description: Digest of the file. projectId: type: string title: Projectid description: ID of the project the file belongs to. type: object required: - name - size - digest - projectId title: FileInitUploadRequest description: Initiate file upload request. FileInitUploadResponse: properties: fileId: type: string title: Fileid description: ID of the file. uploadUrl: anyOf: - type: string - type: 'null' title: Uploadurl description: Upload URL of the file. `None` if the file is already uploaded. aws: anyOf: - additionalProperties: true type: object - type: 'null' title: Aws description: AWS fields to be uploaded with file. type: object required: - fileId title: FileInitUploadResponse description: Initiate file upload response. FriendliBuiltInTool: properties: type: type: string enum: - math:calculator - math:statistics - math:calendar - web:search - web:url - code:python-interpreter title: Type description: The type of the built-in tool provided by Friendli. type: object required: - type title: FriendliBuiltInTool Function: properties: description: anyOf: - type: string - type: 'null' title: Description description: >- A description of what the function does, used by the model to choose when and how to call the function. name: type: string title: Name description: >- The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. parameters: additionalProperties: true type: object title: Parameters description: > The parameters the functions accepts, described as a JSON Schema object. To represent a function with no parameters, use the value `{"type": "object", "properties": {}}`. type: object required: - name - parameters title: Function FunctionResult: properties: arguments: type: string title: Arguments description: >- The arguments for calling the function, generated by the model in JSON format. Ensure to validate these arguments in your code before invoking the function since the model may not always produce valid JSON. name: type: string title: Name description: The name of the function to call. type: object required: - arguments - name title: FunctionResult Functionality: properties: tool_call: type: boolean title: Tool Call builtin_tool: type: boolean title: Built-in Tool parallel_tool_call: type: boolean title: Tool Call Parallel structured_output: type: boolean title: Structured Output default: true type: object required: - tool_call - builtin_tool - parallel_tool_call title: Functionality description: Functionality of the model. HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError ImageContent: properties: type: type: string const: image_url title: Type description: The type of the message content. image_url: $ref: '#/components/schemas/ImageData' description: The image URL data. type: object required: - type - image_url title: ImageContent ImageData: properties: url: type: string title: Url description: The URL of the image. type: object required: - url title: ImageData ImageGenerationContainerResult: properties: image: type: string format: binary title: Image description: The base64-encoded image. type: object required: - image title: ImageGenerationContainerResult ImageGenerationDedicatedResult: properties: data: anyOf: - items: $ref: '#/components/schemas/UrlImageItem' type: array - items: $ref: '#/components/schemas/B64ImageItem' type: array title: Data type: object required: - data title: ImageGenerationDedicatedResult ImageInput: properties: image: type: string title: Image description: >- An input image, provided either as a URL or a base64-encoded string. Maximum supported image size is 50 MiB. examples: - >- https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png - data:image/png;base64,... type: object required: - image title: ImageInput InferenceDeploymentErrorCode: type: string enum: - WORKLOAD_INIT_UNKNOWN_ERROR - WORKLOAD_INIT_SETTINGS_ERROR - WORKLOAD_INIT_GRPC_ERROR - WORKLOAD_INIT_MANIFEST_NOT_FOUND_ERROR - WORKLOAD_INIT_MANIFEST_TYPE_ERROR - WORKLOAD_INIT_DOWNLOAD_ERROR - WORKLOAD_INIT_INVALID_TOKEN_ERROR - WORKLOAD_INIT_CANNOT_ACCESS_REPO_ERROR - WORKLOAD_INIT_HF_WANDB_API_ERROR - WORKLOAD_INIT_INSUFFICIENT_DISK_ERROR - INFERENCE_ENGINE_UNKNOWN_ERROR - INFERENCE_ENGINE_INVALID_ARGUMENT_ERROR - INFERENCE_ENGINE_MEMORY_ERROR - INFERENCE_ENGINE_METERING_CLIENT_CONFIG_ERROR title: InferenceDeploymentErrorCode description: ErrorCode type. InferenceDeploymentStatus: type: string enum: - UNKNOWN - INITIALIZING - RUNNING - UPDATING - SLEEPING - AWAKING - FAILED - STOPPING - TERMINATING - TERMINATED - READY title: InferenceDeploymentStatus description: Status of inference deployment. IntegratedBuiltInTool: properties: type: type: string const: linkup:search title: Type description: The type of the integrated third-party tool. type: object required: - type title: IntegratedBuiltInTool KnowledgeRetrievedChunk: properties: text: type: string title: Text description: Retrieved text matching the search query. score: type: number title: Score description: Numerical relevance score of the retrieved result. contentChunkId: type: string title: Content Chunk Id description: ID of specific text segment. contentId: type: string title: Content Id description: ID of content in knowledge-base. type: object required: - text - score - contentChunkId - contentId title: KnowledgeRetrievedChunk description: Knowledge retrieved chunk. ListDatasetsResponse: properties: data: items: $ref: '#/components/schemas/DatasetInfo' type: array title: Data description: List of items. nextCursor: anyOf: - type: string format: binary - type: 'null' title: Nextcursor description: >- Next cursor to use for pagination. `None` indicates no more items to fetch. type: object required: - data title: ListDatasetsResponse description: List datasets response. ListSamplesResponse: properties: data: items: prefixItems: - type: string - type: string type: array maxItems: 2 minItems: 2 type: array title: Data description: List of items. nextCursor: anyOf: - type: string format: binary - type: 'null' title: Nextcursor description: >- Next cursor to use for pagination. `None` indicates no more items to fetch. type: object required: - data title: ListSamplesResponse description: List samples response. ListSplitsResponse: properties: data: items: $ref: '#/components/schemas/SplitInfo' type: array title: Data description: List of items. nextCursor: anyOf: - type: string format: binary - type: 'null' title: Nextcursor description: >- Next cursor to use for pagination. `None` indicates no more items to fetch. type: object required: - data title: ListSplitsResponse description: List splits response. ListVersionsResponse: properties: versions: items: $ref: '#/components/schemas/VersionInfo' type: array title: Versions description: List of versions. type: object required: - versions title: ListVersionsResponse description: List versions response. Message: oneOf: - $ref: '#/components/schemas/SystemMessage' title: System - $ref: '#/components/schemas/UserMessage' title: User - $ref: '#/components/schemas/AssistantMessage' title: Assistant - $ref: '#/components/schemas/ToolMessage' title: Tool discriminator: propertyName: role mapping: assistant: '#/components/schemas/AssistantMessage' system: '#/components/schemas/SystemMessage' tool: '#/components/schemas/ToolMessage' user: '#/components/schemas/UserMessage' MessagesBase64ImageSource: properties: type: type: string const: base64 title: Type description: Image source type. Must be `base64`. data: type: string minLength: 1 title: Data description: Base64-encoded image bytes (without data URL prefix). media_type: type: string minLength: 1 title: Media Type description: MIME type, for example `image/png`. type: object required: - type - data - media_type title: MessagesBase64ImageSource MessagesCompatibilityBlock: properties: type: type: string enum: - document - search_result - redacted_thinking - server_tool_use - web_search_tool_result - web_fetch_tool_result - code_execution_tool_result - bash_code_execution_tool_result - text_editor_code_execution_tool_result - tool_search_tool_result - container_upload title: Type description: >- Compatibility block type. These blocks are accepted for parsing compatibility and ignored for generation. type: object required: - type title: MessagesCompatibilityBlock MessagesContentBlock: oneOf: - $ref: '#/components/schemas/MessagesTextBlock' title: Text - $ref: '#/components/schemas/MessagesImageBlock' title: Image - $ref: '#/components/schemas/MessagesThinkingBlock' title: Thinking - $ref: '#/components/schemas/MessagesToolUseBlock' title: Tool Use - $ref: '#/components/schemas/MessagesToolResultBlock' title: Tool Result - $ref: '#/components/schemas/MessagesCompatibilityBlock' title: Compatibility Block discriminator: propertyName: type mapping: bash_code_execution_tool_result: '#/components/schemas/MessagesCompatibilityBlock' code_execution_tool_result: '#/components/schemas/MessagesCompatibilityBlock' container_upload: '#/components/schemas/MessagesCompatibilityBlock' document: '#/components/schemas/MessagesCompatibilityBlock' image: '#/components/schemas/MessagesImageBlock' redacted_thinking: '#/components/schemas/MessagesCompatibilityBlock' search_result: '#/components/schemas/MessagesCompatibilityBlock' server_tool_use: '#/components/schemas/MessagesCompatibilityBlock' text: '#/components/schemas/MessagesTextBlock' text_editor_code_execution_tool_result: '#/components/schemas/MessagesCompatibilityBlock' thinking: '#/components/schemas/MessagesThinkingBlock' tool_result: '#/components/schemas/MessagesToolResultBlock' tool_search_tool_result: '#/components/schemas/MessagesCompatibilityBlock' tool_use: '#/components/schemas/MessagesToolUseBlock' web_fetch_tool_result: '#/components/schemas/MessagesCompatibilityBlock' web_search_tool_result: '#/components/schemas/MessagesCompatibilityBlock' MessagesErrorObject: properties: type: type: string title: Type description: >- Error category. For HTTP 422 in Messages API, this is `invalid_request_error`. message: type: string title: Message description: Human-readable error message. additionalProperties: true type: object required: - type - message title: MessagesErrorObject MessagesErrorResponse: properties: type: type: string const: error title: Type description: Top-level response type for Messages API errors. error: $ref: '#/components/schemas/MessagesErrorObject' description: Structured error object. request_id: anyOf: - type: string - type: 'null' title: Request Id description: >- Request identifier for debugging and support. It may be omitted in some failure paths. additionalProperties: true type: object required: - type - error title: MessagesErrorResponse MessagesImageBlock: properties: type: type: string const: image title: Type description: Content block type. Must be `image`. source: $ref: '#/components/schemas/MessagesImageSource' description: Image source descriptor (`url` or `base64`). type: object required: - type - source title: MessagesImageBlock MessagesImageSource: oneOf: - $ref: '#/components/schemas/MessagesBase64ImageSource' title: Base64 - $ref: '#/components/schemas/MessagesUrlImageSource' title: URL discriminator: propertyName: type mapping: base64: '#/components/schemas/MessagesBase64ImageSource' url: '#/components/schemas/MessagesUrlImageSource' MessagesInputMessage: properties: role: type: string enum: - user - assistant title: Role description: >- Author role for this message turn. Use `user` for user input and `assistant` for prior assistant turns. content: anyOf: - type: string - items: $ref: '#/components/schemas/MessagesContentBlock' type: array title: Content description: >- Message payload. Supports plain string shorthand or a typed block array. type: object required: - role - content title: MessagesInputMessage MessagesOutputConfig: properties: effort: anyOf: - type: string enum: - low - medium - high - max - type: 'null' title: Effort description: >- Relative generation effort level (`low`, `medium`, `high`, `max`). Higher effort can improve quality on harder prompts at the cost of additional compute. format: anyOf: - $ref: '#/components/schemas/MessagesOutputFormat' - type: 'null' description: Structured output settings. Currently supports only `json_schema`. type: object title: MessagesOutputConfig MessagesOutputFormat: properties: type: anyOf: - type: string const: json_schema - type: 'null' title: Type description: >- Output format type. Set to `json_schema` to constrain output to a JSON schema. schema: anyOf: - additionalProperties: true type: object - type: 'null' title: Schema description: JSON Schema object applied when `type` is `json_schema`. type: object title: MessagesOutputFormat MessagesResponseContentBlock: oneOf: - $ref: '#/components/schemas/MessagesResponseThinkingBlock' title: Thinking - $ref: '#/components/schemas/MessagesResponseTextBlock' title: Text - $ref: '#/components/schemas/MessagesResponseToolUseBlock' title: Tool Use discriminator: propertyName: type mapping: text: '#/components/schemas/MessagesResponseTextBlock' thinking: '#/components/schemas/MessagesResponseThinkingBlock' tool_use: '#/components/schemas/MessagesResponseToolUseBlock' MessagesResponseTextBlock: properties: type: type: string const: text title: Type description: Content block type. text: type: string title: Text description: Assistant-generated text for this block. type: object required: - type - text title: MessagesResponseTextBlock MessagesResponseThinkingBlock: properties: type: type: string const: thinking title: Type description: Content block type. thinking: type: string title: Thinking description: Reasoning content returned by the model. signature: type: string title: Signature description: Signature associated with the reasoning content. type: object required: - type - thinking - signature title: MessagesResponseThinkingBlock MessagesResponseToolUseBlock: properties: type: type: string const: tool_use title: Type description: Content block type. id: type: string title: Id description: Identifier of the tool call emitted by the model. name: type: string title: Name description: Name of the invoked tool. input: additionalProperties: true type: object title: Input description: Parsed tool argument object generated by the model. type: object required: - type - id - name - input title: MessagesResponseToolUseBlock MessagesSystemTextBlock: properties: type: type: string const: text title: Type description: System block type. Must be `text`. text: type: string title: Text description: System instruction text content. type: object required: - type - text title: MessagesSystemTextBlock MessagesTextBlock: properties: type: type: string const: text title: Type description: Content block type. Must be `text`. text: type: string title: Text description: Text content for this block. type: object required: - type - text title: MessagesTextBlock MessagesThinkingBlock: properties: type: type: string const: thinking title: Type description: Content block type. Must be `thinking`. thinking: type: string title: Thinking description: Reasoning content for this block. signature: anyOf: - type: string - type: 'null' title: Signature description: Optional signature associated with the reasoning content. type: object required: - type - thinking title: MessagesThinkingBlock MessagesThinkingConfig: properties: type: type: string enum: - enabled - disabled - adaptive title: Type description: >- Reasoning mode. `enabled` forces reasoning, `disabled` suppresses it, and `adaptive` lets the system decide. budget_tokens: anyOf: - type: integer minimum: 1 - type: 'null' title: Budget Tokens description: >- Reasoning token budget. Required when `type=enabled`; must not be provided when `type=disabled` or `type=adaptive`. If `max_tokens` is set, this must be smaller than `max_tokens`. type: object required: - type title: MessagesThinkingConfig MessagesToolChoice: properties: type: type: string enum: - auto - any - tool - none title: Type description: >- Tool-calling mode. `auto` lets the model decide, `any` requires at least one tool call, `tool` forces a named tool, and `none` disables tool calls. name: anyOf: - type: string - type: 'null' title: Name description: >- Tool name to force when `type=tool`. Ignored for other `type` values. disable_parallel_tool_use: anyOf: - type: boolean - type: 'null' title: Disable Parallel Tool Use description: >- If true, restricts the model to at most one tool call at a time. Must not be provided when `type=none`. type: object required: - type title: MessagesToolChoice MessagesToolDefinition: properties: name: type: string title: Name description: >- Tool identifier used in model tool-call outputs. Keep this name stable and unique within the request. type: anyOf: - type: string - type: 'null' title: Type description: >- Tool category. If missing, empty, or `custom`, it is converted to a function tool. Any non-empty value other than `custom` is treated as a server-side tool and skipped. description: anyOf: - type: string - type: 'null' title: Description description: >- Natural-language description of what the tool does and when to call it. input_schema: anyOf: - additionalProperties: true type: object - type: 'null' title: Input Schema description: >- JSON Schema describing tool arguments. The model uses this schema to construct the tool input object. strict: anyOf: - type: boolean - type: 'null' title: Strict description: >- If true, applies stricter schema adherence when generating tool arguments. type: object required: - name title: MessagesToolDefinition MessagesToolResultBlock: properties: type: type: string const: tool_result title: Type description: Content block type. Must be `tool_result`. tool_use_id: type: string title: Tool Use Id description: Identifier of the related `tool_use` block this result answers. content: anyOf: - type: string - items: additionalProperties: true type: object type: array - type: 'null' title: Content description: >- Tool result payload. Can be plain text, a list of objects, or null when no body is returned. type: object required: - type - tool_use_id title: MessagesToolResultBlock MessagesToolUseBlock: properties: type: type: string const: tool_use title: Type description: Content block type. Must be `tool_use`. id: anyOf: - type: string - type: 'null' title: Id description: >- Optional tool call identifier. Use this value to link follow-up `tool_result` blocks. name: type: string title: Name description: Name of the tool to invoke. input: additionalProperties: true type: object title: Input description: Tool input argument object. type: object required: - type - name - input title: MessagesToolUseBlock MessagesUrlImageSource: properties: type: type: string const: url title: Type description: Image source type. Must be `url`. url: type: string minLength: 1 title: Url description: Publicly accessible image URL that the service can fetch. type: object required: - type - url title: MessagesUrlImageSource MessagesUsage: properties: input_tokens: type: integer minimum: 0 title: Input Tokens description: Number of billed input tokens for this request. output_tokens: type: integer minimum: 0 title: Output Tokens description: Number of billed output tokens generated by the model. cache_read_input_tokens: anyOf: - type: integer minimum: 0 - type: 'null' title: Cache Read Input Tokens description: >- Number of input tokens served from cache, when cache reads are applicable. type: object required: - input_tokens - output_tokens title: MessagesUsage ModelCatalogResponseItem: properties: id: type: string title: Id name: type: string title: Name max_completion_tokens: type: integer title: Max Completion Tokens context_length: type: integer title: Context Length functionality: $ref: '#/components/schemas/Functionality' description: Model functionality pricing: $ref: '#/components/schemas/PricingModel' title: Pricing description: Token prices for input and output hugging_face_url: type: string minLength: 1 format: uri title: Hugging Face Url description: type: string title: Description license: type: string minLength: 1 format: uri title: License policy: anyOf: - type: string minLength: 1 format: uri - type: 'null' title: Policy created: type: integer title: Created deprecation_date: anyOf: - type: string - type: 'null' title: Deprecation Date description: YYYY-MM-DD (UTC date) type: object required: - id - name - max_completion_tokens - context_length - functionality - pricing - hugging_face_url - description - license - policy - created title: ModelCatalogResponseItem description: Model listing in `/models` API. PricingModel: properties: input: type: number minimum: 0 title: Input description: Price per input token output: type: number minimum: 0 title: Output description: Price per output token prompt: type: number minimum: 0 title: Prompt description: Price per input token (alias of input) completion: type: number minimum: 0 title: Completion description: Price per output token (alias of output) input_cache_read: anyOf: - type: number minimum: 0 - type: 'null' title: Input Cache Read description: Price per cached input token read response_time: type: number minimum: 0 title: Response Time description: Price per response time in seconds default: 0 audio_minute: anyOf: - type: number minimum: 0 - type: 'null' title: Audio Minute description: Price per audio minute unit_type: $ref: '#/components/schemas/ServerlessPriceUnitType' description: Pricing unit type (TOKEN / SECOND / AUDIO_MINUTE) default: TOKEN type: object required: - input - output - prompt - completion title: PricingModel description: Pricing model supporting both token-based and time-based pricing. PromptTokensDetails: properties: cached_tokens: anyOf: - type: integer - type: 'null' title: Cached Tokens description: Cached tokens present in the prompt. type: object title: PromptTokensDetails ResponseFormat: oneOf: - $ref: '#/components/schemas/ResponseFormatJsonSchema' title: Json Schema - $ref: '#/components/schemas/ResponseFormatJsonObject' title: Json Object - $ref: '#/components/schemas/ResponseFormatRegex' title: Regex - $ref: '#/components/schemas/ResponseFormatText' title: Text description: >- The enforced format of the model's output. Note that the content of the output message may be truncated if it exceeds the `max_tokens`. You can check this by verifying that the `finish_reason` of the output message is `length`. For more detailed information, please refer [here](https://friendli.ai/docs/guides/structured-outputs). ***Important*** You must explicitly instruct the model to produce the desired output format using a system prompt or user message (e.g., `You are an API generating a valid JSON as output.`). Otherwise, the model may result in an unending stream of whitespace or other characters. **When `response_format` is specified, `min_tokens` field is unsupported.** discriminator: propertyName: type mapping: json_object: '#/components/schemas/ResponseFormatJsonObject' json_schema: '#/components/schemas/ResponseFormatJsonSchema' regex: '#/components/schemas/ResponseFormatRegex' text: '#/components/schemas/ResponseFormatText' ResponseFormatJsonObject: properties: type: type: string const: json_object title: Type description: 'The type of the response format: `json_object`' type: object required: - type title: ResponseFormatJsonObject ResponseFormatJsonSchema: properties: type: type: string const: json_schema title: Type description: 'The type of the response format: `json_schema`' json_schema: $ref: '#/components/schemas/ResponseFormatJsonSchemaSchema' type: object required: - type - json_schema title: ResponseFormatJsonSchema ResponseFormatJsonSchemaSchema: properties: schema: additionalProperties: true type: object title: Schema description: >- The schema for the response format, described as a JSON Schema object. type: object required: - schema title: ResponseFormatJsonSchemaSchema ResponseFormatRegex: properties: type: type: string const: regex title: Type description: 'The type of the response format: `regex`' schema: type: string title: Schema description: >- The schema of the output. Lookaheads or lookbehinds (e.g., `\a`, `\z`, `^`, `$`, `(?=)`, `(?!)`, `(?<=...)`, `(?- Must be set to `server_vad` to enable manual chunking using server side VAD. prefix_padding_ms: anyOf: - type: integer - type: 'null' title: Prefix Padding Ms description: >- Amount of audio to include before the VAD detected speech (in milliseconds). default: 300 silence_duration_ms: anyOf: - type: integer - type: 'null' title: Silence Duration Ms description: >- Duration of silence to detect speech stop (in milliseconds). With shorter values the model will respond more quickly, but may jump in on short pauses from the user. default: 200 threshold: anyOf: - type: number - type: 'null' title: Threshold description: >- Sensitivity threshold (0.0 to 1.0) for voice activity detection. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments. default: 0.5 type: object required: - type title: ServerVadChunkingStrategy ServerlessAudioTranscriptionBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - openai/whisper-large-v3 file: type: string format: binary title: File description: >- The audio file object (not file name) to transcribe, in one of these formats: mp3, wav, flac, ogg, and many other standard audio formats. chunking_strategy: anyOf: - type: string const: auto - $ref: '#/components/schemas/ServerVadChunkingStrategy' - type: 'null' title: Chunking Strategy description: >- Controls how the audio is cut into chunks. When set to `"auto"`, the server first normalizes loudness and then uses voice activity detection (VAD) to choose boundaries. `server_vad` object can be provided to tweak VAD detection parameters manually. If unset, the audio is transcribed as a single block. language: anyOf: - type: string - type: 'null' title: Language description: >- The language of the input audio. Supplying the input language in [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format will improve accuracy and latency. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the transcription result. When set to `true`, the transcription result will be streamed as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. type: object required: - model - file title: ServerlessAudioTranscriptionBody example: file: '@/path/to/file/audio.mp3' model: openai/whisper-large-v3 ServerlessAudioTranscriptionStreamBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - openai/whisper-large-v3 file: type: string format: binary title: File description: >- The audio file object (not file name) to transcribe, in one of these formats: mp3, wav, flac, ogg, and many other standard audio formats. chunking_strategy: anyOf: - type: string const: auto - $ref: '#/components/schemas/ServerVadChunkingStrategy' - type: 'null' title: Chunking Strategy description: >- Controls how the audio is cut into chunks. When set to `"auto"`, the server first normalizes loudness and then uses voice activity detection (VAD) to choose boundaries. `server_vad` object can be provided to tweak VAD detection parameters manually. If unset, the audio is transcribed as a single block. language: anyOf: - type: string - type: 'null' title: Language description: >- The language of the input audio. Supplying the input language in [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (e.g. `en`) format will improve accuracy and latency. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the transcription result. When set to `true`, the transcription result will be streamed as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. type: object required: - model - file title: ServerlessAudioTranscriptionStreamBody example: file: '@/path/to/file/audio.mp3' model: openai/whisper-large-v3 ServerlessAudioTranscriptionStreamSuccess: $ref: '#/components/schemas/StreamedAudioTranscriptionResult' title: ServerlessAudioTranscriptionStreamSuccess ServerlessAudioTranscriptionSuccess: $ref: '#/components/schemas/AudioTranscriptionResult' title: ServerlessAudioTranscriptionSuccess ServerlessChatCompleteSuccess: $ref: '#/components/schemas/ChatResult' title: ServerlessChatCompleteSuccess ServerlessChatCompletionBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `tools` or `response_format` is specified.** response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' description: >- The enforced format of the model's output. Note that the content of the output message may be truncated if it exceeds the `max_tokens`. You can check this by verifying that the `finish_reason` of the output message is `length`. For more detailed information, please refer [here](https://friendli.ai/docs/guides/structured-outputs). ***Important*** You must explicitly instruct the model to produce the desired output format using a system prompt or user message (e.g., `You are an API generating a valid JSON as output.`). Otherwise, the model may result in an unending stream of whitespace or other characters. **This field is unsupported when `tools` is specified.** **When `response_format` is specified, `min_tokens` field is unsupported.** type: object required: - model - messages title: ServerlessChatCompletionBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user model: meta-llama/Llama-3.1-8B-Instruct ServerlessChatCompletionStreamBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** min_tokens: anyOf: - type: integer - type: 'null' title: Min Tokens description: >- The minimum number of tokens to generate. Default value is 0. This is similar to Hugging Face's [`min_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.min_new_tokens) argument. **This field is unsupported when `tools` or `response_format` is specified.** response_format: anyOf: - $ref: '#/components/schemas/ResponseFormat' - type: 'null' description: >- The enforced format of the model's output. Note that the content of the output message may be truncated if it exceeds the `max_tokens`. You can check this by verifying that the `finish_reason` of the output message is `length`. For more detailed information, please refer [here](https://friendli.ai/docs/guides/structured-outputs). ***Important*** You must explicitly instruct the model to produce the desired output format using a system prompt or user message (e.g., `You are an API generating a valid JSON as output.`). Otherwise, the model may result in an unending stream of whitespace or other characters. **This field is unsupported when `tools` is specified.** **When `response_format` is specified, `min_tokens` field is unsupported.** type: object required: - model - messages title: ServerlessChatCompletionStreamBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user model: meta-llama/Llama-3.1-8B-Instruct ServerlessChatCompletionStreamSuccess: $ref: '#/components/schemas/StreamedChatResult' title: ServerlessChatCompletionStreamSuccess ServerlessChatRenderBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. tools: anyOf: - items: $ref: '#/components/schemas/Tool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. Use this to provide a list of functions the model may generate JSON inputs for. **When `tools` is specified, `min_tokens` and `response_format` fields are unsupported.** type: object required: - model - messages title: ServerlessChatRenderBody example: messages: - content: You are a helpful assistant. role: system - content: Hello! role: user model: meta-llama/Llama-3.1-8B-Instruct ServerlessChatRenderSuccess: properties: text: type: string title: Text description: The rendered text. type: object required: - text title: ServerlessChatRenderSuccess ServerlessCompletionsBody: anyOf: - $ref: '#/components/schemas/CompletionsServerlessBodyWithPrompt' - $ref: '#/components/schemas/CompletionsServerlessBodyWithTokens' title: ServerlessCompletionsBody ServerlessCompletionsStreamBody: anyOf: - $ref: '#/components/schemas/CompletionsStreamServerlessBodyWithPrompt' - $ref: '#/components/schemas/CompletionsStreamServerlessBodyWithTokens' title: ServerlessCompletionsStreamBody ServerlessCompletionsStreamSuccess: $ref: '#/components/schemas/StreamedCompletionsResult' title: ServerlessCompletionsStreamSuccess ServerlessCompletionsSuccess: $ref: '#/components/schemas/CompletionsResult' title: ServerlessCompletionsSuccess ServerlessDetokenizationBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct tokens: items: type: integer type: array title: Tokens description: A token sequence to detokenize. examples: - - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 type: object required: - model - tokens title: ServerlessDetokenizationBody example: model: meta-llama/Llama-3.1-8B-Instruct tokens: - 128000 - 3923 - 374 - 1803 - 1413 - 15592 - 30 ServerlessDetokenizationSuccess: properties: text: type: string title: Text description: Detokenized text output. type: object required: - text title: ServerlessDetokenizationSuccess ServerlessKnowledgeRetrievalBody: properties: query: type: string title: Query description: >- A text string used to find relevant information within the knowledge-base. examples: - Chicken dinner k: type: integer title: K description: >- Maximum number of top-ranked knowledge-base entries to return in results. examples: - 1 knowledgeIds: items: type: string type: array title: Knowledge Ids description: >- A List of knowledge-base IDs. For now, only one knowledge-base is supported. examples: - - knowledge-base-id-1 type: object required: - query - k - knowledgeIds title: ServerlessKnowledgeRetrievalBody description: Knowledge retrieval request. ServerlessKnowledgeRetrievalSuccess: properties: results: items: $ref: '#/components/schemas/KnowledgeRetrievedChunk' type: array title: Results description: A list of result. type: object required: - results title: ServerlessKnowledgeRetrievalSuccess description: Knowledge retrieval response. ServerlessMessagesBody: properties: messages: items: $ref: '#/components/schemas/MessagesInputMessage' type: array minItems: 1 title: Messages description: >- A list of conversation messages ordered from oldest to newest. Must contain at least one item. examples: - - content: Explain top_p in one sentence. role: user max_tokens: anyOf: - type: integer minimum: 1 - type: 'null' title: Max Tokens description: >- Maximum number of tokens to generate for the assistant response. Must be greater than 0 when provided. model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct system: anyOf: - anyOf: - type: string - items: $ref: '#/components/schemas/MessagesSystemTextBlock' type: array description: Payload format for the top-level `system` instruction. - type: 'null' title: System description: >- Optional top-level system instruction applied before conversation turns. Supports a plain string or an array of text blocks. stream: anyOf: - type: boolean - type: 'null' title: Stream description: >- Whether to stream output as server-sent events (`text/event-stream`). When false or omitted, returns a single JSON response. temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Lower values make outputs more deterministic; higher values increase diversity. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Nucleus sampling parameter. The model samples from the smallest token set whose cumulative probability reaches `top_p`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: Limits sampling to the `k` most likely tokens at each decoding step. stop_sequences: anyOf: - items: type: string type: array - type: 'null' title: Stop Sequences description: >- Stop strings that terminate generation when matched in output. The matched value is returned in `stop_sequence` when applicable. tools: anyOf: - items: $ref: '#/components/schemas/MessagesToolDefinition' type: array - type: 'null' title: Tools description: >- Tool definitions available to the model. Use this to allow tool calls with structured arguments. tool_choice: anyOf: - $ref: '#/components/schemas/MessagesToolChoice' - type: 'null' description: >- Controls tool-calling behavior (`auto`, `any`, `tool`, `none`) and optional parallel-call behavior. thinking: anyOf: - $ref: '#/components/schemas/MessagesThinkingConfig' - type: 'null' description: >- Controls reasoning behavior with mode (`enabled`, `disabled`, `adaptive`). `enabled` requires `budget_tokens`; `disabled` and `adaptive` must not include it. output_config: anyOf: - $ref: '#/components/schemas/MessagesOutputConfig' - type: 'null' description: >- Output generation options including effort level and structured output format settings. cache_control: anyOf: - additionalProperties: true type: object - type: 'null' title: Cache Control description: >- Compatibility field accepted for request portability. Parsed but not used for generation. container: anyOf: - additionalProperties: true type: object - type: 'null' title: Container description: >- Compatibility field accepted for request portability. Parsed but not used for generation. context_manager: anyOf: - additionalProperties: true type: object - type: 'null' title: Context Manager description: >- Compatibility field accepted for request portability. Parsed but not used for generation. inference_geo: anyOf: - additionalProperties: true type: object - type: 'null' title: Inference Geo description: >- Compatibility field accepted for request portability. Parsed but not used for generation. metadata: anyOf: - additionalProperties: true type: object - type: 'null' title: Metadata description: >- Compatibility field accepted for request portability. Parsed but not used for generation. service_tier: anyOf: - type: string - additionalProperties: true type: object - type: 'null' title: Service Tier description: >- Compatibility field accepted for request portability. Parsed but not used for generation. additionalProperties: true type: object required: - messages - model title: ServerlessMessagesBody example: max_tokens: 128 messages: - content: Hello, summarize what you can do in one sentence. role: user model: meta-llama/Llama-3.1-8B-Instruct ServerlessMessagesSuccess: properties: id: type: string title: Id description: Unique identifier for this message response. type: type: string const: message title: Type description: Response object type (`message`). role: type: string const: assistant title: Role description: Role of the output message author (`assistant`). content: items: $ref: '#/components/schemas/MessagesResponseContentBlock' type: array title: Content description: Assistant output blocks in generation order. stop_reason: anyOf: - type: string enum: - end_turn - max_tokens - tool_use - stop_sequence - type: 'null' title: Stop Reason description: >- Why generation stopped (`end_turn`, `max_tokens`, `tool_use`, `stop_sequence`). stop_sequence: anyOf: - type: string - type: 'null' title: Stop Sequence description: >- Matched stop string when `stop_reason` is `stop_sequence`; otherwise null. usage: $ref: '#/components/schemas/MessagesUsage' description: Token usage details for this response. model: anyOf: - type: string - type: 'null' title: Model description: Model used to generate the response. additionalProperties: true type: object required: - id - type - role - content - usage title: ServerlessMessagesSuccess ServerlessModelListSuccess: properties: data: items: $ref: '#/components/schemas/ModelCatalogResponseItem' type: array title: Data type: object required: - data title: ServerlessModelListSuccess description: '`/models` API response (for OpenRouters).' ServerlessPriceUnitType: type: string enum: - TOKEN - SECOND - AUDIO_MINUTE title: ServerlessPriceUnitType description: Serverless price unit type. ServerlessTokenizationBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct prompt: type: string title: Prompt description: Input text prompt to tokenize. examples: - What is generative AI? type: object required: - model - prompt title: ServerlessTokenizationBody example: model: meta-llama/Llama-3.1-8B-Instruct prompt: What is generative AI? ServerlessTokenizationSuccess: properties: tokens: items: type: integer type: array title: Tokens description: A list of token IDs. type: object required: - tokens title: ServerlessTokenizationSuccess ServerlessToolAssistedChatCompleteSuccess: $ref: '#/components/schemas/ChatResult' title: ServerlessToolAssistedChatCompleteSuccess ServerlessToolAssistedChatCompletionBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: type: boolean title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: false stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/ToolAssistedChatTool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. A maximum of 128 functions is supported. Use this to provide a list of functions the model may generate JSON inputs for. For more detailed information about each tool, please refer [here](https://friendli.ai/docs/guides/model-apis/tool-assisted-api#built-in-tools). resume_generation: type: boolean title: Resume Generation description: >- Enable to continue text generation even after an error occurs during a tool call. Note that enabling this option may use more tokens, as the system generates additional content to handle errors gracefully. However, if the system fails more than 8 times, the generation will stop regardless. ***Tip*** This is useful in scenarios where you want to maintain text generation flow despite errors, such as when generating long-form content. The user will not be interrupted by tool call issues, ensuring a smoother experience. type: object required: - model - messages title: ServerlessToolAssistedChatCompletionBody example: messages: - content: What is 3 + 6? role: user model: meta-llama/Llama-3.1-8B-Instruct tools: - type: math:calculator ServerlessToolAssistedChatCompletionStreamBody: properties: model: type: string title: Model description: >- Code of the model to use. See [available model list](https://friendli.ai/docs/guides/model-apis/pricing#billing-methods). examples: - meta-llama/Llama-3.1-8B-Instruct messages: items: $ref: '#/components/schemas/Message' type: array title: Messages description: A list of messages comprising the conversation so far. examples: - - content: You are a helpful assistant. role: system - content: Hello! role: user chat_template_kwargs: anyOf: - additionalProperties: true type: object - type: 'null' title: Chat Template Kwargs description: >- Additional keyword arguments supplied to the template renderer. These parameters will be available for use within the chat template. eos_token: anyOf: - items: type: integer type: array - type: 'null' title: Eos Token description: A list of endpoint sentence tokens. frequency_penalty: anyOf: - type: number - type: 'null' title: Frequency Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim. logit_bias: anyOf: - additionalProperties: true type: object - type: 'null' title: Logit Bias description: >- Accepts a JSON object that maps tokens to an associated bias value. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model. logprobs: anyOf: - type: boolean - type: 'null' title: Logprobs description: Whether to return log probabilities of the output tokens or not. max_tokens: anyOf: - type: integer - type: 'null' title: Max Tokens description: >- The maximum number of tokens to generate. For decoder-only models like GPT, the length of your input tokens plus `max_tokens` should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3). For encoder-decoder models like T5 or BlenderBot, `max_tokens` should not exceed the model's maximum output length. This is similar to Hugging Face's [`max_new_tokens`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.max_new_tokens) argument. examples: - 200 min_p: anyOf: - type: number - type: 'null' title: Min P description: >- A scaling factor used to determine the minimum token probability threshold. This threshold is calculated as `min_p` multiplied by the probability of the most likely token. Tokens with probabilities below this scaled threshold are excluded from sampling. Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values result in stricter filtering, while lower values allow for greater diversity. The default value of 0.0 disables filtering, allowing all tokens to be considered for sampling. 'n': anyOf: - type: integer - type: 'null' title: 'N' description: >- The number of independently generated results for the prompt. Defaults to 1. This is similar to Hugging Face's [`num_return_sequences`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.GenerationConfig.num_return_sequences) argument. parallel_tool_calls: anyOf: - type: boolean - type: 'null' title: Parallel Tool Calls description: Whether to enable parallel function calling. presence_penalty: anyOf: - type: number - type: 'null' title: Presence Penalty description: >- Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. repetition_penalty: anyOf: - type: number - type: 'null' title: Repetition Penalty description: >- Penalizes tokens that have already appeared in the generated result (plus the input tokens for decoder-only models). Should be positive value (1.0 means no penalty). See [keskar et al., 2019](https://arxiv.org/abs/1909.05858) for more details. This is similar to Hugging Face's [`repetition_penalty`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.repetition_penalty) argument. reasoning_effort: anyOf: - type: string enum: - low - medium - high - type: 'null' title: Reasoning Effort description: >- Determines the amount of reasoning effort the model applies when generating a response. Higher values may produce more detailed and thoughtful outputs, but can increase response time. This parameter is only effective for reasoning models. reasoning_budget: anyOf: - type: integer - type: 'null' title: Reasoning Budget description: >- Specifies a positive integer that defines a limit on the number of tokens used for internal reasoning tokens. This parameter is only effective for reasoning models. seed: anyOf: - items: type: integer type: array - type: integer - type: 'null' title: Seed description: >- Seed to control random procedure. If nothing is given, random seed is used for sampling, and return the seed along with the generated result. When using the `n` argument, you can pass a list of seed values to control all of the independent generations. stop: anyOf: - items: type: string type: array - type: 'null' title: Stop description: >- When one of the stop phrases appears in the generation result, the API will stop generation. The stop phrases are excluded from the result. Defaults to empty list. stream: type: boolean title: Stream description: >- Whether to stream the generation result. When set to `true`, each token will be sent as [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format) once generated. default: true stream_options: anyOf: - $ref: '#/components/schemas/StreamOptions' - type: 'null' description: |- Options related to stream. It can only be used when `stream: true`. parse_reasoning: anyOf: - type: boolean - type: 'null' title: Parse Reasoning description: >- Parses model reasoning into `reasoning_content` while keeping the answer in `content`. Default value may vary between endpoints. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). include_reasoning: anyOf: - type: boolean - type: 'null' title: Include Reasoning description: >- When `parse_reasoning=true`, include parsed reasoning (`reasoning_content`). Defaults to true. For more detailed information, please refer [here](https://friendli.ai/docs/guides/reasoning#reasoning-parsing-with-friendli). temperature: anyOf: - type: number - type: 'null' title: Temperature description: >- Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., `top_k = 1`) sampling. Defaults to 1.0. This is similar to Hugging Face's [`temperature`](https://huggingface.co/docs/transformers/v4.26.0/en/main_classes/text_generation#transformers.generationconfig.temperature) argument. tool_choice: anyOf: - $ref: '#/components/schemas/ChatCompleteBodyToolChoice' - type: string title: Tool Choice description: > Determines the tool calling behavior of the model. When set to `none`, the model will bypass tool execution and generate a response directly. In `auto` mode (the default), the model dynamically decides whether to call a tool or respond with a message. Alternatively, setting `required` ensures that the model invokes at least one tool before responding to the user. You can also specify a particular tool by `{"type": "function", "function": {"name": "my_function"}}`. top_k: anyOf: - type: integer - type: 'null' title: Top K description: >- Limits sampling to the top k tokens with the highest probabilities. Values range from 0 (no filtering) to the model's vocabulary size (inclusive). The default value of 0 applies no filtering, allowing all tokens. top_logprobs: anyOf: - type: integer - type: 'null' title: Top Logprobs description: >- The number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used. top_p: anyOf: - type: number - type: 'null' title: Top P description: >- Keeps only the smallest set of tokens whose cumulative probabilities reach `top_p` or higher. Values range from 0.0 (exclusive) to 1.0 (inclusive). The default value of 1.0 includes all tokens, allowing maximum diversity. xtc_threshold: anyOf: - type: number - type: 'null' title: Xtc Threshold description: >- A probability threshold used to identify “top choice” tokens for exclusion in XTC (Exclude Top Choices) sampling. Tokens with probabilities at or above this threshold are considered viable candidates, and all but the least likely viable token are excluded from sampling. This option reduces the dominance of highly probable tokens while preserving some diversity by keeping the least confident “top choice.” Values range from 0.0 (inclusive) to 1.0 (inclusive). Higher values make the filtering more selective by requiring higher probabilities to trigger exclusion, while lower values apply filtering more broadly. The default value of 0.0 disables XTC filtering entirely. xtc_probability: anyOf: - type: number - type: 'null' title: Xtc Probability description: >- The probability that XTC (Exclude Top Choices) filtering will be applied for each sampling decision. When XTC is triggered, high-probability tokens above the `xtc_threshold` are excluded except for the least likely viable token. This stochastic activation allows for a balance between standard sampling and creativity-boosting exclusion filtering. Values range from 0.0 (inclusive) to 1.0 (inclusive), where 0.0 means XTC is never applied, 1.0 means XTC is always applied when viable tokens exist, and intermediate values provide probabilistic activation. The default value of 0.0 disables XTC filtering. tools: anyOf: - items: $ref: '#/components/schemas/ToolAssistedChatTool' type: array - type: 'null' title: Tools description: >- A list of tools the model may call. A maximum of 128 functions is supported. Use this to provide a list of functions the model may generate JSON inputs for. For more detailed information about each tool, please refer [here](https://friendli.ai/docs/guides/model-apis/tool-assisted-api#built-in-tools). resume_generation: type: boolean title: Resume Generation description: >- Enable to continue text generation even after an error occurs during a tool call. Note that enabling this option may use more tokens, as the system generates additional content to handle errors gracefully. However, if the system fails more than 8 times, the generation will stop regardless. ***Tip*** This is useful in scenarios where you want to maintain text generation flow despite errors, such as when generating long-form content. The user will not be interrupted by tool call issues, ensuring a smoother experience. type: object required: - model - messages title: ServerlessToolAssistedChatCompletionStreamBody example: messages: - content: What is 3 + 6? role: user model: meta-llama/Llama-3.1-8B-Instruct tools: - type: math:calculator ServerlessToolAssistedChatCompletionStreamSuccess: anyOf: - $ref: '#/components/schemas/StreamedToolAssistedChatToken' - $ref: '#/components/schemas/StreamedToolAssistedChatToolStatus' title: ServerlessToolAssistedChatCompletionStreamSuccess description: A server-sent event containing chat completions content. SplitInfo: properties: id: type: string title: Id description: ID of the split. name: type: string title: Name description: Name of the split. createdAt: type: integer title: Createdat description: Unix timestamp (in seconds) of when the split was created. updatedAt: type: integer title: Updatedat description: Unix timestamp (in seconds) of when the split was last modified. type: object required: - id - name - createdAt - updatedAt title: SplitInfo description: Split info. StreamOptions: properties: include_usage: anyOf: - type: boolean - type: 'null' title: Include Usage description: > When set to `true`, the number of tokens used will be included at the end of the stream result in the form of `"usage": {"completion_tokens": number, "prompt_tokens": number, "total_tokens": number}`. type: object title: StreamOptions StreamedAudioTranscriptionData: properties: type: type: string enum: - transcript.text.delta - transcript.text.done title: Type description: >- The event type, which is `transcript.text.delta` or `transcript.text.done`. delta: anyOf: - type: string - type: 'null' title: Delta description: The incremental transcript text. text: anyOf: - type: string - type: 'null' title: Text description: The transcribed text. usage: anyOf: - $ref: '#/components/schemas/AudioTranscriptionUsage' - type: 'null' type: object required: - type title: StreamedAudioTranscriptionData StreamedAudioTranscriptionResult: properties: data: $ref: '#/components/schemas/StreamedAudioTranscriptionData' type: object required: - data title: StreamedAudioTranscriptionResult StreamedChatChoice: properties: index: type: integer title: Index description: The index of the choice in the list of generated choices. examples: - 0 logprobs: anyOf: - $ref: '#/components/schemas/ChatLogprobs' - type: 'null' delta: $ref: '#/components/schemas/StreamedChatChoiceDelta' finish_reason: anyOf: - type: string enum: - stop - length - tool_calls - 'null' - type: 'null' title: Finish Reason description: >- Termination condition of the generation. `stop` means the API returned the full chat completions generated by the model without running into any limits. `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length. `tool_calls` means the API has generated tool calls. type: object required: - index - delta title: StreamedChatChoice StreamedChatChoiceDelta: properties: content: anyOf: - type: string - type: 'null' title: Content description: The contents of the assistant message. role: anyOf: - type: string - type: 'null' title: Role description: Role of the generated message author, in this case `assistant`. tool_calls: anyOf: - items: $ref: '#/components/schemas/StreamedToolCallResult' type: array - type: 'null' title: Tool Calls type: object title: StreamedChatChoiceDelta StreamedChatData: properties: id: type: string title: Id description: A unique ID of the chat completion. usage: anyOf: - $ref: '#/components/schemas/ChatUsage' - type: 'null' object: type: string const: chat.completion.chunk title: Object description: The object type, which is always set to `chat.completion.chunk`. choices: items: $ref: '#/components/schemas/StreamedChatChoice' type: array title: Choices created: type: integer title: Created description: The Unix timestamp (in seconds) for when the token sampled. model: anyOf: - type: string - type: 'null' title: Model description: >- The model to generate the completion. For dedicated endpoints, it returns the endpoint id. type: object required: - id - object - choices - created title: StreamedChatData StreamedChatResult: properties: data: $ref: '#/components/schemas/StreamedChatData' type: object required: - data title: StreamedChatResult StreamedCompletionData: properties: id: type: string title: Id description: A unique ID of the completion. choices: items: $ref: '#/components/schemas/StreamedCompletionsChoice' type: array title: Choices usage: anyOf: - $ref: '#/components/schemas/TextUsage' - type: 'null' object: type: string const: text_completion title: Object description: The object type, which is always set to `text_completion`. created: type: integer title: Created description: The Unix timestamp (in seconds) for when the token sampled. model: anyOf: - type: string - type: 'null' title: Model description: >- The model to generate the completion. For dedicated endpoints, it returns the endpoint id. type: object required: - id - choices - object - created title: StreamedCompletionData StreamedCompletionsChoice: properties: index: type: integer title: Index description: The index of the choice in the list of generated choices. examples: - 0 text: type: string title: Text description: Generated text output. examples: - This is indeed a test logprobs: anyOf: - $ref: '#/components/schemas/CompletionsLogprobs' - type: 'null' finish_reason: anyOf: - type: string enum: - stop - length - 'null' - type: 'null' title: Finish Reason description: >- Termination condition of the generation. `stop` means the API returned the full completions generated by the model without running into any limits. `length` means the generation exceeded `max_tokens` or the conversation exceeded the max context length. token: anyOf: - type: integer - type: 'null' title: Token description: The token. type: object required: - index - text title: StreamedCompletionsChoice StreamedCompletionsResult: properties: data: $ref: '#/components/schemas/StreamedCompletionData' type: object required: - data title: StreamedCompletionsResult StreamedFunctionResult: properties: arguments: type: string title: Arguments description: >- The arguments for calling the function, generated by the model in JSON format. Ensure to validate these arguments in your code before invoking the function since the model may not always produce valid JSON. name: anyOf: - type: string - type: 'null' title: Name description: The name of the function to call. type: object required: - arguments title: StreamedFunctionResult StreamedToolAssistedChatToken: properties: data: $ref: '#/components/schemas/StreamedChatData' type: object required: - data title: StreamedToolAssistedChatToken StreamedToolAssistedChatToolStatus: properties: event: type: string const: tool_status title: Event data: $ref: '#/components/schemas/ToolStatusData' type: object required: - event - data title: StreamedToolAssistedChatToolStatus StreamedToolCallResult: properties: type: type: string const: function title: Type description: The type of the tool. id: anyOf: - type: string - type: 'null' title: Id description: The ID of the tool call. index: type: integer title: Index description: The index of tool call being generated. function: $ref: '#/components/schemas/StreamedFunctionResult' type: object required: - type - index - function title: StreamedToolCallResult SystemMessage: properties: role: type: string const: system title: Role description: The role of the messages author. content: type: string title: Content description: The content of system message. name: anyOf: - type: string - type: 'null' title: Name description: >- The name for the participant to distinguish between participants with the same role. type: object required: - role - content title: SystemMessage TextClassificationResult: properties: data: items: $ref: '#/components/schemas/BaseClassificationData' type: array title: Data object: type: string const: list title: Object description: The object type, which is always set to `list`. usage: $ref: '#/components/schemas/TextClassificationUsage' type: object required: - data - object - usage title: TextClassificationResult TextClassificationUsage: properties: prompt_tokens: type: integer title: Prompt Tokens description: Number of tokens in the input text. examples: - 10 total_tokens: type: integer title: Total Tokens description: Total number of tokens used in the request. examples: - 10 type: object required: - prompt_tokens - total_tokens title: TextClassificationUsage TextContent: properties: type: type: string const: text title: Type description: The type of the message content. text: type: string title: Text description: The text content of the message. type: object required: - type - text title: TextContent TextUsage: properties: prompt_tokens: type: integer title: Prompt Tokens description: Number of tokens in the prompt. examples: - 5 completion_tokens: type: integer title: Completion Tokens description: Number of tokens in the generated completions. examples: - 7 total_tokens: type: integer title: Total Tokens description: >- Total number of tokens used in the request (`prompt_tokens` + `completion_tokens`). examples: - 12 prompt_tokens_details: anyOf: - $ref: '#/components/schemas/PromptTokensDetails' - type: 'null' description: Breakdown of tokens used in the prompt. type: object required: - prompt_tokens - completion_tokens - total_tokens title: TextUsage TokenSequence: properties: tokens: anyOf: - items: type: integer type: array - type: 'null' title: Tokens description: A List of token IDs. type: object title: TokenSequence Tool: properties: type: type: string const: function title: Type description: The type of the tool. Currently, only `function` is supported. function: $ref: '#/components/schemas/Function' type: object required: - type - function title: Tool ToolAssistedChatTool: oneOf: - $ref: '#/components/schemas/Tool' title: Function - $ref: '#/components/schemas/FriendliBuiltInTool' title: Friendli - $ref: '#/components/schemas/IntegratedBuiltInTool' title: Integrated - $ref: '#/components/schemas/FileBuiltInTool' title: File discriminator: propertyName: type mapping: code:python-interpreter: '#/components/schemas/FriendliBuiltInTool' file:text: '#/components/schemas/FileBuiltInTool' function: '#/components/schemas/Tool' linkup:search: '#/components/schemas/IntegratedBuiltInTool' math:calculator: '#/components/schemas/FriendliBuiltInTool' math:calendar: '#/components/schemas/FriendliBuiltInTool' math:statistics: '#/components/schemas/FriendliBuiltInTool' web:search: '#/components/schemas/FriendliBuiltInTool' web:url: '#/components/schemas/FriendliBuiltInTool' ToolCallResult: properties: type: type: string const: function title: Type description: The type of the tool. id: type: string title: Id description: The ID of the tool call. function: $ref: '#/components/schemas/FunctionResult' type: object required: - type - id - function title: ToolCallResult ToolMessage: properties: role: type: string const: tool title: Role description: The role of the messages author. content: type: string title: Content description: >- The content of tool message that contains the result of tool calling. tool_call_id: type: string title: Tool Call Id description: The ID of tool call corresponding to this message. name: anyOf: - type: string - type: 'null' title: Name description: An optional name of the tool call corresponding to this message. type: object required: - role - content - tool_call_id title: ToolMessage ToolStatusData: properties: tool_call_id: type: string title: Tool Call Id description: The ID of the tool call. name: type: string enum: - linkup:search - math:calculator - math:statistics - math:calendar - web:search - web:url - code:python-interpreter - file:text title: Name description: The name of the built-in tool. status: type: string enum: - STARTED - UPDATING - ENDED - ERRORED title: Status description: Indicates the current execution status of the tool. parameters: items: $ref: '#/components/schemas/ToolStatusParameter' type: array title: Parameters result: anyOf: - type: string - type: 'null' title: Result description: The output from the tool's execution. files: anyOf: - items: $ref: '#/components/schemas/ToolStatusFile' type: array - type: 'null' title: Files message: anyOf: - type: string - type: 'null' title: Message description: Message generated by the tool's execution. error: anyOf: - $ref: '#/components/schemas/ToolStatusError' - type: 'null' timestamp: type: number title: Timestamp description: The Unix timestamp (in seconds) for when the event occurred. type: object required: - tool_call_id - name - status - parameters - timestamp title: ToolStatusData ToolStatusError: properties: type: type: string title: Type description: The type of error encountered during the tool's execution. msg: type: string title: Msg description: The message of error. type: object required: - type - msg title: ToolStatusError ToolStatusFile: properties: name: type: string title: Name description: The name of the file generated by the tool's execution. url: type: string title: Url description: URL of the file generated by the tool's execution. type: object required: - name - url title: ToolStatusFile ToolStatusParameter: properties: name: type: string title: Name description: The name of the tool's function parameter. value: type: string title: Value description: The value of the tool's function parameter. type: object required: - name - value title: ToolStatusParameter UrlImageItem: properties: seed: type: integer title: Seed description: The seed used during image generation. response_format: type: string enum: - url - raw - png - jpeg - jpg title: Response Format description: >- The format of the generated image. One of `url(default)`, `raw`, `png`, `jpeg`, and `jpg`. url: type: string title: Url description: The URL of the generated image. type: object required: - seed - response_format - url title: UrlImageItem UserMessage: properties: role: type: string const: user title: Role description: The role of the message's author. content: anyOf: - type: string description: >- The content of user message, which is plain text. For **multi-modal format**, use `object[]` type. Support for non-text input is currently in **Beta**. - items: $ref: '#/components/schemas/UserMessageContentMultiModal' type: array description: >- The content of user message. **Multi-modal format** can handle not just text, but also audio, image, and video content, allowing for more complex message structures. Support for non-text input is currently in **Beta**. title: Content name: anyOf: - type: string - type: 'null' title: Name description: >- The name for the participant to distinguish between participants with the same role. type: object required: - role - content title: UserMessage UserMessageContentMultiModal: oneOf: - $ref: '#/components/schemas/TextContent' title: Text - $ref: '#/components/schemas/AudioContent' title: Audio - $ref: '#/components/schemas/ImageContent' title: Image - $ref: '#/components/schemas/VideoContent' title: Video discriminator: propertyName: type mapping: audio_url: '#/components/schemas/AudioContent' image_url: '#/components/schemas/ImageContent' text: '#/components/schemas/TextContent' video_url: '#/components/schemas/VideoContent' ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type type: object required: - loc - msg - type title: ValidationError VersionInfo: properties: id: type: string title: Id description: ID of the version. comment: type: string title: Comment description: Comment for the version. createdAt: type: integer title: Createdat description: Unix timestamp (in seconds) of when the version was created. type: object required: - id - comment - createdAt title: VersionInfo description: Version info. VideoContent: properties: type: type: string const: video_url title: Type description: The type of the message content. video_url: $ref: '#/components/schemas/VideoData' description: The video URL data. type: object required: - type - video_url title: VideoContent VideoData: properties: url: type: string title: Url description: The URL of the video. type: object required: - url title: VideoData securitySchemes: token: type: http description: >- When using Friendli Suite API for inference requests, you need to provide a **Friendli Token** for authentication and authorization purposes. For more detailed information, please refer [here](https://friendli.ai/docs/openapi/introduction#authentication). scheme: bearer tags: - name: Serverless.Chat - name: Serverless.ToolAssistedChat - name: Serverless.Messages - name: Serverless.ChatRender - name: Serverless.Completions - name: Serverless.Token - name: Serverless.Audio - name: Serverless.Model - name: Serverless.Knowledge - name: Dedicated.Chat - name: Dedicated.Messages - name: Dedicated.ChatRender - name: Dedicated.Completions - name: Dedicated.Embeddings - name: Dedicated.TextClassification - name: Dedicated.Token - name: Dedicated.Image - name: Dedicated.Audio - name: Dedicated.Endpoint - name: Container.Chat - name: Container.Messages - name: Container.Completions - name: Container.TextClassification - name: Container.Token - name: Container.Image - name: Container.Audio - name: Dataset - name: File x-speakeasy-retries: strategy: backoff backoff: initialInterval: 500 maxInterval: 60000 maxElapsedTime: 3600000 exponent: 1.5 statusCodes: - 429 - 500 - 502 - 503 - 504 retryConnectionErrors: true