Skip to main content
Friendli offers structured outputs capability with two core guarantees:
  • Model-agnostic: Supported on all chat‑capable models on Friendli.
  • High schema fidelity: Generates outputs that reliably conform to your provided schemas.

​
What is Structured Outputs?

Structured Outputs ensures LLMs return predictable, machine‑readable results (e.g., JSON) instead of free‑form text. This is essential for workflows that require validation or downstream automation.

​
Structured Outputs with Friendli

  • Schema‑aligned generation: High‑accuracy adherence to your JSON Schema.
  • Flexible modes: Choose strict or loose JSON mode, or apply regex constraints as needed.
  • OpenAI compatible: Use standard response_format options with OpenAI SDKs.

​
Structured Outputs Parameters

TypeDescriptionName at OpenAI
json_schemaThe model returns a JSON object that conforms to the given schema.Structured Outputs
json_objectThe model can return any JSON object.JSON mode
regexThe model returns a string that conforms to the given regex schema.N/A

​
Supported JSON schemas

We support all seven standard JSON schema types (null, boolean, number, integer, string, object, array). The supported JSON schema keywords are listed below.
Using unsupported or unexpected JSON schema keywords may result in them being ignored, triggering an error, or causing undefined behavior.

​
Type-specific keywords

  • integer
    • exclusiveMinimum, exclusiveMaximum, minimum, maximum (Note: these are not supported in number)
  • string
    • pattern
    • format
      • Supported values: uuid, date-time, date, time
  • object
    • properties
    • additionalProperties is ignored, and is always set to False.
    • required: We support both required and optional properties, but have these limitations:
      • The sequence of the properties is fixed.
      • The first property should be required. If not, the first required property is moved to the first.
  • array
    • items
    • minItems: We support only 0 or 1 for minItems.

​
Constant values and enumerated values

const and enum only support constant values of null, boolean, number, and string.

​
Schema composition

We support only anyOf for schema composition.

​
Referencing subschemas

We only support referencing ($ref) to “internal” subschemas. These subschemas must be defined within $defs, and the value of $ref must be a valid URI pointing to a subschema.

​
Annotations

JSON schema annotations such as title or description are accepted but ignored.

​
Simple Example

  • json_schema
  • json_object
  • regex
This example provides a step-by-step guide of how to create a structured output response in JSON format. We use Python and the pydantic library to define a schema for the output in this example.
1

Define Object Schema

Define a schema that contains information about a dish.
from pydantic import BaseModel

class Result(BaseModel):
    dish: str
    cuisine: str
    calories: int
2

Asking the model for structured output

Call structured output and use schema to structure the response.
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.getenv("FRIENDLI_TOKEN"),
)

completion = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Suggest a popular Italian dish in JSON format.",
        },
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "schema": Result.model_json_schema(),
        }
    }
)
3

Using structured output results

You can use the output in the following way.
response = completion.choices[0].message.content
print(response)
The code output result is as follows.
Result:
{
    "dish": "Spaghetti Bolognese",
    "cuisine": "Italian",
    "calories": 540
}

​
Advanced Examples

For more advanced use cases, see our blog: Structured Output for LLM Agents.