Large language models (LLMs) excel at creative text generation, but we often face a case where we need LLM outputs to be more structured. This is where our exciting new “structured output” feature comes in.

Structured Outputs is also available in Friendli Dedicated Endpoints and Friendli Container.

For more advanced use cases of our Structured Outputs feature, check out our detailed blog post on Structured Output for LLM Agents.

​
Structured response modes

TypeDescriptionName at OpenAI
json_schemaThe model returns a JSON object that conforms to the given schema.Structured Outputs
json_objectThe model can return any JSON object.JSON mode
regexThe model returns a string that conforms to the given regex schema.N/A

​
How to use

This guide provides a step-by-step example of how to create a structured output response in JSON form.
In this example, we will use Python and the pydantic library to define a schema for the output.

1

Define Object Schema

Define a schema that contains information about a dish.

from pydantic import BaseModel

class Result(BaseModel):
    dish: str
    cuisine: str
    calories: int
2

Asking the model for structured output

Call structured output and use schema to structure the response.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.friendli.ai/serverless/v1",
    api_key=os.getenv("FRIENDLI_TOKEN"),
)

completion = client.chat.completions.create(
    model="meta-llama-3.1-8b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Suggest a popular Italian dish in JSON format.",
        },
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "schema": Result.model_json_schema(),
        }
    }
)
3

Using structured output results

You can use the output in the following way.

response = completion.choices[0].message.content
print(response)

The code output result is as follows.

Result:
{
    "dish": "Spaghetti Bolognese",
    "cuisine": "Italian",
    "calories": 540
}

​
Supported JSON schemas

We ensure super-fast schema-guided generation by disabling JSON schema features that cause computation inefficiencies. We support all seven standard JSON schema types (null, boolean, number, integer, string, object, array), and the supported JSON schema keywords are listed below.

Using unsupported or unexpected JSON schema keywords may result in them being ignored, triggering an error, or causing undefined behavior.

​
Type-specific keywords

  • integer
    • exclusiveMinimum, exclusiveMaximum, minimum, maximum (Note: these are not supported in number)
  • string
    • pattern
    • format
      • Supported values: uuid, date-time, date, time
  • object
    • properties
    • additionalProperties is ignored, and is always set to False.
    • required: We support both required and optional properties, but have these limitations:
      • The sequence of the properties is fixed.
      • The first property should be required. If not, the first required property is moved to the first.
  • array
    • items
    • minItems: We support only 0 or 1 for minItems.

​
Constant values and enumerated values

const and enum only support constant values of null, boolean, number, and string.

​
Schema composition

We support only anyOf for schema composition.

​
Referencing subschemas

We only support referencing ($ref) to “internal” subschemas. These subschemas must be defined within $defs, and the value of $ref must be a valid URI pointing to a subschema. Please refer here for more details.

​
Annotation

JSON schema annotations such as title, $comments or description are accepted but ignored.

Was this page helpful?

Function CallingOpenAI Compatibility