Structured Output Requires More Than Guided Generation

Structured output is a response from an LLM that follows a predefined schema, such as a JSON schema. This allows downstream applications to parse and consume model outputs reliably. For example, when using LLMs for sentiment analysis, you would want a model to return a JSON response such as {“sentiment”: “positive”} rather than “The sentiment is positive because the review expresses satisfaction with the product.”

The chat completions API supports structured output via the response_format field, as shown below, which allows you to specify the desired output schema.

"response_format": {
  "type": "json_schema",
  "json_schema": {
    "schema": {
      "type": "object",
      "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "negative"]}
      },
      ...
    }
  }
}

Internally, the inference engine enforces the schema using constrained decoding. At each decoding step, it masks tokens that would produce a schema-invalid output. However, such schema enforcement during generation alone does not guarantee correct output. Consider the example below.

Figure 1 shows testing Gemma 4 31B with a complex JSON schema. Even though the response_format field was specified, the model repeatedly emits newline characters. Because a whitespace character was still valid under the JSON grammar, constrained decoding had no basis for masking those tokens. This illustrates an important property of structured output: the inference engine restricts generation to schema-valid continuations, but the model still determines which valid continuation to produce. To see why, it helps to examine how constrained decoding operates internally.

How does constrained decoding work?

At each decoding step, the model assigns a probability to every possible next token. The inference engine then masks tokens that would make the output incompatible with the requested schema. It renormalizes the remaining probabilities and samples the next token from this constrained distribution.

For example, suppose the model has already generated ‘{“rating”:’ and the schema requires “rating” to be an integer. The model may assign high probability to candidate tokens such as “5”, “five”, “good”, and “{“. The inference engine then masks any token that would make the output incompatible with the schema, leaving only valid numeric tokens to sample from. The model then chooses among those remaining candidates. This is how constrained decoding prevents generating tokens that violate the schema. However, for structured output to work as intended, the model must assign sufficient probability to the right valid tokens.

Prompting the Model to Generate the Right Tokens

Constrained decoding enforces the schema, but it does not determine which token the model should generate next. When the prompt provides insufficient guidance about the expected output, the model may favor valid but unintended continuations, such as repeated whitespace. In the previous example, after generating {, the model did not receive a strong signal to produce the expected field names. Constrained decoding did not mask the whitespace because it was still valid under the JSON grammar. The model therefore continued generating whitespace until it reached the token limit.

Including a concrete example in the prompt can prevent this behavior, as it provides a clear signal of the intended structure and content. For example, the prompt in Figure 1 can be revised as follows to resolve the issue. With an example JSON value in the prompt, the model can produce output that matches it, unlike in the previous case.

You are a trip planner agent.

Analyze the user's request and respond in JSON.

## Example output
{
  "travel_plan": {
    "bookings": [
      {
        "task_number": 3,
        "service_id": "accommodation_agent",
        "rating": 2,
        "instruction": "..."
      }
    ]
  }
}

With the example included, the model produces the correct output below.

{
  "travel_plan": {
    "bookings": [
      {
        "task_number": 1,
        "service_id": "accommodation_agent",
        "rating": 4,
        "instruction": "Find beachfront accommodation..."
      },
    ]
  }
}

Reliable structured output depends on both the prompt steering the model toward the right set of tokens and the inference engine preventing invalid ones through constrained decoding. A generic instruction to “write in JSON” does not provide sufficient guidance. The prompt must also clarify what information the model should generate and how that information should map to the schema.

FriendliAI supports structured output through constrained decoding implemented directly in its inference engine. This optimized implementation delivers higher throughput than unconstrained generation while efficiently managing the additional processing required at each decoding step.

Performance Impact of Structured Output

To evaluate the performance impact of constrained decoding, we compared two requests that produced the same JSON structure. One request used response_format with a JSON schema, while the other relied on unconstrained generation without schema enforcement. All experiments were conducted on FriendliAI Endpoints using MiniMaxAI/MiniMax-M2.7. Although schema enforcement adds work at every decoding step, requests using response_format achieved substantially higher throughput.

To understand this behavior, we repeated the experiment with speculative decoding disabled. The throughput gain from response_format dropped from 21% to 3%, indicating that most of the improvement comes from its interaction with speculative decoding. By narrowing the set of valid tokens, constrained decoding increases draft-token acceptance, making speculative decoding more effective. On FriendliAI Endpoints, this combination delivers schema-compliant output with higher throughput.

Building Reliable Structured Output

Structured output is foundational to agentic AI, enabling agents to evaluate results, invoke tools, and coordinate multi-step workflows through predictable, machine-readable responses. Producing those responses reliably requires both the model and the inference engine: the prompt guides the model toward the intended fields and values, while constrained decoding prevents schema-invalid continuations.

At FriendliAI, we’re building an inference stack optimized for agentic AI. By combining reliable structured output with high-performance inference, we help developers build production-ready AI agents with confidence.

Get Started with Structured Output on FriendliAI

Ready to build agents with reliable, schema-compliant responses?

• Try Structured Outputs on Friendli Model APIs — Test response_format with open-weight models like MiniMax-M2.7, serverless and pay-per-token
• Read the Structured Outputs docs — Step-by-step guide covering JSON schema, regex patterns, and supported keywords
• Talk to our team — Building agentic workflows at scale? We'll help you design prompts and schemas that produce reliable structured output in production