Structured Output: How Engines Force Valid JSON, Token by Token

Your application asks a model for JSON and gets almost-JSON: a Markdown fence around it, a field name copied from the instructions, a priority spelled "High" when your code expects "high". Prompting reduces the problem; it does not remove it. Serving engines can remove it, by changing which tokens the model is allowed to pick at every step. This piece shows how, what it costs, and the one failure it cannot prevent.

Asking is not the same as forcing

A model produces text one token at a time, each picked from a probability distribution over its whole vocabulary (about 150,000 tokens for Qwen3). Writing "answer with JSON matching this schema" in the prompt shifts that distribution toward valid JSON. It does not take the other tokens away.

How much that matters depends on the model. We gave Qwen3 the same task 200 times: turn a short bug report into a ticket with five fields, the schema written into the prompt. Then we ran the same 200 requests with the schema also passed as a constraint:

The 8-billion-parameter model never failed, even at a high sampling temperature. The 0.6-billion one almost always did: it wrapped the answer in a Markdown code fence, or copied the schema's own "type": "object" into its answer. With the constraint, both produced valid tickets, apart from one or two answers that never finished (more on those below). Small models, high temperatures, and long or unusual schemas are where forcing pays; on a strong model with a clear prompt, it is insurance.

Forcing: a mask on every step

A structured output engine turns the schema into a grammar, a set of rules for which characters may come next at each point of the document. Then, at every step, it does three things:

1. Ask the grammar what may come next. After {"priority": ", only the start of low, medium, high, or urgent is allowed.
2. Mask the vocabulary. Every token that would break the grammar gets a score of minus infinity, so its probability becomes zero.
3. Sample as usual among the tokens left, then advance the grammar with the token picked.

The model itself is unchanged. It runs the same forward pass; only its choices are narrowed. When the model's favorite token is allowed, which with a good prompt is almost always, the constraint changes nothing at all.

Try it on real answers from our run, including the failures above:

Four things to see:

1. "Markdown fence": the grammar blocks the very first character. A JSON document must start with {, so with the constraint the model could never have written the fence.
2. "Schema copied": blocked at the key "type", because the schema allows only its five fields.
3. "Rambling in a string": never blocked. Inside a free-text string almost any character is allowed, and this model, constrained, wandered until it hit its token limit. The answer is valid so far, and never finished.
4. Edit the valid answer: change "high" to "High" or 1200 to "1200" and watch where it stops.

The widget checks characters. Real engines work on tokens, which is harder: a token can be a single character or a whole chunk like ": or "title, and one token can cross a boundary the grammar cares about. The engine has to decide, for each grammar state, which of the vocabulary's tokens are acceptable as a whole. Doing that for 150,000 tokens at every step would be slow, so XGrammar, the default backend in vLLM and SGLang, splits the vocabulary in two: tokens whose validity depends only on the current grammar position, prechecked, and the others, which depend on the deeper parsing stack and are checked at run time.

Inside vLLM

In vLLM 0.31.0 the work is split between the CPU and the GPU, and most of it is on the CPU:

• Choosing a backend. The default, auto, picks XGrammar, and falls back to llguidance (guidance) or Outlines when XGrammar cannot handle a schema feature (source).
• Compiling, once per schema. The grammar is compiled on a thread pool, without blocking the engine (source), and XGrammar keeps compiled grammars in a cache (source): the second request with the same schema skips compilation.
• Filling the mask, every step. For each constrained request, the grammar writes a bitmask over the vocabulary, one bit per token, on CPU threads, in parallel above 128 requests (source).
• Applying it, on the GPU. The bitmask is applied to the logits in place, just before sampling (source).
• Reasoning models. With a reasoning parser configured, the constraint starts only after the model's thinking section ends, unless you ask otherwise with enable_in_reasoning (source). The model can think freely, then answer in the required format.

SGLang 0.5.21 does the same with --grammar-backend (XGrammar by default; llguidance and Outlines also available).

Using it

The OpenAI-compatible response_format works in both engines:

from openai import OpenAI

TICKET_SCHEMA = {...}  # the JSON Schema dict: properties, required, enums
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
ticket = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Turn this bug report into a ticket: ..."}],
    response_format={"type": "json_schema",
                     "json_schema": {"name": "ticket", "schema": TICKET_SCHEMA, "strict": True}},
    max_tokens=400,
)

vLLM accepts more shapes through extra_body={"structured_outputs": {...}}: choice (one of a fixed list, the cheapest way to get a label), regex, grammar (a context-free grammar), and structural_tag (JSON inside given tags, for tool calls embedded in text) (docs). The older guided_json-style fields were removed in v0.12.0.

Measured: what it costs

Same machine as the other pieces in this series (NVIDIA GB10, vLLM 0.31.0 and SGLang 0.5.21, Qwen3-8B in BF16), same ticket schema:

Medians of 10 requests for time to first token, 10 to 48 for time per token. For a schema of this size, compiling the grammar costs a few milliseconds, and the per-step mask costs nothing we can measure: the CPU fills it while the GPU runs the model. Two things can change that: very large schemas (hundreds of fields, deep nesting, big enums), whose first compilation takes longer, and hundreds of constrained requests at once, where filling masks competes for CPU time. Measure the first-request latency of your own schemas.

What the constraint does not do

• It cannot make an answer finish. The grammar decides which tokens are allowed, not when to stop. Our constrained 0.6B model sometimes looped inside a string until it hit max_tokens, and a JSON document cut in the middle is not valid. Set max_tokens with room to spare, give free-text fields a maxLength, and check finish_reason: length means truncated.
• It cannot make the content right. A forced "priority": "high" is valid; whether the bug deserves "high" is still the model's judgment. Forcing a format can even hurt the content when the model would have reasoned before answering: let reasoning models think first (the default with a reasoning parser), or add a reasoning field before the answer fields in your schema.
• It does not replace a tool-call parser. Tool calls go through each model's own format and the engine's --tool-call-parser; structured output is for the answer you want back.

Next

Structured output narrows the model's choices at each step without changing how many steps it takes. Speculative decoding attacks the number of steps itself: a cheap model guesses several tokens ahead and the big model checks them all at once.