Guardrails for Structured Output: Why "JSON Mode" Isn't the Same as a Schema Guarantee

“JSON mode” sounds like a guarantee. It isn't — the model can still skip required fields or invent new ones. The real guarantee comes from a different, deeper technology: constrained decoding.

JSON mode vs. Structured Outputs: the difference that matters

JSON mode forces syntactically valid JSON, but the model can still skip required fields, invent new ones, or return the wrong types. Structured Outputs, by contrast, guarantees that every field in the defined schema will appear in the output exactly as defined, with no surprises — the difference between “it's valid JSON” and “it's the JSON you asked for, in the exact shape you asked for.”

How constrained decoding works under the hood

Grammar-constrained decoding excludes specific tokens at given positions in the output — avoiding harmful words or forcing adherence to a user-defined output grammar for languages like JSON or SQL. Constrained decoding enforces validity during generation itself, masking invalid tokens at every step — it doesn't validate after generating, it prevents invalid generation from happening in the first place.

Speed is no longer the obstacle

XGrammar has been the default structured-generation backend for vLLM, SGLang, and TensorRT-LLM since March 2026, achieving under 40 microseconds per token with near-zero overhead on JSON generation. Microsoft's llguidance uses a Rust-based Earley parser at roughly 50 microseconds per token with negligible startup costs — the old concern that structured output was “slower” has effectively disappeared.

Who already supports it natively

OpenAI has supported native structured output since August 2024, Google Gemini since 2024 (expanded in 2026), Anthropic in beta since November 2025 with general availability in early 2026, and Cohere and xAI (Grok) support it too — by 2026, it's no longer a single provider's exclusive feature, it's a de facto standard across the frontier labs.

When to use each approach

JSON mode is enough for low-risk cases where an occasional missing field is tolerable. Function calling makes sense when the structured output represents a call to a specific tool with a schema already defined by that tool. Structured Outputs with a real schema guarantee is the right choice when the output feeds directly into a downstream system (a database, another service) where a missing or mistyped field breaks something beyond the LLM itself.