Structured output from language models: getting reliable JSON instead of prose

You want a language model to pull four fields out of a document — name, date, amount, category — and hand them to the next step in your code. So you ask it to, and most of the time it returns clean JSON. Then one call in fifty comes back as Sure, here is the information you requested: followed by JSON wrapped in a code fence, or a trailing comma, or an extra field it decided to invent, or a friendly note explaining a caveat. Your parser throws, the pipeline halts, and the failure is intermittent enough to be maddening.

This is the core tension of putting a model into an automated system. A language model is trained to produce fluent text for a human. A pipeline needs a predictable data structure for a machine. Getting reliable structured output is the work of closing that gap, and it is very solvable — but “ask nicely and hope” is not the way.

Why free text breaks pipelines

The problem is not that the model is wrong. It is that “correct” for a text generator and “parseable” for your code are different targets.

• Prose leaks in. Preambles, apologies, explanations, and closing remarks wrap the data you wanted. Every one of them breaks a strict parser.
• Format drifts. Markdown fences, single quotes instead of double, trailing commas, numbers rendered as words. All human-friendly, all invalid JSON.
• The shape wanders. Fields appear, disappear, or get renamed between calls. An enum you expected to be one of three values comes back as a fourth, politely.
• It is non-deterministic. The same prompt can succeed a thousand times and fail on call 1,001, which makes the failure hard to catch in testing and easy to hit in production.

Any one of these turns a model from a component you can build on into a source of random breakage.

Constrain the generation, do not just request it

The shift that changes everything is moving from asking for a format to enforcing one. Most current model APIs offer some version of this, usually under one of two names.

Tool or function calling. You describe a function with a typed parameter schema, and the model responds by “calling” it — returning arguments that match the schema rather than free text. Even when you have no real function to run, this is a reliable way to get a structured object out, because the field names and types are defined up front rather than left to the model’s discretion.

Schema-constrained output. Many providers now accept a JSON Schema directly and guarantee the response conforms to it. Under the hood, the decoding step is restricted so the model can only emit tokens that keep the output valid against the schema. Instead of hoping for well-formed JSON, you get output that is structurally valid by construction.

The practical difference is large. With a plain prompt, valid JSON is a probability. With a schema constraint, structural validity is close to a guarantee — the model can no longer wander into prose, wrap the answer in a fence, or drop a required field, because the format is enforced at generation time rather than checked afterward.

Validate anyway, then retry

Constrained generation fixes the shape of the output. It does not fix the meaning. A response can be perfectly valid JSON and still be wrong — a date in the wrong format inside a string field, an amount that is negative, a category that is spelled right but semantically absurd for the document. So the durable pattern has three stages, not one.

1. Generate under a schema. Get output that is structurally valid by construction.
2. Validate against your real rules. Parse it into a strict schema in your own code — a validation library such as Zod or a JSON Schema validator — and check the business constraints the model’s schema could not express: value ranges, required combinations, enum membership, cross-field consistency.
3. Retry on failure, with the error. When validation fails, send the model another turn that includes the invalid output and the specific validation error, and ask it to correct just that. Models are usually good at fixing a fault they are shown. Cap the retries — two or three — and fall back to a human queue or a flagged record rather than looping forever.

That loop — constrain, validate, repair — is what turns a probabilistic generator into something you can put in a pipeline and trust to fail loudly and rarely rather than quietly and often.

Where structured output fits

This approach shines when the task is genuinely a transformation into a known shape:

• Extraction — pulling fields from invoices, emails, contracts, or forms.
• Classification and routing — tagging a support ticket, choosing which downstream handler gets a request.
• Normalization — turning messy human input into clean records.
• Tool use — deciding which action to take and with what arguments, which is the mechanism behind most useful agents.

In each of these, the model’s judgment is the valuable part and the structure is just the delivery format. That is the sweet spot.

Where it does not fit

Structured output is a technique, not a reason to reach for a model. Two cautions.

When prose is the actual product, do not force it into fields. Summaries, drafts, explanations, and anything a human will read are meant to be text. Cramming them into rigid JSON usually degrades the very thing you wanted.

When the task should not be a model at all. If the input is regular enough that a parser, a regular expression, or a lookup table would do the job, use those — they are faster, free, and deterministic. A model earns its place when the input is messy and genuinely needs judgment. We drew that line in more detail in when not to use an LLM, and it is worth holding to.

The failure modes to instrument

Even with constraints and validation, plan for these:

• Truncation. If the output hits the token limit mid-object, you get valid-so-far JSON that is incomplete. Set a generous limit for the expected size and detect truncation explicitly.
• Valid but wrong. The most dangerous case, because nothing throws. Only your business-rule validation catches it, which is why step two is not optional.
• Confident hallucination. A model will fill a required field with a plausible invention rather than admit it does not know. Give it an explicit “unknown” or “not present” value in the schema so it has an honest option, and treat that value as a real signal.
• Silent schema drift. Provider and model updates can shift behaviour. A small held-out set of examples you run on every change is the cheapest way to notice before your users do.

Log the raw output, the validation result, and the retry count on every call. When something does go wrong, that record is the difference between a five-minute fix and an afternoon of guessing.

Structured output is one of the quiet techniques that separates a model demo from a system you can run. It is a large part of how we build retrieval and agent pipelines that hold up in production — the work described on our RAG development and AI pages. If you are trying to get a model to feed a real pipeline reliably, send us two paragraphs about the task, and we will reply in writing within one business day.