Guide · AI engineering
Every LLM feature that touches a database eventually needs the model to return structured data instead of prose. There are four mechanisms for that, they are not equivalent, and three of them still need a validation layer you have to write yourself. Here is the ladder, the failure modes at each rung, and the two problems — truncation and duplicated side effects — that no amount of schema enforcement solves.
The first version of every extraction feature asks the model to "return JSON" and then parses whatever comes back. It works in testing, because in testing you send five inputs and they all look alike. It fails in production, and it fails in a specific, tediously predictable set of ways.
Here is the extracted data: followed by the object."N/A" in an integer field. Or "unknown". Or null in a field you did not mark nullable.Each of these is individually fixable with a regex, and the regex pile is exactly how you end up with a 200-line parser that is itself the least reliable component in the system. The fix is not a better parser. The fix is to stop letting the model emit invalid output in the first place, and then to validate what it does emit anyway.
| Mechanism | What it guarantees | What it does not |
|---|---|---|
| Prompt and parse | Nothing. | Everything. Do not build on this. |
| JSON mode | The output is syntactically valid JSON. | That it matches your schema, has your fields, or uses your types. |
| Tool / function calling | The model emits a named call with an arguments object shaped like your schema. | Deep validation of nested constraints, unless the provider offers a strict mode. |
| Constrained decoding | Token-by-token, the model is only permitted to emit tokens that keep the output schema-valid. Invalid output is unrepresentable. | That the content is correct. A schema-perfect object can still be wrong. |
JSON mode is a flag that says "the response body must parse as JSON." It solves the markdown fence, the preamble, and the trailing comma. It does not solve the schema — you can get back valid JSON with none of your fields, or with your fields carrying types you never asked for. It is a real improvement over parsing prose and it is not a solution.
Tool calling is the mechanism most people should reach for, because it was designed for exactly this. You describe a function with a JSON Schema for its parameters, the model returns a structured call to that function, and the provider handles the shaping. The subtle win is that it aligns with the model's training: models are heavily trained to emit well-formed tool calls, so you get better adherence than from a prompt asking for the same shape.
Many providers now offer a strict mode on top of this — a flag on the tool definition that makes the provider actually enforce the schema rather than treat it as a strong suggestion. Where that exists, use it. It typically requires additionalProperties: false and an explicit required array, which is a reasonable price for the guarantee.
Constrained decoding is the strongest mechanism and the one most people do not know they have. At each generation step, the model produces a probability distribution over the entire vocabulary; a grammar engine masks out every token that would make the partial output invalid against your schema, and sampling happens only over what remains. The model is not asked nicely to produce valid JSON — it is structurally incapable of producing anything else.
This is available locally for free. Ollama accepts a JSON Schema in the format field and enforces it during generation; llama.cpp exposes the same capability as GBNF grammars; libraries like Outlines and XGrammar implement it over other runtimes. If you are running a model locally, this is the single highest-leverage feature you are probably not using:
Constrained decoding guarantees the shape and says nothing about the content. A model that does not know the rate will still emit a number, because the grammar requires one. Schema enforcement converts "unparseable garbage" into "parseable garbage" — which is a large improvement, since parseable garbage can be validated, logged, and caught, while unparseable garbage just throws.
Whichever mechanism you use, a validation layer belongs in front of your business logic. It is the only place that can enforce the constraints schemas cannot express — that a date is in the past, that a total equals the sum of the line items, that an ID exists in your database — and it is the layer that gives the model a second chance with a useful error message instead of dying.
The pattern is short. Validate; on failure, feed the validation error back as a new turn and try once more; cap the attempts and fail loudly.
from pydantic import BaseModel, ValidationError, Field from typing import Literal class Load(BaseModel): origin_city: str = Field(min_length=2) rate_usd: float = Field(gt=0) equipment: Literal["dry_van", "reefer", "flatbed", "unknown"] pickup_date: str # ISO 8601 def extract(text: str, max_attempts: int = 3) -> Load: messages = [{"role": "user", "content": PROMPT + text}] for attempt in range(max_attempts): raw, finish = call_model(messages, schema=Load.model_json_schema()) if finish == "max_tokens": # truncation, not a parse bug raise OutputTruncated(attempt) # retrying identically will not help try: return Load.model_validate_json(raw) except ValidationError as e: messages += [ {"role": "assistant", "content": raw}, {"role": "user", "content": f"That failed validation:\n{e}\nReturn corrected JSON only."}, ] raise ExtractionFailed(f"{max_attempts} attempts exhausted")Two details in there are load-bearing. First, the validation error text goes back to the model verbatim — a good validator's error message names the field and the constraint, which is exactly the information needed to fix it, and models are strikingly good at acting on it. Second, truncation is detected separately and never retried the same way, for reasons in the next section.
When a response hits the output token limit, generation stops mid-token. With constrained decoding you get a syntactically valid prefix of an invalid document; without it you get a string ending in "origin_city": "Sto. Both surface at your parser as a parse failure, and treating them as one is the bug: no amount of retrying an identical request fixes a limit that is too low.
Every major API returns a stop reason or finish reason on the response. Check it before you check the parse. If it indicates the token limit was hit, that is a configuration failure — raise the limit, shrink the requested output, or split the extraction into smaller pieces. Retrying the same call gets the same truncation and burns money doing it.
The related trap is streaming. If you stream a structured response and parse incrementally, every intermediate state is an incomplete document. Partial-JSON parsers exist and are genuinely useful for progressive UI, but nothing downstream of a partial parse may perform a side effect — you are looking at a document that is still being written and whose remaining fields may contradict what you already read.
Here is the failure that reaches production: extraction succeeds, the write succeeds, the response times out on the way back, the retry logic fires, and the same record is written twice. The model behaved perfectly. The retry loop did the damage.
The rule is simple and easy to violate: the retry loop must never contain a side effect. Structure the code so the model call and validation happen inside the loop and the write happens strictly after it returns.
# wrong — a validation retry can re-run the insert for attempt in range(3): data = call_model(...) db.insert(data) # side effect INSIDE the loop # right — retries are pure; the write happens once, keyed data = extract(text) # loop is contained here, no side effects db.upsert(data, key=idempotency_key(source_document_id))Then make the write itself idempotent with a key you derive, not one the model invents. A hash of the source document plus the extraction version is ideal: deterministic, collision-resistant, and stable across retries. Never let the model generate the idempotency key — it will generate a different one on the retry, which is precisely the case the key exists to prevent.
Most validation failures are avoidable by writing a better schema, which is cheaper than handling them at runtime.
"unknown" or "other" as a permitted value. Without one, a model facing an input that fits nothing on your list will pick the closest wrong option, and you will never know it guessed.reasoning string before classification lets the model work through the problem in its own output before committing. It costs tokens and measurably improves accuracy on judgment calls."rate_usd": total linehaul rate in USD, excluding fuel surcharge prevents a category of error that no amount of type enforcement catches.additionalProperties: false. Both a real constraint and, in strict modes, a prerequisite for enforcement.Structured output failures are close to impossible to debug after the fact without the raw string. Log, on every failure: the raw model output before parsing, the stop reason, the validation error, the attempt number, and the schema version. That last one matters more than it sounds — schemas drift, and "this used to work" is usually "the schema changed and nothing recorded which version produced this row."
Track your retry rate as an actual metric. A stable 2% is the system working. A rate climbing over weeks means your inputs are drifting away from what your prompt and schema were built for, and the retry loop is quietly absorbing it until the day it cannot. This is the same discipline as the automated quality gates I use for generated outreach drafts: the model produces, a deterministic checker validates, and nothing ships that fails the check.
| Situation | Use |
|---|---|
| Local model, any structured task | Constrained decoding — Ollama's format field with a JSON Schema. Free, strongest guarantee available. |
| Hosted API, extraction or classification | Tool calling with strict mode if the provider has it, plus a validation layer. |
| Hosted API, no strict mode | Tool calling plus validation with a capped retry loop. |
| Output is long (large arrays, many records) | Split into batched calls. Long structured outputs truncate, and truncation is the failure retries cannot fix. |
| Any write to a database or external service | Idempotency key derived from the source, upsert outside the retry loop, always. |
| Free-text summary with no downstream consumer | Do not force a schema. Structured output is for machine consumers, and JSON overhead on prose buys nothing. |
The through-line is that no single mechanism is sufficient. Constrained decoding gives you shape, validation gives you semantics, idempotency gives you safety under retry, and logging gives you the ability to fix it when the inputs change. Skip any one of them and the gap shows up eventually — usually as a duplicated row someone finds three weeks later. If you are building this into a real product, the same care belongs on the surrounding infrastructure; the security checklist covers the parts that bite next.
FAQ
JSON mode only guarantees that the response parses as valid JSON — it says nothing about which fields are present or what types they hold. Tool calling has you define a JSON Schema for a function's parameters and the model returns a structured call shaped to that schema, which is far closer to what you actually want. Tool calling is the better default because models are heavily trained on emitting well-formed tool calls.
Constrained decoding masks the model's token choices at every generation step so that only tokens keeping the output valid against your schema are sampleable. The model is structurally unable to emit invalid output, rather than being asked to avoid it. Ollama exposes this through the format field, llama.cpp through GBNF grammars, and libraries like Outlines and XGrammar over other runtimes.
Yes. Schema enforcement guarantees shape, not semantics — it cannot check that a date is in the past, that a total equals the sum of its line items, or that an ID exists in your database. A validation layer also gives you a place to catch the failure and feed a specific error message back to the model for one corrective attempt.
Check the response's stop or finish reason before you check the parse. If generation hit the token limit, that is a configuration failure and retrying the identical request will truncate identically — raise the limit, request less output, or split the task into batches. Treating truncation as a parse error is the most common way to burn money on a loop that cannot succeed.
Keep every side effect outside the retry loop. The model call and validation belong inside; the database write happens once after the loop returns, keyed by an idempotency key you derive deterministically, such as a hash of the source document plus the extraction version. Never let the model generate the key — it will produce a different one on retry.
No. Temperature 0 makes sampling greedy, which reduces variation, but it does not guarantee byte-identical outputs across runs — batching, hardware, and kernel differences all introduce nondeterminism. Design for the possibility that the same input produces a different valid object, and validate every response rather than assuming a cached earlier result still applies.
Give every enum an explicit unknown or other value so the model is not forced to guess, put a reasoning string field before the answer field so it can work through the problem in-output, write a description on every field, keep the structure flat, and mark genuinely optional fields optional so the model is never asked to fabricate a value it cannot find.
No. Structured output exists to serve machine consumers. If the result is prose a human reads, wrapping it in JSON adds token overhead and a failure mode with no corresponding benefit. Reach for a schema when something downstream needs to branch on a field, not by default.