Valid JSON, wrong values: validate LLM structured outputs

Reliable JSON from an LLM (large language model) takes two layers: a JSON Schema the provider enforces, and validation of every response in your code. Refusals, cut-off responses and wrong values still get past the schema, so your code checks for each one. A library such as Pydantic does the shape check, and a bounded retry that sends the error back fixes some failures.

The provider and library behaviour below comes from each one’s official docs, last verified on 1 November 2026. No provider API was called for this post. The Python examples were last verified on 1 November 2026 against Pydantic 2.13.4, with a stub in place of the model call.

This is part 28 of the series. Part 26 built an eval suite for what a model says. This part makes sure your code can read it.

JSON mode, structured outputs and tool calling do different jobs

JSON mode, schema-constrained structured outputs and tool calling are three ways to get machine-readable output from a model, and they promise different things.

JSON mode asks for valid JSON in any shape. OpenAI’s JSON mode docs say it “will not guarantee the output matches any specific schema, only that it is valid and parses without errors.”

Structured outputs hold the response to a JSON Schema you send with the request. Anthropic’s docs describe the mechanism: the API compiles your schema “into a grammar that constrains Claude’s output”. The keys and types come out as declared.

Tool calling, also called function calling, uses a schema for the arguments of a function the model asks your code to run. With strict mode on, those arguments follow the schema too.

Pick by where the output goes. OpenAI’s guide says: “If you are connecting the model to tools, functions, data, etc. in your system, then you should use function calling”. A structured response format suits the model’s reply to the user. Anthropic’s docs split it the same way: JSON outputs control “what Claude says”, and strict tool use controls “how Claude calls your functions”. One request can use both.

What OpenAI, Anthropic and Gemini document

Every row comes from the linked docs, last verified on 1 November 2026; none was tested for this post.

The same schema can get different treatment from each provider. A maxLength on a string gets a 400 error from Anthropic’s raw API, and Gemini drops it without a word. Anthropic’s SDK helpers move such constraints into the field description, and a helper that validates responses checks them there.

Validate every response in code, even with strict mode on

Validation in your own code still matters when the provider enforces the schema. OpenAI’s guide says “Structured Outputs can still contain mistakes.” Gemini’s best practices say: “While output is syntactically correct JSON, always validate values in your application.”

A validation library lets one definition do both jobs. You send its JSON Schema to the provider and check the response against the same model. That also keeps the schema and your types from drifting apart, which OpenAI’s guide warns about. Pydantic is the example here because all three providers’ Python SDKs accept it. In TypeScript, Zod plays the same role, and its .safeParse() returns a result object with either the parsed data or the error.

An example Pydantic model for a made-up feature that reads customer emails and pulls out a refund request:

from typing import Literal

from pydantic import BaseModel, ConfigDict, Field

SCHEMA_VERSION = "refund_request.v2"


class RefundRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    order_id: str = Field(pattern=r"^ORD-\d{6}$")
    amount_minor: int = Field(gt=0)  # cents, pence or paise
    currency: Literal["GBP", "EUR", "INR", "USD"]
    reason: Literal["damaged", "late", "wrong_item", "other"]
    summary: str = Field(max_length=300)

RefundRequest.model_json_schema() gives the JSON Schema to send, and RefundRequest.model_validate_json(text) checks the response. With extra="forbid", an unexpected key fails validation, and the generated schema gets the additionalProperties: false that OpenAI and Anthropic require. Pydantic’s models docs say “Providing extra data is not permitted.” Text that is not valid JSON at all fails with the json_invalid error type.

A failed check raises one ValidationError holding every problem. Pydantic’s error docs list errors() for the details of each one, including its location loc and a message msg. str(e) gives a readable version, and that is what the retry below sends back.

What still breaks after the schema passes

A response can pass through a strict schema setting and still be unusable. Each failure below shows up in a different place, so the code checks for each one separately, before and after parsing.

The rows quote the linked docs, last verified on 1 November 2026. Gemini’s structured output page doesn’t cover refusals or truncation, so check how its API reports a stopped or blocked response before you parse.

Wrong values pass every schema

A schema checks shape and type, and a constrained response can still carry the wrong number. An email that mentions two orders can produce a perfectly valid order_id for the wrong one. Checks against your own records catch some of these.

An example business check for the made-up refund feature:

def check_against_order(req: RefundRequest, order) -> list[str]:
    if order is None:
        return [f"{req.order_id} is not a known order"]
    problems = []
    if req.currency != order.currency:
        problems.append("currency differs from the order")
    if req.amount_minor > order.total_minor:
        problems.append("refund is more than the order total")
    return problems

Send these failures to a person. A retry that says “the amount is too high” invites the model to pick a lower number, and that number is a guess. Treat the parsed object as untrusted input, like anything else a model writes. Part 20 covers what that means for the browser and the email. Some wrong values look right to any check you can write, and measuring those is the job of an eval suite.

Retry with the validation error, a fixed number of times

A retry that sends the validation error back tells the model which field failed and why. OpenAI’s JSON mode docs suggest the same pairing when schema enforcement isn’t available: “use a validation library and potentially retries”. With strict mode on, Anthropic’s docs promise “No retries needed for schema violations”. Retries then mostly cover constraints the provider couldn’t enforce, like a maxLength that Gemini strips.

An example loop, where call_model and log_failure stand in for your own provider wrapper and logging:

from pydantic import ValidationError

MAX_ATTEMPTS = 3  # the first try plus two retries


def extract_refund(email_text: str) -> RefundRequest | None:
    messages = [{"role": "user", "content": email_text}]
    for attempt in range(1, MAX_ATTEMPTS + 1):
        result = call_model(messages, schema=RefundRequest.model_json_schema())

        if result.stop == "refusal":
            log_failure("refusal", attempt)
            return None  # send to a person; resending won't help
        if result.stop == "max_tokens":
            log_failure("truncated", attempt)
            return None  # raise the limit or shrink the task, then try again

        try:
            return RefundRequest.model_validate_json(result.text)
        except ValidationError as err:
            log_failure("invalid", attempt, err.errors())
            messages += [
                {"role": "assistant", "content": result.text},
                {"role": "user", "content": (
                    "Your JSON failed validation:\n"
                    f"{err}\n"
                    "Return the corrected JSON only."
                )},
            ]
    return None  # out of attempts: send to a person

Your wrapper maps each provider’s stop signal to result.stop. On Anthropic, that’s stop_reason. On OpenAI’s Responses API, a cut-off shows as status: "incomplete" with incomplete_details.reason set to max_output_tokens, per its guide. A refusal there is a content item of type refusal.

The cap lives in code, and the last step is always a fallback a person can see. Each retry sends the whole conversation again and pays for it, as Part 22 covers. Log every failure with its kind and the schema version, so your traces show which field keeps failing (Part 21 covers tracing).

Version the schema like an API contract

A schema change is a contract change for every piece of code that reads the output, including rows you stored last month. Adding an optional field is usually safe. Renaming a field or changing its type breaks older readers. Part 18 covers breaking and non-breaking changes to a contract, and the same rules apply here.

Keep three habits:

1. Give each schema a name and a version, such as refund_request.v2, and store that version with every saved result and trace.
2. Keep a parser for each version still in storage, so old rows stay readable.
3. Run your eval set against the new schema before you switch traffic.

An example of reading stored rows by version:

PARSERS = {
    "refund_request.v1": RefundRequestV1,
    "refund_request.v2": RefundRequest,
}


def load_refund(row: dict):
    return PARSERS[row["schema_version"]].model_validate(row["data"])

Plan the rollout around the provider’s schema cache too. Anthropic’s docs say the first request with a new schema has “additional latency while the grammar compiles”. The compiled grammar is cached for 24 hours from last use, and changing the schema’s structure invalidates it. OpenAI’s guide also mentions extra latency on the first request with a schema, and one of its notes limits that to fine-tuned models.

To keep the JSON Schema and your code in step, OpenAI’s guide suggests “a CI step that automatically generates the JSON Schema from type definitions (or vice-versa)”. With Pydantic, model_json_schema() is that generator.

Two prompts for structured output in an AI feature

A coding agent can find every place your code reads model output and add the checks. Neither prompt relies on one tool’s features.

Prompt 1: find where model output is parsed

An example prompt for mapping the risk before you change anything.

Task: review how <the feature or service> reads model responses.
Start from: <path or entry point>

Do not change code and do not call any model API.
For every place the code uses a model's response:
1. Which mode is used: plain text, JSON mode, a JSON Schema
   (structured outputs), or tool calling? Is strict mode on?
2. Is the response validated in code? With what?
3. Are refusals and token-limit cut-offs checked before parsing?
4. Are values checked against the system's own data?
5. Are retries capped? What is sent back on a retry?
6. Is the schema versioned, and is the version stored with results?

Cite the file and line for each answer. Mark anything you could not
confirm as unverified.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

End with the three places most likely to break on a bad response.

Prompt 2: add validation and bounded retries to one call

An example prompt for fixing one model call at a time.

Task: add schema validation to the model call in <file and function>.
Validation library: <Pydantic, Zod or the one the project uses>

1. Define the output as one schema in that library, and generate the
   JSON Schema sent to the provider from it.
2. Check the provider's documented schema limits, and list any
   constraint it will reject or ignore.
3. Before parsing, handle refusals and token-limit cut-offs.
4. Validate every response. On a validation error, retry at most
   <N> times, sending the error back. Then fall back to <fallback>.
5. Never retry business-rule failures; return them for a person.
6. Add a schema version and log it with every failure.
7. Add tests for: invalid JSON, a missing field, an extra field,
   a refusal, a cut-off, and a value that breaks a business rule.

Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

Read the tests it writes first; a test that mocks the validator proves nothing.

A structured output checklist

An example checklist, in the order of this post:

STRUCTURED OUTPUT: REVIEW

□ Structured outputs or strict tool calling, not JSON mode alone
□ One schema in code; the JSON Schema is generated from it
□ Provider's schema limits checked; ignored constraints known
□ Refusal checked before parsing
□ Token-limit cut-off checked before parsing
□ Every response validated in code
□ Values checked against your own data
□ Retries capped; the validation error sent back
□ Business-rule failures go to a person, never a retry
□ Schema versioned; version stored with results and traces
□ Old versions still parse
□ Eval set run before a schema change ships

Quick answers

What is the difference between JSON mode and structured outputs?

JSON (JavaScript Object Notation) mode asks a model for valid JSON in any shape. Structured outputs hold the response to a JSON Schema you supply, so required keys and types come out as declared. OpenAI recommends structured outputs over JSON mode where a model supports them.

Do structured outputs guarantee correct values?

Structured outputs constrain the shape of a response, and the values inside it can still be wrong. OpenAI’s guide says structured outputs “can still contain mistakes”, and Gemini’s docs ask developers to validate values in the application.

Why does an LLM return incomplete JSON?

A large language model (LLM) returns incomplete JSON (JavaScript Object Notation) most often when the response hits the output token limit. Anthropic reports this as stop_reason: "max_tokens", and OpenAI’s Responses API marks the response incomplete. Check the stop reason before parsing.

When should you use tool calling instead of structured outputs?

Tool calling suits a model that needs your code to run a function, such as a database lookup. Structured outputs suit the model’s final answer. OpenAI and Anthropic both document strict modes for tool arguments, and one request can use both features.

How should you retry an invalid LLM response?

A retry for an invalid large language model (LLM) response sends the validation error back with the previous output. It runs a fixed number of times, set in code. After the last attempt, the request falls back to a person or a safe default.

More on AI coding is in the AI category.