LLM Output Validation

Why it matters

The lesson on hallucination testing asked whether an answer is true to its sources. This lesson asks whether the application can safely use it. A missing field crashes a parser, "Refund" instead of "refund" sends a ticket to the wrong queue, an invented order number triggers an action on someone else’s order, and a script hidden in a reply runs in an agent’s browser. These are ordinary defects from an unusual source.

For the tester, a contract turns vague output into something you can assert on. A schema check is deterministic even when the wording is not, so it can run on every build. And because a model will sometimes produce bad output however good the prompt is, the useful question is not "does the model always comply?" but "what does the application do when it does not?" — and that you can test.

Structured output: give the response a contract

Structured output means the model answers in a machine-readable format, almost always JSON, with fixed field names and allowed values. A line in the prompt such as "reply only in JSON" is a request, not a guarantee. Many model APIs also offer a JSON mode, a structured-output option or tool calling, which constrain the response, to different degrees, to valid JSON or to a schema you supply. Read what your provider actually guarantees: it may cover syntax only, or only part of JSON Schema, and a response can still be cut off at the output token limit or be a refusal instead of data.

Write the contract as a JSON Schema kept in the repository and used by both the application’s validator and the tests. In the schema below, additionalProperties: false rejects fields nobody asked for, enum fixes the allowed values (case-sensitive), and pattern constrains the order number. order_id is required but may be null: a message without an order number gets an explicit null, not a missing field or an invented number. The guide on JSON explains how schema validation goes beyond checking that the text is valid JSON. Note that format values such as date are annotations by default in recent drafts, so many validators ignore them unless configured; use pattern or a code check when the format matters.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["intent", "priority", "order_id", "reply"],
  "additionalProperties": false,
  "properties": {
    "intent":   { "enum": ["refund", "delivery", "account", "other"] },
    "priority": { "enum": ["high", "normal", "low"] },
    "order_id": { "type": ["string", "null"], "pattern": "^ORD-[0-9]{6}$" },
    "reply":    { "type": "string", "minLength": 1, "maxLength": 800 }
  }
}

Try it: JSON Schema Validator Try it: JSON Validator

Validate in layers: parse, schema, rules, guardrails

Each layer catches what the one before it cannot. A response that fails any layer never reaches the consumer as it is: it is logged with the prompt and model versions, and the application retries within a hard limit or falls back to something safe, such as a human queue. Test that failure path as carefully as the happy path: what the user sees, that the retry limit holds, and that the log is enough to reproduce the case.

• Parse: is it JSON at all? Prose before the JSON, a Markdown code fence or a truncated object fails here. If the application repairs output, for example by stripping a fence, that repair is specified behaviour to test.
• Schema: required fields, types, allowed values, lengths, no extra fields. A schema failure makes the output unusable, however good the wording is.
• Business rules: do the values make sense for this input and this data? The order_id must appear in the message and belong to that customer. This layer catches invented values that are perfectly schema-valid.
• Guardrails: policy checks around the model that block, redact or replace output — personal data in a reply, a forbidden topic, a link outside an allow-list. Test both directions: bad output is stopped, and normal output is not blocked by mistake.

Try it: Regex Tester

Consistency across runs

The same input can give different output on different calls. Lowering the temperature reduces variation but does not guarantee identical responses, so one passing run proves little. Run each important case several times, for example 10, with the production model version, prompt version and parameters, and record N and every result.

Do not compare whole responses as strings. Decide which fields must be stable — intent, priority, extracted IDs and amounts — and which may vary, such as the wording of reply. A decision that flips between runs is a defect even when each answer looks reasonable, because the same request gets different treatment. Consistent output can still be consistently wrong, so consistency checks complement correctness checks; they never replace them.

• Schema validity: every run should pass. At production volume, one failure in 10 calls means many failed requests every day.
• Decision agreement: count the distinct values each stable field took across the runs.
• When a decision flips, ask whether the rule is defined at all. Often the fix is to state the rule in the prompt, or to compute that field in ordinary code.

Try it: JSON Diff

Treat model output as untrusted input

The OWASP Top 10 for LLM Applications 2025 lists improper output handling (LLM05) as a risk of its own: model output passed to other components without sufficient validation, sanitisation and handling. Whoever wrote the input, or the documents the model read, can influence what it writes, so receiving code must never trust output because "it came from our own model".

Schema validity is not safety: "reply": "<b>Sorry</b>" is a valid string. Each consumer handles output for its own context: encode it, or use a strict sanitiser, before showing it as HTML; pass it into SQL as a parameter, never by concatenation; check URLs, file names and commands against an allow-list; and never let the model trigger an action the user may not perform. Test with harmless markup and metacharacters, only on systems you own or have written permission to test. The guide on LLM security testing lists benign cases for each context.

Try it: Special Character Dataset Generator Try it: HTML Entity Encoder

Worked example: one triage case, ten runs

Input: "Hi, order ORD-204913 arrived broken. I want my money back." The tester ran this case 10 times against the schema above and recorded, for each distinct response, the first layer it failed, if any. Five patterns appeared (below). R4 is the instructive one: its order number matches the pattern but has two digits swapped, so only the business rule catches it.

In all 10 runs intent was refund, but priority was normal 7 times and high 3 times. The bug report states the input, N = 10, the split, the model and prompt versions, and one question for the product owner: are damaged goods high priority? Once the rule is agreed, either state it in the prompt with examples, or have the model extract only the fact (did the item arrive damaged?) and compute priority from it in ordinary code, which an exact test can check.

R1  {"intent":"refund","priority":"normal","order_id":"ORD-204913","reply":"..."}
    parse ok, schema ok, rules ok                        -> accept
R2  Here is the JSON: {"intent":"refund", ...}
    parse fails: text before the JSON                    -> reject, one retry
R3  {"intent":"refund","priority":"normal","reply":"..."}
    schema fails: required field order_id is missing     -> reject, one retry
R4  {..., "order_id":"ORD-204931", ...}
    rules fail: order number is not in the message       -> reject, human queue
R5  {..., "reply":"<b>Sorry</b>, the item arrived broken..."}
    all layers pass                                      -> accept, show as text

Try it: JSON Schema Validator

Frequently asked questions

References

• OWASP Top 10 for LLM Applications 2025OWASP Gen AI Security Project · 2025The ten most critical security risks of LLM applications (LLM01:2025–LLM10:2025), starting with prompt injection and sensitive information disclosure.
• JSON Schema SpecificationJSON SchemaThe vocabulary for describing and validating the shape of JSON documents (types, required properties, formats).
• OWASP Cheat Sheet Series: Cross Site Scripting PreventionOWASP FoundationPreventing XSS: output encoding for each context, sanitizing user-authored HTML, safe DOM sinks and framework protections, with CSP only as defence in depth.
• Define success criteria and build evaluationsAnthropicA model provider's guidance on writing measurable success criteria and evaluation sets graded by code, by exact match or by a model.