LLM Evaluation: How to Evaluate LLM Outputs

Step 1: define success criteria for the task

Start from the feature, not from the model. What is the task, who uses it, what does a good answer do, and what must an answer never do? Model providers’ own evaluation guidance makes the same point: criteria must be specific and measurable before any test can be written. "Answers should be helpful" is not a criterion; "for a return-policy question, the answer states the correct return window for the product category and links the policy page" is.

Write criteria at three levels: hard requirements that every answer must meet (schema valid, correct language, no personal data, no disallowed action), task-quality criteria that are scored (correct, complete, grounded, relevant), and operational limits (p95 latency, maximum tokens, cost per request). The hard requirements become deterministic checks; the quality criteria become rubrics.

• Name the intents the feature must handle and the ones it must refuse or redirect.
• Agree what "correct" means for each intent — an exact value, a set of required points, or agreement with a source document.
• Decide the acceptable failure rate per criterion before you see results, not after.

Step 2: build the evaluation dataset

An evaluation dataset (often called a golden set) is a list of inputs with the information needed to grade the output: an expected value, a list of required points, reference sources, or a label saying the model must decline. Build it from real intents — anonymised support tickets, search logs, product questions — plus deliberately designed cases: edge cases, ambiguous questions, out-of-scope requests, long inputs, other languages and adversarial phrasing. Never place real personal data in it; replace names, e-mails and IDs with fictional ones.

Coverage matters more than size. Tag every case with its intent, language, difficulty and risk, and check that each intent has enough cases to show a trend. A few dozen well-chosen cases per important intent is a practical start; grow the set every time production shows you a failure you did not have a case for.

• Store the set as JSON Lines (one case per line) in version control and tag each release of it; results are only comparable on the same dataset version.
• Keep tuning data and evaluation data separate. If you iterate on the prompt using the same cases you use to accept it, you are measuring memorisation of your own examples — the same contamination problem that affects public benchmarks whose questions leak into training data.
• Hold out a set that nobody looks at while editing prompts, and use it only for release decisions.
• Include negative cases: questions with no answer in the available data, requests outside the scope, and inputs that should trigger a refusal.

{"id":"ret-001","intent":"return_policy","lang":"en","input":"Can I return headphones after 20 days?","expected":{"must_include":["30 days","original packaging"],"must_not_include":["60 days"]},"tags":["core"]}
{"id":"ret-014","intent":"return_policy","lang":"ar","input":"هل يمكنني إرجاع سماعة بعد 20 يوماً؟","expected":{"must_include":["30"],"answer_lang":"ar"},"tags":["core","arabic"]}
{"id":"oos-003","intent":"out_of_scope","lang":"en","input":"Which stock should I buy this week?","expected":{"behaviour":"decline_and_redirect"},"tags":["scope"]}
{"id":"unk-007","intent":"return_policy","lang":"en","input":"Can I return a gift card bought in 2019?","expected":{"behaviour":"say_not_covered"},"tags":["no_answer"]}

Try it: JSON Lines Validator Try it: JSON Lines Formatter Try it: Multilingual Test Data Generator

Step 3: decide how to check each quality dimension

Each dimension needs its own check. Prefer the cheapest reliable check: deterministic first, rubric second.

• Correctness — exact match for labels, IDs, numbers and dates; required-point lists for explanations; comparison with a reference answer reviewed by a domain expert.
• Relevance — the answer addresses the question asked, not a neighbouring one; scored by rubric, with "answered a different question" as an explicit failure category.
• Groundedness / faithfulness — every factual claim is supported by the provided context or cited source. Check cited IDs exist deterministically; check support by rubric.
• Hallucination (confabulation) — the answer invents facts, policies, references or URLs. Include cases where the correct answer is "not available"; an invented answer there is a clear failure.
• Consistency — the same question asked twice, or in two phrasings, gets compatible answers. Run repeated and paraphrased variants and compare the decisions, not the wording.
• Robustness — typos, extra whitespace, mixed scripts, emoji, Unicode look-alike characters and very long inputs should not change the decision.
• Safety and toxicity — the output contains no abusive, harmful or disallowed content, including when the input is hostile. Use a written content policy as the rubric.
• Privacy and data leakage — the output never contains personal data, secrets or system-prompt text it was not meant to reveal. Seed fictional marker values in the context and assert they never appear.

Try it: Unicode Dataset Generator Try it: Special Character Dataset Generator

Step 4: choose graders — code, human, model

Code-based grading (ESTABLISHED) is fast, cheap and exact: JSON Schema validation, exact match, regular expressions, word counts, set membership of cited IDs. Use it for everything it can decide. Human grading (ESTABLISHED) against a written rubric is the reference for quality judgements, but it is slow and must itself be checked for consistency.

Model-graded evaluation (EMERGING), often called LLM-as-a-judge, asks a model to score an output against a rubric or to compare two outputs. It scales rubric grading, and research reports substantial agreement with human preferences on some tasks. The same research (Zheng et al., 2023) also reports biases: position bias (favouring an answer because of its position in the prompt), verbosity bias (preferring longer answers even when they are not better) and self-enhancement bias (favouring answers the judge model generated itself), as well as weakness at grading mathematical and reasoning answers. Treat these as research findings to design around, not as a reason to trust or reject the method wholesale.

• Calibrate a model grader on a sample that humans have already scored; measure how often they agree before relying on it.
• In pairwise comparisons, run both orders (A,B and B,A) and count only consistent verdicts.
• Give the grader a specific rubric with examples of each score, and ask for a short reason before the score.
• Pin the grader model version; a grader update changes every score.
• Keep humans on the high-risk slice — safety, legal, medical or financial content — and on a random sample of the rest.

Human evaluation: rubrics and inter-rater agreement

A rubric turns "is this a good answer?" into several narrow questions, each with defined score levels and examples. Two reviewers should reach the same score for the same answer; if they often do not, the rubric is ambiguous, not the reviewers careless. Measure inter-rater agreement on an overlapping sample — the simple percentage of matching scores, or a chance-corrected statistic such as Cohen’s kappa for two raters — and refine the wording of the levels where disagreement clusters.

Rubric: support-answer v3 (score each 0-2)

Correctness   2 = all facts match the policy source
              1 = minor omission, nothing false
              0 = any false statement
Groundedness  2 = every claim traceable to a cited source
              1 = one unsupported but harmless claim
              0 = invented policy, number or link
Relevance     2 = answers the question asked
              1 = answers it partly
              0 = answers a different question
Tone/Language 2 = user's language, required tone
              1 = correct language, tone off
              0 = wrong language

Hard fail (score = 0 overall): personal data in output,
disallowed action suggested, system prompt revealed.

Step 5: run, repeat and set thresholds

Because outputs vary, one run per case is a sample of one. For cases that matter, run each input several times at the production sampling settings and record the pass rate. Set thresholds per criterion in advance — for example, schema validity 100%, hard-fail cases 100%, correctness at least 90% of runs, groundedness at least 95% — and report results with the number of cases and runs so readers can judge the uncertainty. Small differences between two runs of the same configuration are noise; do not celebrate or panic over them.

• Use the same temperature and parameters as production; testing at a different setting evaluates a different system.
• Record model ID, prompt version, dataset version, grader version and parameters with every result file.
• Track latency, token usage and cost per case alongside quality — a better answer that doubles cost is a trade-off to decide, not a free win.

Try it: JSON Schema Validator

Step 6: compare with the baseline on every change

Regression evaluation (ESTABLISHED as a practice, even when some graders are emerging) compares the candidate configuration with the last accepted one on the same dataset version. Look at three things: aggregate scores per criterion, cases that flipped from pass to fail, and a diff of outputs for a sample of cases. The flipped cases are where the real information is; an average can improve while an important intent degrades.

• Defect example: "After prompt v14, return-policy answers in Arabic switch to English in 6 of 20 cases".
• Defect example: "Model update 2026-09 invents a 60-day return window for gift cards (no source in corpus)".
• Defect example: "Answer JSON omits the required field sources[] when the context has more than 8 chunks".
• Defect example: "Model grader scores verbose answers higher; 12 correct short answers marked as failures".

Try it: Text Diff Try it: JSON Diff

Public benchmarks: research, not acceptance tests

Public benchmarks are RESEARCH instruments. TruthfulQA, for example, is a research benchmark designed to reveal whether a model repeats common misconceptions; HELM is a research framework that evaluates models on many scenarios and several metrics at once, such as accuracy, calibration, robustness and fairness. They are useful when choosing between models and for understanding the kinds of failure to look for. They are not evidence that your feature works: your prompts, data, users and languages are not in them, and a widely published benchmark may have leaked into a model’s training data. Your own versioned evaluation set is the acceptance test.

Frequently asked questions

References

• Define success criteria and build evaluationsAnthropicA model provider's guidance on writing measurable success criteria and evaluation sets graded by code, by exact match or by a model.
• Working with evalsOpenAIA model provider's guidance on testing model outputs against defined criteria and datasets, and re-running evals as prompts and models change.
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)NIST · 2024The generative-AI profile of the AI RMF: risks specific to generative models, such as confabulation and information integrity, and suggested actions to manage them.
• Certified Tester AI Testing (CT-AI) v2.0ISTQB · 2026ISTQB's specialist syllabus on testing AI-based systems, covering machine-learning and generative-AI systems.
• ISO/IEC TR 29119-11:2020 — Software and systems engineering — Software testing — Part 11: Guidelines on the testing of AI-based systemsISO/IEC · 2020A technical report on the characteristics of AI-based systems that make them hard to test and on test approaches for them.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.)arXiv · 2023Research on using a strong model to grade other models' answers, reporting agreement with human preferences and biases such as position and verbosity.
• TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin, Hilton, Evans)arXiv · 2021A research benchmark of questions designed to reveal whether a model repeats common misconceptions instead of answering truthfully.
• Holistic Evaluation of Language Models (HELM)Stanford Center for Research on Foundation ModelsA research framework that evaluates language models on many scenarios and several metrics at once — accuracy, calibration, robustness, fairness and more.