Hallucination and Groundedness Testing

Why it matters

Hallucinated answers are fluent and confident, so people act on them: they rely on a deadline, a price or a policy that does not exist. The OWASP Top 10 for LLM Applications 2025 lists misinformation (LLM09) as a risk and names hallucination as one of its main causes. For the product, that means wrong decisions, complaints and liability.

For a tester, hallucination breaks the usual oracle: the output looks plausible, changes between runs, and one passing run proves little. A language model generates likely text, not checked facts, so no prompt makes hallucination impossible. Your job is to measure how often and where it happens and to verify the safeguards: sources, citations, refusals. Testing shows the presence of defects, not their absence.

Two reference points and the shapes hallucination takes

First decide what you check an answer against. In a closed-book setting the model answers from its training, so you need an external reference. In a grounded setting — retrieval-augmented generation (RAG), an uploaded document, a tool result — the system hands the model its sources, and you check the answer against exactly those.

The two checks can disagree. An answer can be true but ungrounded: the model answered from memory, which works until its memory is wrong or outdated. It can also be grounded but false: it faithfully repeats an outdated document — a data defect, not a model defect.

• Fabricated entities: a product, policy section, API function, person or URL that does not exist.
• Wrong specifics: numbers, dates, prices or names altered, or mixed up between sources.
• Contradiction: the answer says the opposite of its source.
• Unsupported additions: a plausible extra promise, such as "return shipping is free", that no source contains.
• Accepted false premises: explaining "why the Basic plan includes phone support" when it does not.

Groundedness: check the answer claim by claim

A whole answer is too coarse to judge. Split it into atomic claims — single statements that are either supported or not — and label each against the sources: supported, contradicted or not found. "Not found" matters as much as "contradicted": an unsupported claim is a hallucination even when it happens to be true.

Fix the context in your test cases: give the model a known set of passages, not whatever retrieval returns today, so that a failure points at generation and can be reproduced.

A groundedness rate shows trends across a test set. Per claim, it is supported claims divided by all factual claims, as the Ragas research paper computes its faithfulness score with a language model as the checker; some teams count fully supported answers instead. For a single high-stakes case, use an acceptance rule: one contradicted or not-found claim fails it.

• Code checks what is exact: numbers, IDs and quoted strings appear in the source; every cited ID exists and was provided. When answers are structured JSON, a schema can require a citation for every claim.
• Human reviewers judge meaning with a written rubric. A model-based grader can scale this only once its labels match human labels on your data, because it can hallucinate too.

Try it: Text Diff Try it: JSON Schema Validator

Worked example: a returns assistant

A shop’s support assistant answers from the returns policy, which the system passes to the model as its only context. The honest answer to the test question is "no": questions that invite a friendly "yes" from general knowledge are where grounding breaks.

The labelled answer below fails although part of it is correct. A grounded answer: "No. Sale items cannot be returned (rule 3). Other unused items can be returned within 14 days of delivery (rule 1)."

Context given (policy-returns v3):
  "1. Unused items can be returned within 14 days of delivery.
   2. Refunds go to the original payment method within 5 business days.
   3. Sale items cannot be returned."

Question: Can I return a sale item I bought last week?

Answer under test:
  "Yes. You can return any item within 30 days of delivery, as stated
   in section 4 of our policy. The refund goes back to your original
   payment method within 5 business days."

Claim                                     Label          Source
C1  A sale item can be returned           contradicted   rule 3
C2  The return window is 30 days          contradicted   rule 1
C3  The policy has a section 4            not found      -
C4  Refund to original payment method     supported      rule 2
C5  Refund within 5 business days         supported      rule 2

Result: FAIL - 2 of 5 claims supported, 2 contradicted, 1 not found

Fact-checking when there is no single source

Closed-book answers, and the general-knowledge parts of grounded answers, need a different oracle. Build it before you test, or the most confident answer will set your expectation; store it in the test set, for example as one JSON Lines record per question.

• Reference answers verified by a domain expert, listing the facts that must and must not appear.
• Authoritative sources: the product’s database, official documentation or the standard itself — never another model’s answer.
• Deterministic checks: links resolve, cited documents exist, totals recompute, named API functions exist.
• Consistency as a signal: ask the same question repeatedly and in paraphrased forms; facts that change point to guessing. But a model can also repeat the same wrong answer every time.
• Trap questions: a near-identical product, an old version, or a detail that sounds standard but differs in your domain.

Try it: JSON Lines Validator Try it: JSON Diff

Unanswerable questions: test the right to say "I don’t know"

Add unanswerable questions to your test set with the expected behaviour: a clear statement that the information is not available, no invented detail, no citation and, where offered, a route to a human.

Track two rates together: invented answers to questions that cannot be answered, and inappropriate refusals of questions that can. A system that refuses everything never hallucinates and is useless, and a prompt change that lowers invented answers can raise refusals.

• Not in the sources: plausible questions that the documents do not cover.
• False premise: the question assumes something untrue; the answer should correct it.
• Partly answerable: answer what the sources support and say what is missing.
• Answerable controls: these must be answered; a refusal here is an inappropriate refusal.

Frequently asked questions

References

• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)NIST · 2024The generative-AI profile of the AI RMF: risks specific to generative models, such as confabulation and information integrity, and suggested actions to manage them.
• OWASP Top 10 for LLM Applications 2025OWASP Gen AI Security Project · 2025The ten most critical security risks of LLM applications (LLM01:2025–LLM10:2025), starting with prompt injection and sensitive information disclosure.
• Ragas: Automated Evaluation of Retrieval Augmented Generation (Es et al.)arXiv · 2023Research proposing reference-free metrics for RAG pipelines, such as faithfulness, answer relevance and context relevance.
• TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin, Hilton, Evans)arXiv · 2021A research benchmark of questions designed to reveal whether a model repeats common misconceptions instead of answering truthfully.