LLM Testing: How to Test LLM Applications

What LLM testing is — and what it is not

Two different activities share the words "AI" and "testing". Testing AI systems means treating an LLM-based product as the system under test: a support assistant, a document summariser, a code reviewer, an agent that books appointments. Using AI to help QA means using a model as a tool inside the test process — drafting test cases, bug reports or test data. This guide is about the first. The second is useful, and it has one short section at the end, but it answers a different question.

LLM testing is still software testing. Requirements, risk analysis, test design, regression, defect reports and exit criteria all apply. What changes is the oracle — the way you decide whether an output is correct. A payment API either returns 201 with the right total or it does not; a model asked to summarise a contract can produce a hundred different acceptable summaries and a few subtly wrong ones. Most of the new techniques in this field are ways of building an oracle for open-ended output.

What makes LLM systems different to test

The differences below are why a conventional test suite, on its own, gives false confidence. ISO/IEC TR 29119-11 and the ISTQB CT-AI syllabus both describe characteristics of AI-based systems — such as non-determinism, lack of a precise specification and difficulty of defining expected results — that make them harder to test than rule-based code.

• Non-determinism — the same input can produce different outputs. Models generate text by sampling tokens from a probability distribution; parameters such as temperature and top-p control how much variation is allowed. Lowering temperature reduces variation, but in practice it does not guarantee identical output across calls, so tests must not depend on one exact string unless the output format forces it.
• Open-ended output — for most tasks there is no single correct answer, only answers that are acceptable against criteria (correct facts, required points covered, right tone, no disallowed content).
• Change you do not control — a hosted model can be updated or retired by its provider; a prompt edit of one sentence can change behaviour across unrelated cases. Every change to the model version, the system prompt, the retrieval settings or the tool definitions is a regression risk.
• Context windows — the model sees only what fits in its context: the system prompt, the conversation, retrieved documents and tool results. Long conversations and large documents truncate or dilute instructions, which creates defects that appear only at length.
• Tool use and agents — when the model can call functions (search, send e-mail, update a record), its output becomes an action. Testing then covers which tools it chooses, the arguments it passes, what it does with errors, and what it must never do without confirmation.
• Natural-language attack surface — any text the model reads, including a web page or an uploaded file, can contain instructions. This is why prompt injection sits first in the OWASP Top 10 for LLM Applications 2025.

Established, emerging, research: how to read this field

LLM testing moves fast, and much of what is published is experimental. Label every practice you adopt, so that a team does not treat a research metric as a release criterion. The labels used across these guides are:

• ESTABLISHED — regression suites of fixed test cases re-run on every change; exact-match, regular-expression and JSON Schema checks on structured output; human review against written rubrics; security testing against the OWASP Top 10 for LLM Applications 2025 (prompt injection, sensitive information disclosure, excessive agency, improper output handling and the rest); conventional performance, cost and timeout testing of the application.
• EMERGING — model-graded evaluation ("LLM-as-a-judge"), where one model scores another model’s output against a rubric; automated red-teaming, where a model or a script generates adversarial inputs at scale; production monitoring of sampled conversations with automatic scoring. Useful, widely adopted, but without agreed standards and with known weaknesses.
• RESEARCH — public benchmarks such as TruthfulQA and the HELM framework; RAG metrics proposed in research papers such as Ragas (faithfulness, answer relevance, context relevance). They inform how you think about quality; they do not tell you whether your product works for your users.

The map: what you test in an LLM system

Treat an LLM product as layers, each with its own tests. The model is usually a dependency you select and pin, not something you train. The prompt is code and is reviewed and versioned like code. The application around the model — retrieval, tools, guardrails, user interface, logging — is conventional software and is tested conventionally, with extra cases for the model’s unpredictability.

• Output evaluation — correctness, relevance, groundedness (faithfulness to the provided sources), hallucination (NIST AI 600-1 uses the term confabulation for confidently stated false content), consistency, robustness to paraphrased and noisy input, safety and toxicity, privacy and data leakage. The guide on LLM evaluation covers datasets, graders and rubrics in depth.
• Security — direct and indirect prompt injection, sensitive information and system prompt disclosure, excessive agency of tools, improper handling of model output by the rest of the application, unbounded consumption. The guide on LLM security testing gives benign, defensive test cases for each.
• Retrieval-augmented generation (RAG) — retrieval quality measured separately from generation, citation correctness, "I don’t know" behaviour when the corpus has no answer, freshness, and access control on retrieved documents. The guide on RAG testing covers each stage.
• Application behaviour — timeouts and retries when the model API is slow or fails, streaming and partial responses, token and cost limits, rate limiting, fallback messages, rendering of Markdown and links in the UI, logging without personal data, and right-to-left rendering of Arabic answers.

Prompt testing

A system prompt is a specification written in natural language, and it deserves the same scrutiny as a requirement: is every instruction testable, do any instructions conflict, what should happen outside the intended scope? For each instruction, write at least one test case that shows it being followed and one that tries to make the model break it with an ordinary, non-malicious request — a user who asks for something out of scope, writes in another language, or pastes a very long text.

Test the prompt with realistic variation, not only with the examples used while writing it. The same intent phrased formally and informally, with typos, in Arabic, in Hindi and in mixed scripts, with emoji and with special characters, should lead to the same decision. Keep prompts in version control, give each version an identifier that is logged with every response, and never edit a production prompt without re-running the evaluation set.

• Scope — out-of-scope questions get the agreed refusal or redirection, not an improvised answer.
• Format — the output follows the required structure (JSON, headings, length limit) for every case in the set.
• Language — the answer is in the user’s language when the prompt says so; Arabic answers render correctly right-to-left.
• Persona and tone — the required tone holds across long conversations, not only in the first reply.
• Robustness — paraphrases, typos, extra whitespace, Unicode look-alikes and very long inputs do not change the decision.

Try it: Multilingual Test Data Generator Try it: Unicode Dataset Generator Try it: Special Character Dataset Generator

How to assert on non-deterministic output

You cannot compare an open-ended answer to one expected string, but you can still write strict tests. The strongest technique is to constrain the output: ask the model for structured output (JSON) and validate it against a JSON Schema. The schema check is deterministic even when the wording inside the fields is not. Then assert on properties that every acceptable answer must have, and, for the remaining judgement calls, run each case several times and apply a threshold.

• Schema validation (established) — required fields present, types correct, enums respected, no extra fields. A schema failure is a hard defect regardless of wording.
• Exact match where the task allows it (established) — classification labels, extracted IDs, dates, amounts, yes/no decisions.
• Property-based checks (established) — the answer contains the order number from the input; it does not contain an e-mail address; it is under 120 words; every cited source ID exists in the retrieved set; the language is Arabic.
• Thresholds over repeated runs — run a case N times (for example 5 or 10) and require a pass rate (for example 9 of 10). Record N and the rate; a single green run of a non-deterministic case proves little.
• Rubric grading for the rest — human reviewers, or a model grader calibrated against human labels (emerging), score answers against written criteria.

{
  "id": "refund-policy-017",
  "input": "Can I return a laptop I bought 20 days ago?",
  "checks": [
    { "type": "json_schema", "schema": "schemas/support-answer.json" },
    { "type": "equals", "path": "$.intent", "value": "return_request" },
    { "type": "contains", "path": "$.answer", "value": "30 days" },
    { "type": "not_matches", "path": "$.answer", "regex": "[\\w.]+@[\\w.]+" },
    { "type": "max_words", "path": "$.answer", "value": 120 }
  ],
  "runs": 10,
  "min_pass_rate": 0.9
}

Try it: JSON Schema Validator Try it: JSON Diff

Regression evaluation on every prompt and model change

The single most valuable LLM testing practice is also the most conventional: keep a versioned evaluation set and re-run it on every change to the prompt, the model version, the retrieval configuration or the tool definitions. Compare results with the last accepted baseline, not with an ideal. A change that improves the average score while breaking three previously passing safety cases is a regression.

In a pipeline this looks like any other test stage. Deterministic checks (schema, exact match, property checks) run on every pull request and block the merge when they fail. Slower or costlier evaluations — repeated runs, model-graded rubrics — run nightly or before a release, with thresholds agreed in advance. Every result is stored with the model identifier, prompt version, dataset version and date, so a later drop in quality can be traced to the change that caused it.

• Pin the model version in configuration; treat a provider’s model update as a release that needs a full evaluation run.
• Store evaluation datasets as JSON Lines in the repository, with a version tag, and never tune the prompt on the same cases you use to accept it.
• Diff outputs between baseline and candidate for a sample of cases and read the differences — numbers hide what changed.
• Keep a small set of cases that must pass 100% of the time (safety, privacy, refusal of disallowed actions); treat any failure there as a blocker.

Try it: JSON Lines Validator Try it: Text Diff

AI quality metrics and their limits

Useful LLM metrics are tied to a defined task and a defined dataset. Task success rate (the share of cases where the user’s goal was met), rubric scores, pass rates of deterministic checks, groundedness rate (answers fully supported by their sources), refusal rate split into appropriate and inappropriate refusals, latency percentiles and cost per request are all meaningful when you know what they were measured on.

Be honest about limits. A score is only as good as its dataset: a 95% pass rate on 40 easy cases says little about production. Model-graded scores inherit the grader’s biases. A refusal rate that falls may mean the model got more helpful or that it got easier to misuse. Report metrics with the sample size, the dataset version and the grader used, and read failing cases, not only the averages.

Standards and risk frameworks

No standard prescribes a complete LLM test method, but several give structure and vocabulary. The NIST AI Risk Management Framework (AI RMF 1.0) organises AI risk work into four functions — Govern, Map, Measure and Manage; evaluation and testing sit mainly under Measure. Its Generative AI Profile, NIST AI 600-1, lists risks that are specific to or amplified by generative AI, including confabulation, data privacy, information integrity and information security, with suggested actions — a useful checklist when you decide what to evaluate.

ISO/IEC 42001 specifies requirements for an AI management system: it is the organisational context (policies, roles, risk assessment, monitoring) in which testing happens, not a test method. ISO/IEC TR 29119-11 is a technical report with guidelines on testing AI-based systems. The ISTQB Certified Tester AI Testing (CT-AI) syllabus covers testing AI-based systems, and the OWASP Top 10 for LLM Applications 2025 is the reference list for LLM security risks.

The other direction: using AI to help QA

Models are also useful inside the test process: turning a user story into draft test cases, suggesting edge cases, structuring a rough bug note into a report. Mutqan’s AI QA tools belong to this direction, and ISTQB’s CT-GenAI syllabus covers it. The same discipline applies in reverse: an AI-generated test case is a draft that a tester reviews for correctness, relevance and missing cases before it enters a suite. Do not use a model’s output as the oracle for testing that same model.

Try it: AI Test Case Generator

Frequently asked questions

References

• AI Risk Management Framework (AI RMF 1.0)NIST · 2023NIST's voluntary framework for managing AI risks across the lifecycle, organised into the Govern, Map, Measure and Manage functions.
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)NIST · 2024The generative-AI profile of the AI RMF: risks specific to generative models, such as confabulation and information integrity, and suggested actions to manage them.
• OWASP Top 10 for LLM Applications 2025OWASP Gen AI Security Project · 2025The ten most critical security risks of LLM applications (LLM01:2025–LLM10:2025), starting with prompt injection and sensitive information disclosure.
• ISO/IEC TR 29119-11:2020 — Software and systems engineering — Software testing — Part 11: Guidelines on the testing of AI-based systemsISO/IEC · 2020A technical report on the characteristics of AI-based systems that make them hard to test and on test approaches for them.
• Certified Tester AI Testing (CT-AI) v2.0ISTQB · 2026ISTQB's specialist syllabus on testing AI-based systems, covering machine-learning and generative-AI systems.
• ISO/IEC 42001:2023 — Information technology — Artificial intelligence — Management systemISO/IEC · 2023Requirements for an organisation's AI management system — the governance context in which AI systems are built, evaluated and monitored.
• Certified Tester Specialist Level – Testing with Generative AI (CT-GenAI)ISTQBISTQB's syllabus on using large language models across the test process — the other direction from testing AI systems.
• Define success criteria and build evaluationsAnthropicA model provider's guidance on writing measurable success criteria and evaluation sets graded by code, by exact match or by a model.