Prompt Testing

Why it matters

Prompts change more often than almost any other part of an LLM application, and a one-sentence edit can change behaviour in cases that seem unrelated. Nothing compiles or type-checks natural-language instructions: a prompt that reads better can still drop a policy condition, switch the answer language or stop escalating complaints. Without a test set and a recorded baseline, such regressions reach users unnoticed.

For the tester, prompt testing turns "the new prompt seems better" into evidence: which cases improved, which broke, on which model and prompt version. It also makes LLM defects easier to reproduce, because a report that names the prompt version, the model and the input can be re-run.

What exactly is under test

The guide on LLM testing showed that a system prompt is a specification written in natural language. What you version and test is slightly larger: everything your team controls that shapes the answer, treated as one configuration with one identifier, so every result can be traced to what produced it.

Each instruction is a test condition: write one case where following it is required and one where an ordinary user makes it hard to follow. Partitions, boundaries and decision rules from test design still apply.

• The system prompt: role, scope, rules, tone and output format.
• The template and its variables, such as {customer_name}, filled at run time — test them empty, very long and in other scripts.
• Few-shot examples, the sample inputs and ideal answers written into the prompt to show what is expected: changing or reordering them is a prompt change.
• The model identifier and parameters such as temperature: the same text can behave differently with another model version or other settings.

Building a test set of inputs

A prompt test set is a versioned file of test cases, each with an input, the variable values and the checks every acceptable answer must pass. It is testware: reviewed, stored in the repository and changed on purpose. Keep a held-out part used only to accept a version; a prompt tuned until it passes its own acceptance cases looks better on paper than it is.

Draw the cases from several sources. The excerpt below is YAML for readability; teams often store the same structure as JSON Lines.

• Instructions: a case that follows each one and a case that puts it under pressure.
• Real requests: frequent intents from logs or support tickets, with personal data removed.
• Variation: formal and informal wording, typos, Arabic, Hindi and mixed scripts, very short and very long inputs.
• Out of scope: questions the assistant must decline or redirect.
• Critical cases: safety, privacy and hand-off rules that must pass every time.
• Past failures: every prompt defect you fix becomes a case.

# support-set v7: 3 of its 40 cases
- id: DEL-004
  input: "متى يصل طلبي رقم 5521؟"
  vars: { order_status: shipped }
  checks: [valid_json, language_is_ar, mentions_order_5521]
- id: CMP-002
  input: "मेरी केतली टूटी हुई आई है"
  checks: [valid_json, language_is_hi, handoff_is_true]
  critical: true
- id: OOS-001
  input: "Can you book me a flight to Riyadh?"
  checks: [valid_json, declines_politely, no_invented_service]

Try it: JSON Lines Validator Try it: Multilingual Test Data Generator Try it: Unicode Dataset Generator

Versioning prompts

A prompt that can be edited in place with no record, for example in an admin screen, cannot be tested reliably: nobody can say which text produced which answer. Keep each prompt as a file in version control, change it through review, and give every released version an identifier that the application logs with each response.

Record each accepted version with the conditions it was tested under: model identifier, parameters, test set version, date and results. A provider’s model update or a new temperature is a new configuration that needs a new run, even if the prompt text is unchanged.

• One change per version, with its reason: "v4: keep answers under 60 words; customers found replies too long".
• Never edit an accepted version in place; create the next one, so a rollback is just a switch to the previous identifier.
• Diff the two versions before testing to see which instructions changed and which cases should be affected.

Try it: Text Diff

Prompt regression: compare with the baseline, case by case

A prompt regression is a case that passed with the accepted version and fails with the new one. You find it by comparing the candidate with the baseline: the stored results of the last accepted version on the same test set, model and parameters. Only an accepted version’s results become the new baseline. Change only the prompt; if the model changes too, you cannot tell which change caused a difference.

Run every case. Re-running the cases that failed because of the problem the change is meant to fix is confirmation testing; re-running all the others is regression testing, and you need both. Run each case several times with a pass rule agreed in advance, then sort it into one of four groups:

• Fixed: failed with the baseline, passes now — the intended improvement.
• Regressed: passed before, fails now. A regression in a critical case blocks the change; any other needs investigation and the team’s explicit acceptance.
• Still failing: failed with both — a known problem this change did not cause.
• Unchanged pass: passed with both. Still read a sample side by side; tone or content can drift while the checks pass.

Try it: JSON Diff Try it: JSON Schema Validator

Worked example: a shorter answer that broke two cases

Version v3 of the support assistant’s prompt is the baseline. Customers found its answers too long, so v4 adds: "Keep every answer under 60 words." Both versions run support-set v7 (40 cases) on the same pinned model, 5 runs per case; a case passes when all its checks pass in at least 4 runs, and critical cases in all 5.

Passing cases rise from 34 to 37, which looks like an improvement. Case by case, five verbosity cases are fixed, but RET-012 now drops the 30-day return condition, and the critical case CMP-002 fails to hand a damaged-item complaint to a person in 2 of 5 runs; a text diff of the two RET-012 answers shows the missing sentence. The critical regression blocks v4. The team writes v5 — "Keep every answer under 60 words, but always state policy conditions and never skip a hand-off" — and runs the whole set again.

Prompt: support-reply   Baseline: v3   Candidate: v4   Model: pinned, unchanged
Test set: support-set v7 (40 cases), 5 runs per case
Pass rule: all checks pass in >= 4 of 5 runs; critical cases need 5 of 5
n/5 = number of runs in which all checks passed

Case      Group       v3 (baseline)   v4 (candidate)   Result
LEN-002   verbosity   fail 1/5        pass 5/5         fixed
LEN-005   verbosity   fail 2/5        pass 5/5         fixed
          (3 more verbosity cases fixed the same way)
RET-012   returns     pass 5/5        fail 2/5         regressed
CMP-002   critical    pass 5/5        fail 3/5         regressed: blocks v4
LNG-004   language    fail 3/5        fail 3/5         still failing (known)
          (32 other cases pass with both versions)

Passing cases         34 / 40         37 / 40

Try it: Text Diff

Frequently asked questions

References

• Define success criteria and build evaluationsAnthropicA model provider's guidance on writing measurable success criteria and evaluation sets graded by code, by exact match or by a model.
• Working with evalsOpenAIA model provider's guidance on testing model outputs against defined criteria and datasets, and re-running evals as prompts and models change.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.