AI Testing Fundamentals

Why it matters

AI features can fail without a single wrong line of code. A model can pass every demo and still misroute most tickets written in a language that was rare in its training data, answer differently after a retraining nobody announced, or slowly get worse as customer behaviour changes. A test suite that only checks code paths stays green through all of it.

For the tester the risk runs both ways: approving a release on evidence that proves less than it seems, such as a few hand-picked examples or a score measured on unrepresentative data, or being unable to say whether a wrong answer is a defect at all. Knowing what is different lets you collect evidence a team can trust and write defect reports a data scientist can act on.

Rule-based software and ML systems

In rule-based software every behaviour traces back to a requirement and to code that a person wrote and can read. For any input you can work out the expected result from the specification, and when a test fails, the defect is in the code, the configuration or the requirement.

In an ML system the model’s behaviour is learned. Developers write the training pipeline and the code around the model, but nobody writes the rule that sends a given ticket to Billing: it comes from patterns in the training data, stored as numeric parameters that cannot be read as rules. This is data-dependent behaviour: the same code trained on different data is a different system.

• Keyword rules: a ticket saying "I was charged twice" goes to Other, because it does not contain "refund", although it is a billing complaint. The defect is a missing rule, and the fix is a new rule.
• ML model: the same ticket probably goes to Billing, because similar tickets were labelled Billing in the training data. A new kind of complaint with no similar examples can still be misrouted, and the fix is better data, not a new line of code.
• Many products combine both: the model proposes, and rules decide what happens next, such as sending low-confidence results to a person.

Probabilistic and non-deterministic behaviour

Rule-based code is normally deterministic: the same input in the same state gives the same output. ML systems add two kinds of uncertainty. They are probabilistic: a classifier returns a score and is right for a share of the inputs, so one wrong answer can be expected behaviour while the error rate stays within the agreed limit. They can also be non-deterministic: the same input can give different outputs.

For tests this means: record the model version, data version and settings with every result; when an output can vary, run the case several times and judge the pass rate; and test the code around the model deterministically by replacing the model with a stub that returns fixed scores. To see whether two outputs for the same input differ in meaning or only in wording, compare them side by side, for example with Text Diff. Given the same input, a fixed model version usually computes the same scores, so the variation comes from these sources:

• Sampling: generative models such as LLMs pick each next token, a word or part of a word, from a probability distribution, and settings such as temperature control how much the output varies.
• Retraining: training usually involves random choices, such as starting values and the order of the data, so a model retrained on the same data can answer some inputs differently.
• Changes you do not control: a provider updates a hosted model, or a self-learning system changes as it learns from new data.
• Small numeric differences between hardware or library versions, enough to move a score across a threshold.

Try it: Text Diff

Training data is the source of behaviour

Training data plays the part in an ML system that requirements and code play in rule-based software. Whatever patterns it contains become behaviour, including its gaps, its mistakes and its biases. If almost all training tickets were written in English, the classifier handles Arabic tickets poorly, although no line of code mentions language. If the people who labelled the data filed delivery complaints under Billing, the model does the same.

The same applies to the test data used to evaluate the model: a score describes only the population that data represents. Test data must therefore be representative of the operating environment, the inputs and conditions the system meets in production: languages and dialects, channels, devices, regions, time periods and rare but important cases, in realistic proportions. A test set that is 95% English says little about a product whose customers mostly write in Arabic, however high the score. Before you trust an evaluation, ask:

• Where and when was the data collected, and does that match where and when the system will run?
• Which groups does it contain, and in what proportions compared with production?
• Who labelled it, following which written guideline?
• Was the test data kept apart from the training data? The next lesson shows how overlap between them inflates scores.

What is different to test

ISO/IEC TR 29119-11 and the ISTQB AI Testing (CT-AI) syllabus describe characteristics that make AI-based systems harder to test than rule-based ones, and the NIST AI Risk Management Framework lists similar differences between AI risks and traditional software risks. In daily work, six things change:

• The expected result is often unknown. For many inputs nobody can write the exact correct output in advance: this is the test oracle problem. Testers use other oracles instead: labels that people agree on for a test set, and partial oracles that check only part of the output, such as ranges of acceptable answers and rules the output must never break.
• Acceptance criteria are statistical. "At least 90% of tickets routed correctly on a representative test set, and at least 85% in each language" replaces "this ticket goes to Billing". One passing case proves little, and one failing case may be within the agreed error rate, unless it is a must-pass case that may never fail.
• Defects can live in the data and the model set-up: wrong labels, missing groups, an input field that gives the answer away, a badly chosen threshold. Report them like any other defect, with the model and dataset versions as part of the environment.
• A fix can move behaviour anywhere. Retraining to fix one group of inputs can change the answers for others, so every new model version runs the whole test set and is compared case by case with the previous version.
• Quality can change after release. Real inputs drift away from the training data, so a model that met its criteria at release can get worse without anyone changing it, and testing continues as monitoring in production.
• More quality characteristics matter: bias, robustness to small changes in the input, transparency and explainability. Later lessons in this module cover them.

Worked example: testing an ML ticket router

A support team replaces its keyword rules with an ML classifier that reads each ticket, in Arabic or English, and returns a category and a confidence between 0 and 1. Tickets below 0.70 go to a human triage queue. The tester splits the plan into conventional checks, with the model replaced by a stub, and model checks on recent labelled tickets.

Most of the plan is ordinary testing. C1 is boundary value analysis on the threshold: scores have two decimals, so it tests the values on each side of 0.70. C2 to C4 are negative, error-handling and edge-case tests, and C5 makes every decision reproducible. Mixed-language inputs for C4 can come from a test data generator; they test the code path, not the model’s accuracy, because synthetic text does not represent real customers. The test levels still apply: the model is evaluated as a component, its link to the help desk at integration level, and the whole flow at system and acceptance level, with the performance and security checks of non-functional testing.

What is new is M1 to M5: criteria over a set, results per language, a must-pass group, a check that the test data still looks like production, and a consistency check. M1 has two limits because a model that sends almost every ticket to the human queue could pass an accuracy limit alone. The testing principles hold too: a high score does not prove that the router works for every customer, because testing shows the presence of defects, not their absence, and exhaustive testing of free text is impossible, which is why representative test data matters so much.

Feature: route support tickets with an ML classifier (Arabic and English)
Output: category + confidence 0.00..1.00 (2 decimals); below 0.70 -> human triage queue

Conventional checks (model replaced by a stub that returns fixed results)
  C1  stub returns 0.69 -> human queue; stub returns 0.70 -> category queue
  C2  model service gives no answer within 3 s -> human queue, error logged
  C3  empty ticket -> validation error, model not called
  C4  ticket mixing Arabic and English -> routed or queued, never an error
  C5  every decision stores ticket id, model version and confidence

Model checks (labelled test set: 2,000 tickets from the last 3 months)
  M1  >= 90% of auto-routed tickets in the correct category; <= 25% to the human queue
  M2  >= 85% for Arabic tickets and for English tickets, measured separately
  M3  0 of 40 "account locked" tickets routed to Other (must-pass)
  M4  language mix of the test set within 5 points of production
  M5  identical results on two runs of the same model version

Try it: Boundary Value Generator Try it: Multilingual Test Data Generator

Frequently asked questions

References

• Certified Tester AI Testing (CT-AI) v2.0ISTQB · 2026ISTQB's specialist syllabus on testing AI-based systems, covering machine-learning and generative-AI systems.
• ISO/IEC TR 29119-11:2020 — Software and systems engineering — Software testing — Part 11: Guidelines on the testing of AI-based systemsISO/IEC · 2020A technical report on the characteristics of AI-based systems that make them hard to test and on test approaches for them.
• AI Risk Management Framework (AI RMF 1.0)NIST · 2023NIST's voluntary framework for managing AI risks across the lifecycle, organised into the Govern, Map, Measure and Manage functions.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.