AI-Assisted QA

Why it matters

A language model produces fluent, plausible text. It has not seen your product, does not know your users or risks, and can state an invented detail as confidently as a correct one: the hallucinations you tested for earlier in this module, which the NIST Generative AI Profile (NIST AI 600-1) calls confabulation. A test case with an invented expected result is worse than none: it can pass against wrong behaviour or fail against correct behaviour.

There is a second risk: whatever you paste into an online AI tool leaves your machine, including a customer’s phone number in a bug note or an access token in a request log. With review and clean input, AI saves real drafting time; without them, it adds defects and data leaks.

An assistant for drafts, not a replacement for judgement

Think of an AI tool as a fast, well-read colleague who has never used your product and cannot be held responsible for anything. It is not a test oracle: expected results come from the test basis (requirements, acceptance criteria, designs) or from the product owner, never from the model’s guess. The ISTQB syllabus on testing with generative AI (CT-GenAI) treats this use of AI as a subject of its own, and the fundamentals stay the same: you choose what to test by risk, apply test design techniques, observe the real product and own the result.

• Good uses: test ideas and edge cases, first drafts of test cases from a clear story, ambiguity questions for a requirements review, structuring a bug report from your own notes, explaining an unfamiliar error message.
• Poor uses: deciding scope or priority, writing expected results the test basis does not state, setting severity without knowing the business impact, replacing exploration of the real product.
• Not acceptable: reporting a defect you did not reproduce yourself, or recommending a release because an AI summary says the results look fine.

Privacy: decide what you paste before you paste it

Most AI tools run as an online service: your input is sent to the provider to produce the answer. Before you paste anything, check your organisation’s policy on which AI tools are approved and which data may go into them. Personal data in a test artefact is still personal data; in Saudi Arabia, for example, its processing is governed by the Personal Data Protection Law (PDPL). The model needs the structure of the problem, not the real values, so clean the input first.

• Personal data (names, phone numbers, e-mail addresses, national IDs, card numbers): replace it with synthetic values from a test data generator.
• Secrets (passwords, API keys, access tokens, session cookies, Authorization headers): write <redacted> instead.
• Internal details (hostnames, IP addresses, customer names, unreleased features): leave them out unless the policy allows them.
• A real secret pasted by mistake must be reported to its owner and revoked; deleting the message is not enough.

Before cleaning: do not paste this
POST https://orders.int.example-shop.local/v2/checkout
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI0NDIxIn0.Xk2...
{"customer": "Sara Al-Harbi", "phone": "+966 55 012 3456", "code": "SAVE20"}
-> 500 Internal Server Error

After cleaning: enough for the AI to help
POST /v2/checkout
Authorization: Bearer <redacted>
{"customer": "Test User 01", "phone": "+966 50 000 0001", "code": "SAVE20"}
-> 500 Internal Server Error

Try it: Test Data Factory

Verify every draft before you use it

Treat AI output like any work product that has not been reviewed. Fluent wording is not evidence of correctness, and the same request can return a different draft next time, so review the version you actually use, in a fixed order.

• Trace: link each item to the test basis. An item you cannot trace is a gap in the requirement (a question for the product owner) or an invention (delete it).
• Check the facts: limits, error messages, status codes, field names and standards, against the requirement, the API contract or the product.
• Check that it can be executed: concrete data and an observable expected result. "Check that the feature works correctly" is not a test.
• Find what is missing: apply your own techniques (partitions, both sides of each boundary, decision rules, risks) and remove duplicates from the same partition.
• Generated code: run it, see each check fail when it should, and read every selector and assertion. Asking the same tool whether its output is correct is not verification.
• Record: mark the artefact as AI-assisted and name the reviewer.

Worked example: reviewing an AI draft for a discount code

Acceptance criteria of a checkout story: (1) SAVE20 gives 20% off orders of 200.00 SAR or more; (2) each customer can use a code once; (3) an expired code is rejected with the message "This code has expired". SAVE20 is valid until 23:59 on 31 December 2026, Riyadh time. An AI tool returned the draft below; review it item by item.

• TC-01: keep; it traces to criterion 1, and 250.00 minus 20% is 200.00.
• TC-02: keep the input and expect no discount, but the message is invented: criterion 1 does not say whether the code is rejected under 200.00 SAR or what the customer sees. Ask the product owner.
• TC-03: keep, with concrete data: SAVE20 at 00:00 on 1 January 2027, Riyadh time, in a test environment whose clock you can set.
• TC-04: the draft assumed codes are not case-sensitive; the requirement is silent, so this is a question.
• TC-05 and TC-06: discard. TC-05 has no data and no expected result; TC-06 is in the same partition as TC-01.
• Add: 199.99 SAR (no discount), 200.00 SAR (total 160.00), a second use by the same customer (rejected), a first use by another customer (accepted) and SAVE20 at 23:59 on 31 December (accepted). The draft missed criterion 2 entirely.
• Result: two kept, two questions, two discarded, five added. The AI saved typing; the coverage of the rules came from the review.

TC-01  Apply SAVE20 to an order of 250.00 SAR
       Expected: total 200.00 SAR
TC-02  Apply SAVE20 to an order of 150.00 SAR
       Expected: message "Minimum order is 200 SAR"
TC-03  Apply an expired code
       Expected: message "This code has expired"
TC-04  Apply "save20" in lower case
       Expected: code accepted
TC-05  Apply SAVE20 and check that the discount works correctly
TC-06  Apply SAVE20 to an order of 300.00 SAR
       Expected: 20% discount applied

Try it: Boundary Value Analysis

Using the Mutqan AI tools responsibly

The Mutqan AI QA tools cover test generation, requirement analysis, bug analysis and test optimisation: for example test cases from a user story, ambiguity detection, bug descriptions and severity suggestions. They need a free account and count against a daily allowance. Their privacy note says that your input is sent to the Mutqan AI service and that prompts and results are not stored; the input still leaves your device, so the rules on what you paste apply in full.

• Where a technique has fixed rules, prefer a non-AI Mutqan tool such as boundary value analysis: it runs in your browser and needs no sign-in.
• Do your own analysis first, then use the AI tool as a second opinion and compare the two.
• Give precise input: the story, its acceptance criteria, field limits and user roles.
• Treat a suggested severity or priority as an input to triage, not as the decision.

Try it: AI Test Case Generator Try it: AI Requirement Ambiguity Detector Try it: AI Bug Description Generator Try it: User Story → Test Cases

Frequently asked questions

References

• Certified Tester Specialist Level – Testing with Generative AI (CT-GenAI)ISTQBISTQB's syllabus on using large language models across the test process — the other direction from testing AI systems.
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)NIST · 2024The generative-AI profile of the AI RMF: risks specific to generative models, such as confabulation and information integrity, and suggested actions to manage them.
• Personal Data Protection Law (English translation)Saudi Data & AI Authority (SDAIA) · 2023The Saudi law governing personal data — why test environments should use synthetic data rather than copies of real customer records.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.