End-to-End Product Quality Assurance

Step 1, R1 and R2. R1 looks testable, which is why its gaps are easy to miss. Questions for Reem: are 10 and 5,000 SAR both allowed, as the wording suggests, and may a donor give halalas, such as 10.50? When is a donation charged in a month that does not have its day — one started on the 31st, in a 30-day month, or one started on the 29th, 30th or 31st, in February? Is the first donation charged at set-up? And what happens when a monthly charge fails? With whole riyals and both ends allowed, the 2-value boundary values are 9 (rejected), 10 (accepted), 5,000 (accepted) and 5,001 (rejected); if halalas are allowed the step is 0.01, so the values are 9.99, 10.00, 5,000.00 and 5,000.01. For the charge day, start donations on the 28th, 29th, 30th and 31st and follow them through a 30-day month and through February. R2 fails the guide’s question outright: nobody can say whether a page is "fast" during a "campaign", so it can be neither passed nor failed. Questions for Reem: which pages and API calls count as donation pages, and how many donors were giving at the same time at the peak of the last campaign? Then rewrite it with numbers Reem agrees, taken from the traffic of the last campaign, for example: "With 500 donors completing a donation at the same time, the 95th percentile (p95) response time of the donation page and of the donation API is at most 2 seconds, and fewer than 1% of requests fail." The figures are only an example; the form — load, percentile, limit, error rate — is what makes the requirement testable.

Step 1, R3. It describes the success path only, while the guide notes that most production incidents live on the error paths. Acceptance criteria with two error paths: (a) Given a donor with an active monthly donation of 100 SAR, when the monthly charge succeeds, then exactly one charge of 100.00 SAR is recorded and one receipt e-mail with the amount, the date and a reference number is sent in the donor’s language. (b) Given the same donor, when the payment provider declines the charge, then no receipt is sent, the donation is marked "Payment failed" and the donor is asked to update the card. (c) Given a charge that succeeded, when the receipt e-mail cannot be sent, then the card is not charged again and only the e-mail is retried. Criterion (c) is D-71: written before development, it would have been a question in refinement, not a Critical defect one week before go-live. How many retries, and when a donation is paused, are Reem’s decisions; your criteria make her take them. All of this is a review — static testing in ISTQB terms — done on text, before any code runs.

Step 2, the quality characteristics. Functional suitability: wrong amounts, charge dates or receipts (R1, R3); boundary value and negative tests, mostly at the integration level through the API, with the charge-date calculation also covered by the developers’ unit tests. Reliability and performance efficiency: a charge that fails, times out or repeats — the risk D-71 has already shown — and the campaign peak three weeks after go-live; failure tests at the API level with the payment provider and the e-mail service mocked so that they return a decline, a timeout or a send failure, and a load test against the R2 targets on an environment configured like production, a system-level activity with its own slot in the plan. Security: donors’ personal data and their payments; the OWASP baseline on staging with two test donor accounts — can one donor see or cancel another donor’s monthly donation by changing an id in an API request (access control), and are session handling and the security headers right — at the API and system levels. Compatibility and usability: an Arabic-first product with a right-to-left layout, Arabic names in receipts, phones and the main browsers; system tests and an exploratory session in the Arabic UI with Arabic test data, which would have found D-74 and D-75 before the demo. Maintainability: a later change to the charge or receipt logic that silently breaks recurring donations, while the only API regression check runs by hand once per release; the API collection in CI as the regression suite, at the integration (API) level, so that the next change can be made safely. Above all of these sits acceptance testing: Reem runs or watches the acceptance criteria of R1–R3 on staging, and the release decision rests on it.

Step 3, the practices against the checklist. P1: acceptance criteria come after development and cover the success path only; write them before development, error paths included, and review them with Reem. P2: rank the release’s features by risk, likelihood times impact — recurring charges and receipts first — and let the test plan follow the ranking. P3: unit tests on every commit are good practice; the problem is only the metric (step 4). P4: the API collection is the team’s best regression asset, and it runs by hand, once, before a release; put it in CI on every change, with assertions on status codes and error bodies, and make a green run a release condition. P5: re-running tests until they pass makes "green" mean nothing; quarantine the five flaky tests, fix or remove them, and count only a green first run. P6: exploratory, performance and security testing each need their own scheduled activity (step 2); none of them happens as a side effect of functional testing. P7: Latin-only test data cannot find Arabic defects; add Arabic names with diacritics, long names and input that mixes Arabic and English. P8 is the most serious: real donors’ personal data sits in an environment open to everyone with test access, which the Saudi Personal Data Protection Law treats as processing that needs a basis and safeguards; and with real e-mail addresses on staging, every test of the receipt feature can e-mail a real donor. Stop the weekly copy, remove the copied data and generate synthetic donors in Arabic and English, kept as a known baseline dataset the suite can reset to. Note that D-74 surfaced only through Reem’s real account, copied from production, because no test account has an Arabic name; synthetic Arabic test data finds the same defect without anyone’s personal data. P9: mock the payment provider at its boundary so that a test can choose a decline, a timeout or a late answer, and keep a few sandbox runs for the real integration. P10: give every defect a priority as well as a severity, and write entry and exit criteria (step 5).

Step 4, the numbers. Escape rate of release 1.9: 6 ÷ (18 + 6) = 6 ÷ 24 = 25%, so one in four of the defects known 30 days after the release was found only in production; the defect detection percentage (DDP), its complement, is 75%. Some teams state the escape rate as a ratio instead, 6 production defects to 18 found before release (one escaped for every three caught); either form works if the report says which one it uses and keeps it from release to release. Requirement coverage of release 2.0: 14 of the 20 acceptance criteria have a passing test, which is 70%. 16 of 20 (80%) have a test of some kind, but a criterion whose test fails is not yet covered, and 4 are simply untested. "Line coverage 72%" says which lines the unit tests executed, not whether any requirement works: the guide calls it a developer metric, not a quality metric. Keep it for the developers and report three numbers at every release instead: the escape rate, requirement coverage, and regression suite health — today a 45-minute UI run in which 5 of 38 tests (about 13%) are flaky, and an API collection that is not in CI at all.

Step 5, exit criteria and the answer. Exit criteria for release 2.0, written down and agreed with Reem: every planned test has been executed; every acceptance criterion has a passing test; no open Blocker or Critical defect; the API and UI regression suites green on their first run; the R2 performance targets met on the production-like environment; and any open Major defect accepted by Reem in writing, with a workaround. Applied to rc1, none of them is met: the API collection has not run on rc1 and 4 criteria have no test; 2 criteria fail; D-71 is an open Critical defect; the UI suite passed only on its third run; performance has neither a target nor a measurement; and D-74 is an open Major defect with neither acceptance nor a workaround. The recommendation for rc1 is therefore no-go. Answer Reem in writing: "Not with rc1. Go-live on 1 November is still possible if, by Thursday 29 October, D-71 and D-74 are fixed and pass confirmation testing, a new release candidate passes the API collection and the UI suite on their first run, the 6 open criteria have passing tests, and a load test meets the R2 targets you agree. D-75 can ship as a known issue." D-74 belongs on the must-fix list even though it is Major: in an Arabic-first product, most receipts would carry a name nobody can read. For the next candidate, add entry criteria: the build deploys to staging, the smoke test passes, and every story in it has acceptance criteria. The release decision remains Reem’s; your part is to make the risk visible and written down. If the conditions cannot be met in time, a later go-live that is still before the campaign costs less than double charges during it.

Step 5, the three changes to make first, in order. First P8, synthetic data instead of production copies: it is a legal and privacy exposure every week it continues, whatever happens to the release. Second P4, the API collection in CI on every change and in the exit criteria: one of the two starting points the guide gives a small team, and the level where D-71’s scenario — a failed e-mail after a successful charge — can be tested with a mocked e-mail service. Third P1, acceptance criteria with error paths before development: the guide’s other starting point, and R3 shows what it costs when they come later. Performance and security targets (P6) follow straight after, before the campaign, and an exploratory session per new feature once these three are in place. Another order is fine if you defend it with risk; leaving P8 for later is not.

How to check your own work. Your answer should contain: at least one question for each requirement, and numbers in R2; an error-path criterion for R3 that matches D-71; a test level for every activity in step 2; P8 named as a personal data problem, not a convenience; 25% (or the ratio 6 to 18, labelled as a ratio) and 70% with the formulas behind them; and a no-go for rc1 together with the conditions that would turn it into a go. Different example figures for R2, a different first change or extra exit criteria are fine when the reason is written down.