Risk-Based Testing

Product risk vs project risk

The ISTQB Foundation syllabus separates two kinds of risk, and mixing them is the most common reason a risk register becomes useless. A product risk (related to the product’s quality characteristics) is a way the product itself could fail to meet a need: a wrong total at checkout, a slow search page, a data leak between accounts. A project risk is a way the project could fail to deliver: the test environment arrives late, the only person who knows the payment integration goes on leave, a supplier ships the API a sprint late.

Testing mitigates product risks directly — a test is a way to find out whether the feared failure is present. Project risks are managed by the test manager and the delivery team through planning, staffing and escalation; they belong in the test plan, but they are not reduced by writing more test cases. Keep them in separate lists with separate owners.

• Product risk examples: incorrect VAT calculation, session not invalidated on logout, Arabic text overflowing a button, report timing out at month-end volumes.
• Project risk examples: staging environment shared with another team, late delivery of test data, no access to the payment provider’s sandbox, unclear acceptance criteria.

How to identify risks

Risk identification works best when several perspectives are in the room, because each role sees different failures. A developer knows which module is fragile, a product owner knows which failure would reach the news, support knows what customers already complain about, and a tester knows where defects clustered last time.

• Risk workshop — 45 to 90 minutes with product, development, testing, operations and, where relevant, security and support. Walk the feature flow step by step and ask at each step: what could go wrong here, and who would notice?
• Checklists — reuse a list of risks from similar features (payments, authentication, file upload, reporting) so the workshop does not start from a blank page.
• Quality characteristics — use ISO/IEC 25010 as a prompt list: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability, flexibility and safety. Ask of each: does a failure here matter for this feature?
• Past defects — group the defects from previous releases by component and root cause; areas with many escaped or reopened defects are likely to fail again.
• Change impact — what is new or modified in this release, what depends on it, and how complex the change is. New code, rewritten code and code touched by many people carry higher likelihood.
• Interviews and expert review — for regulated or specialised domains, ask the people who know the rules (finance, compliance, clinical staff) what an unacceptable failure looks like.

Try it: AI Risk-Based Test Generator

Assessing risk: likelihood × impact

Each identified product risk gets two ratings. Likelihood is how probable it is that the failure is present and will occur in use; it rises with complexity, novelty, poor requirements, time pressure, many integrations and a history of defects. Impact is how bad the failure would be if it occurred: financial loss, legal or regulatory exposure, safety, number of users affected, reputational damage, availability of a workaround. The risk level combines the two, most often as a product of two ordinal scores.

Use qualitative scales with written definitions so two people score the same way. A 1–4 scale is usually enough; finer scales invite arguments about the difference between a 6 and a 7 that nobody can settle. Agree the definitions before the workshop, not during it. A 1–5 scale works the same way — the Bustan Market examples in the test strategy and test plan guides use one — as long as each register uses a single scale with written definitions and fixed bands, and scores from different scales are never compared. The Risk-Based Test Generator linked below uses such a different scale: likelihood and impact from 1 to 5, bands Low (below 4), Medium (4–8), High (9–14) and Critical (15–25), and no impact floor. Rate risks again on its scale rather than entering the 1–4 scores from this page, and apply your own impact rule to its output if your strategy has one.

Likelihood                      Impact
1 Rare     – simple, unchanged,   1 Low      – cosmetic, easy workaround
             well tested          2 Medium   – feature degraded, some users
2 Possible – moderate change      3 High     – core flow broken or wrong data
3 Likely   – complex or new code  4 Severe   – money, legal, security or
4 Very likely – new + complex +                 safety consequence
             weak requirements

                     IMPACT
               1 Low   2 Medium  3 High   4 Severe
LIKELIHOOD 4 |  4 M  |   8 H   |  12 VH |  16 VH |
           3 |  3 L  |   6 M   |   9 H  |  12 VH |
           2 |  2 L  |   4 M   |   6 M  |   8 H  |
           1 |  1 L  |   2 L   |   3 L  |   4 M  |

Bands: 1–3 Low · 4–6 Medium · 8–9 High · 12–16 Very high
Rule: any risk with Impact 4 is at least High, whatever the product.

Try it: Risk-Based Test Generator

From risk level to test priority and test depth

Risk level decides two things: the order of testing (high risks first, so that if time runs out the untested work is the least dangerous) and the depth of testing (how many techniques, how much coverage and how much independence). Write the mapping down in the test strategy so that depth is not renegotiated feature by feature.

• Very high — several design techniques combined (boundary value analysis, decision tables, state transition testing), negative and error-path tests, non-functional checks where the risk is non-functional, automated regression at API level run on every build, an exploratory session by a second tester, and review of the tests themselves.
• High — boundary and negative tests on every input that matters, all acceptance criteria covered, automated regression on every build or nightly, one exploratory session.
• Medium — all acceptance criteria covered with the main negative cases; regression in the nightly or release suite; exploratory testing if time allows.
• Low — happy path and acceptance criteria checked, or covered only by smoke tests; the decision to test lightly is recorded, not silent.
• Order — start with Very high risks as soon as a testable build exists; test Low risks last and cut them first.

Try it: Requirements → Test Cases

Using risk to select regression tests

A full regression suite rarely fits the release window, so every release selects. Risk-based regression selection combines change impact analysis with the risk register: tests on the changed code and on what depends on it run first, weighted by the risk level of the affected areas.

• List what changed — commits, pull requests, release notes, database migrations, configuration and dependency upgrades.
• Trace dependencies — which screens, APIs, jobs, reports and shared components call or read what changed; a shared utility or a database table touches more than its file name suggests.
• Always run the core — a fixed set of smoke and critical-path tests (sign-in, payment, the main business transaction) runs on every build regardless of the change.
• Add impacted tests — tests traced to changed and dependent areas, highest risk first.
• Fill the remaining time — by risk level of untouched areas, rotating Medium and Low areas across releases so none goes untested for long.
• Record what was not run — the unselected tests are part of the residual risk statement.

Try it: Regression Scenario Generator Try it: AI Regression Test Generator

Worked example: risk register for a checkout feature

For example, a fictional online store adds a new checkout with saved cards, discount codes and 3-D Secure. The workshop produced the register below. Scores use the 1–4 scales above, not the 1–5 scale of the Risk-Based Test Generator linked below; typing these numbers into the generator gives different levels (R4 would come out Medium, because the generator has no Impact-4 rule). The test response is what the plan commits to for each risk.

• R1 and R2 get most of the effort: two techniques each, automated API tests in the pipeline, and an exploratory session on payment failures.
• R4 has a low likelihood but a severe impact; its score of 4 alone would make it Medium, and the Impact-4 rule lifts it to High so that the score cannot hide a potential data breach.
• R7 gets one test case; if the schedule slips, it is the first thing cut, and the cut is written into the residual risk statement.

Feature: Checkout v2 (saved cards, discount codes, 3-D Secure)   Release 5.3

ID  Risk                                            L  I  Score Level  Test response
R1  Card charged but no order created               3  4  12    VH     State transitions of payment; gateway
    (gateway timeout after capture)                                     timeout/5xx via mock; retry and
                                                                        reconciliation check; exploratory
R2  Wrong total when discount and VAT combine       3  4  12    VH     Decision table promo x VAT x rounding;
    (e.g. 99.99 SAR, 10% code, 15% VAT)                                boundary amounts 0.01 / max; API tests
R3  Double submit creates two orders                2  4   8    H      Double click, browser back, network
                                                                        retry; idempotency key verified
R4  Another user's saved card visible              1  4   4    H*     Two-account authorization tests;
    (broken access control)                                            added to security regression
R5  3-D Secure challenge fails on mobile Safari      2  3   6    M      Target devices: happy + cancel path
R6  Confirmation email delayed over 5 minutes       2  2   4    M      One end-to-end check; alert in staging
R7  Arabic address truncated on receipt PDF          3  1   3    L      One case in localisation pass

* raised from Medium to High by the Impact-4 rule.
Order of work: R1, R2 -> R3, R4 -> R5, R6 -> R7.

Try it: Risk-Based Test Generator

Risk mitigation and monitoring during the release

Testing is one mitigation among several. A high risk can also be reduced by changing the design (idempotency keys against double orders), by limiting exposure (a feature flag or a staged rollout), or by detecting failure quickly in production (monitoring and alerts on payment reconciliation). The risk register should name the mitigation, not only the tests.

Risks change as the release progresses. Re-score at fixed points — after the first test cycle, after each significant defect, before the go/no-go meeting. A cluster of defects in one area raises its likelihood; a clean run of deep tests lowers it. Tracking the number of open High and Very high risks over time gives a simple risk burn-down that stakeholders understand.

• Add a risk when testing reveals a new failure mode; do not wait for the next workshop.
• Link each defect to the risk it confirms, so the register shows evidence, not only opinions.
• Close a risk only when its planned tests have run and passed, or when the owner formally accepts it.

Reporting residual risk

Residual risk is what remains after testing: risks that were tested lightly, tests that were not run, defects that are open, and areas nobody examined. Reporting it is the most useful thing a test report can do, because it lets the people who own the release accept the remaining risk knowingly instead of discovering it in production.

Residual risk — Checkout v2, release 5.3

Reduced:   R1, R2, R3, R4 — planned tests executed and passed; no open defects
Partial:   R5 — tested on 2 of 4 target devices (Android tablet and
           older iOS not available); owner: product manager, accepted
Open:      R6 — defect CHK-214 (email delayed up to 12 min under load)
           open, severity Major; mitigation: alert on queue length
Not tested: R7 — cut for schedule; one known cosmetic issue possible
Out of scope: refunds flow (unchanged this release; smoke only)

Recommendation: release with the R5 and R6 conditions above accepted in writing.

Try it: AI Test Coverage Analyzer

Limitations of risk-based testing

• Scores are subjective. Likelihood in particular is a judgement; different groups score the same risk differently, and the loudest person in the workshop can anchor everyone else. Score independently first, then discuss the differences.
• False precision. Multiplying two ordinal numbers produces a number that looks exact but is not: a 12 is not exactly 50% riskier than an 8. Use scores to sort and to band, never to calculate percentages or averages.
• Unknown risks stay unknown. A register only contains the failures someone imagined. Keep exploratory testing time that is not tied to any listed risk.
• Registers go stale. A register written at the start of a project and never updated gives false confidence. Re-score at fixed points.
• Low risk is not no risk. Low-risk areas still get a minimum check, and repeated deferral across releases is itself a risk.
• Bureaucracy. For a small change, a five-minute conversation using the same two questions — how likely, how bad — is enough; the method scales down.

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• Certified Tester Advanced Level Test Management (CTAL-TM) v3.0ISTQB · 2024ISTQB's advanced syllabus on managing testing: test planning and estimation, risk-based testing, monitoring, metrics and reporting.
• ISO/IEC/IEEE 29119-2:2021 — Software and systems engineering — Software testing — Part 2: Test processesISO/IEC/IEEE · 2021The test process model: organisational, test management and dynamic test processes, with their activities and outcomes.
• ISO/IEC/IEEE 29119-1:2022 — Software and systems engineering — Software testing — Part 1: General conceptsISO/IEC/IEEE · 2022The international software-testing standard: concepts, processes, documentation and techniques.
• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality modelISO/IEC · 2023The product quality model: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability and flexibility.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.