Confirmation and Regression Testing

Why it matters

A regression is a loss of quality caused by a change: a feature that worked yesterday fails today. Products share code, data and services, so a one-line fix in one place can break a screen nobody touched, and such a defect reaches production easily when the team checks only the reported bug.

The two give different evidence: closing a defect without a confirmation test reports a fix that may not exist, and confirming a fix without regression testing implies a safe release that nobody checked. Choosing a scope that fits the time, and what to automate, is a call you make after almost every fix.

Confirmation testing: is the defect really fixed?

The ISTQB Foundation syllabus describes confirmation testing as confirming that an original defect has been successfully fixed. Depending on the risk, you run all the test cases that failed because of the defect and add new tests that cover the changes made for the fix; when time or money is short, you at least repeat the steps that reproduced the failure and check that it no longer occurs.

Use the original steps, data and environment first, so a pass really means the fix works. Then vary one thing at a time — another input format, another role, the neighbouring boundary value — because a fix often covers only the reported case. The edge cases from the previous lesson are a good source of variants.

• Check the build: the defect’s resolution names the build with the fix.
• Pass: record build, data and result, then run regression tests before closing the defect.
• Fail: reopen the same defect with the new evidence instead of opening a second one.

Regression testing: did the change break anything else?

Regression testing re-runs tests that passed before, to detect defects that a change has introduced or uncovered in parts not meant to change. "Uncovered" matters: a fix can expose an older defect that the first one was hiding, such as a second check that never ran because the first always rejected the input.

The syllabus notes that the effects can appear in the changed component, in other components or in connected systems, and that regression testing is not limited to the test object: it can also concern the environment. A fix that passed its confirmation test still needs regression testing, because the fix is itself a change, and both kinds of testing are needed at every test level where a defect is fixed or a change is made. Typical triggers:

• Defect fixes, new features and refactoring (restructuring code without changing what it does).
• Upgrades of libraries, frameworks, the database or the operating system.
• Configuration changes, feature flags (switches that turn a feature on or off), database migrations and changes in connected systems.

Impact analysis: choosing what to re-run

Running every test after every change is the safest choice and, for a small suite, the right one. The syllabus advises an impact analysis first, to decide the extent of regression testing: it shows which parts of the software a change could affect. It matters most once the full suite no longer fits the time available. Read the change and ask the developer instead of choosing from memory.

The answers to the questions below give a scope in three rings: the changed area, its direct dependents, and a small core of critical journeys that runs after every change. Record what you leave out and why; a scheduled full run catches what the selection missed.

• What exactly changed: code, data structure, configuration or a dependency? The pull request (the change the developer submits for review) and its diff of changed lines tell you.
• Who else uses it: screens, APIs, reports, jobs or services that call the same function or read the same table. A traceability matrix shows which tests cover a changed requirement.
• Which stored data does it touch: records saved before the change must still work after it.
• What is high risk regardless: payments, login, data export, and areas that broke after similar changes.

Try it: Regression Scenario Generator Try it: Requirements → Test Cases Try it: Risk-Based Test Generator

Worked example: a fix in shared validation code

Defect BUG-2143: at checkout, mobile numbers in international format, such as +966 50 000 1234, are rejected as "Invalid phone number". Build 3.4.2 fixes the shared function normalizePhone(). Asking "who else uses it?" shows that sign-up, profile editing and the OTP SMS service call it too.

The confirmation test passed: the number and its variants are accepted and saved in one format, and a too-short number is still rejected. Regression testing found a new failure: customers whose numbers were saved before the fix no longer receive their OTP, because the SMS service now looks numbers up in the new format. That new defect is reported separately and linked to BUG-2143, not reopened as the original.

• The stored-data question of the impact analysis caught it: new numbers worked; only records saved before the change failed.
• Comparing one existing customer’s profile API response on the old and the new build shows the format change at a glance.
• The note below records the plan, including what was deliberately left out.

BUG-2143 | build 3.4.2 | changed: normalizePhone() in shared validation

1. CONFIRMATION  (original steps, data and environment first)
   [ ] checkout with +966 50 000 1234        -> accepted, order placed
   [ ] 0500001234, 966500001234, with spaces -> accepted, saved as +966500001234
   [ ] +966 50 000 12 (too short)            -> still rejected

2. REGRESSION  (selected by impact analysis)
   changed area  phone field at checkout: all existing test cases
   dependents    sign-up, profile edit, OTP by SMS: same stored format
   stored data   customers saved before 3.4.2 still receive their OTP
   core          log in, search, card payment (critical journeys)
   left out      wishlist and reviews: they do not use phone numbers
   safety net    full regression suite in the scheduled nightly run

Try it: API Response Comparator

Why regression tests are the first to automate

Regression tests run again and again and the suite grows with every release, so the syllabus calls regression testing a strong candidate for automation, started early. In continuous integration they run on every change, often at several levels: component tests for logic, API tests for services, a few end-to-end journeys.

A confirmation test is often run once, by hand. Keep it as an automated regression test at the lowest level that can express it — here, a component test of normalizePhone() with the reported number and its variants — so the defect cannot return unnoticed.

• Good candidates: stable features, deterministic expected results, critical journeys, rules checked with many data values.
• Poor candidates: features still changing every sprint, one-off checks, judgements about layout or ease of use.
• Automation runs tests but does not choose them, and the suite needs maintenance like any code.

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISO/IEC/IEEE 29119-1:2022 — Software and systems engineering — Software testing — Part 1: General conceptsISO/IEC/IEEE · 2022The international software-testing standard: concepts, processes, documentation and techniques.