Regression Defects

Why it matters

Users lose something they already relied on, often in a part of the product the release notes do not mention, so nobody was looking there. Support often hears about it before the team does.

For the tester, a regression has an advantage most defects lack: a known good version to compare with. A report that names the last version that worked and the first that failed lets the developer search a few changes instead of the whole code base, and each regression shows a gap the regression suite can close.

What makes a defect a regression

Two facts make a defect a regression: the behaviour worked in an earlier version, and a change made it fail. Check the first fact instead of assuming it: run the same test on the previous build or release. Many teams then add a regression label and record both versions.

• Introduced: the change contains the mistake, such as a shared function edited for one caller that breaks another.
• Uncovered: the change exposes an older, hidden defect, for example a fix lets input reach a check that never ran before.
• Reintroduced: a fixed defect comes back, often through a merge that lost the fix. Some teams call it a recurring defect.
• Not a regression: behaviour that never worked, in a new feature or as an old defect found late. It fails on the previous version too.

Why regressions happen

A change is usually reviewed for its own purpose, not for every place that relies on it. The ISTQB Foundation syllabus notes that its adverse effects can appear in the changed component, in other components of the same system or in connected systems, and that regression testing is not limited to the test object: it can also concern the environment. A regression reaches users when no test checks the behaviour that broke, the test did not run, or it failed and nobody acted. Typical causes:

• Shared code: a function, component, style rule or API used in several places, changed with one of them in mind.
• Fixes and refactoring: right for the reported case, or meant to change nothing, yet different for an edge case.
• Upgrades of libraries, frameworks, browsers or the database; configuration and feature-flag changes.
• Data: a migration, or a new rule meeting records saved under the old one.
• Merges and connected systems: a conflict resolved the wrong way, or a service whose contract changed.

Detecting regressions: suites, baselines and bisecting

The regression suite exists to detect regressions: tests of behaviour that already worked, re-run after changes. Because it runs again and again and grows with each release, the ISTQB syllabus calls regression testing a strong candidate for automation. A test that passed on the previous build and fails on this one is the clearest regression signal. First rule out a flaky test, an environment or data problem, or a test made outdated by an intended change, as in the lesson on failed and blocked tests. Then run the test on the last version that should be good (the baseline), in the same environment. Comparing the outputs of the two versions, such as two API responses, often shows exactly what changed.

Bisecting narrows the range between the last good and the first bad version to one change. It is a binary search: test the version halfway, keep the half where the result switches from pass to fail, and repeat. 16 candidate builds need at most 4 tests; about 1,000 commits (changes saved in Git) need 10. git bisect does the bookkeeping over commits; without the source, you can bisect build numbers or releases in a test environment the same way.

• Use a reliable check: a flaky test sends the search into the wrong half.
• Change only the version; keep the environment, configuration and data the same.
• Skip a step that cannot be tested (git bisect skip) instead of guessing.
• The result is the first change where the failure appears. It may have uncovered an older fault, so it is evidence for the developer, not blame.

git bisect start
git bisect bad                # the version you are on fails
git bisect good v2.7.0        # the last version known to pass
# Git checks out a commit halfway: build it, run the test, then mark it
git bisect good               # or: git bisect bad
# repeat until Git prints "<commit> is the first bad commit"
git bisect reset              # return to where you started

Try it: API Response Comparator Try it: JSON Diff Try it: Text Diff

Worked example: bisecting a checkout regression

The nightly regression run passes on build 140 on Monday and fails on build 156 on Tuesday: TC-311, checkout with a saved address that has no postcode, now stops at payment with "Address is incomplete". Re-run on build 156, TC-311 fails the same way; on build 140, in the same environment with the same customer, it passes. So this is a regression, entered in one of builds 141 to 156, each containing one merged change.

Four tests find it. Build 147 contains only PR-882, which added a required postcode check to the shared validateAddress() function for a new courier: right for new addresses, wrong for every address saved before it. The pull-request checks had passed because their test data held only new addresses.

• Report title: "Checkout rejects saved addresses without a postcode (regression since build 147)".
• Evidence: last good build 146, first bad build 147, suspected change PR-882, caught by TC-311 in the nightly run.
• The fix applies the check only to the new courier, and TC-311 joins the pull-request checks.

Known good: build 140   Monday nightly run, TC-311 passed
Known bad:  build 156   Tuesday nightly run, TC-311 failed
Candidates: builds 141..156, one merged change each (16 builds)

Test  Build  TC-311  First bad build is in
1     148    fail    141..148
2     144    pass    145..148
3     146    pass    147..148
4     147    fail    147  -> first bad build

Build 147 contains one change: PR-882 "Require postcode for new courier"

Preventing regressions

No team prevents every regression, but each can be made less likely and caught closer to its cause. The earlier a run catches it, the fewer changes lie between the last good and the first bad result: one change at a pull-request check, a day’s changes in a nightly run, the release’s changes in the release regression with exploratory testing, and the same release’s changes in production, but after users are affected.

• Do a quick impact analysis before merging: who else uses this code, and which stored data meets the new rule?
• Run the regression tests relevant to the change, chosen by impact and risk, on every pull request and the full suite on the main branch, as a quality gate that can stop a change.
• Keep test data that looks like production history: old records, empty optional fields, data saved under earlier rules.
• For every regression fixed, add the test that would have caught it, at the lowest level that can express it.
• Fix or quarantine flaky tests, so that a red run always means something.
• Prefer small changes, and upgrade libraries in their own change with a regression run; small changes are easier to review and to bisect.

Try it: Regression Scenario Generator Try it: Risk-Based Test Generator

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• Reference — GitGit projectThe official reference for every Git command and its options, grouped by task — setup, branching, sharing, patching, inspection and debugging.
• Continuous Integration (Martin Fowler)martinfowler.com · 2024The widely referenced description of continuous integration: integrate often, keep the build self-testing and fast, and fix a broken build immediately.