Reproducing Failures

Why it matters

A failure that nobody can make happen again is hard to fix, and a fix for it cannot be confirmed by retesting. If the developer cannot reproduce it, the report often comes back as "cannot reproduce" and the defect stays in the product, where users may meet it. Precise conditions shorten diagnosis: "fails for any customer whose saved address has no postal code" points to one area of the code instead of the whole checkout.

For the tester, reproduction is also a check on your own observation: trying again under controlled conditions shows whether the failure is real or came from a slip of your own. And the steps that reproduce it are the basis of confirmation testing after the fix; without reliable steps, a passing retest proves little.

Start from the conditions of the first failure

When a test fails, do not start changing things. First write down everything that was true at that moment. Then repeat the failure exactly as it happened: same build, environment, account, data and steps. If it fails again, you have a reproducible failure and a baseline to compare against. If it does not, something differs from the first time, and that difference is your first clue.

Record more than the steps of the test case:

• Build and environment: build number, environment, configuration and feature flags (switches that turn a feature on or off without a new build).
• Client: browser and version, operating system, device, screen size, language and direction; an Arabic right-to-left page can fail where the English page passes.
• User: role, permissions and account state, such as new, locked or with an expired subscription.
• Data: the exact input values, files and records involved, such as the cart contents and the coupon.
• State: what happened earlier in the session, cookies and cache.
• Time, with the time zone, so the failure can be matched with the server logs.

Try it: User-Agent Parser

Isolate the variables: change one thing at a time

Once the failure reproduces, find out which conditions matter. Treat each condition as a variable and change exactly one between attempts, keeping the rest fixed. If the failure disappears when you switch the browser, the browser is part of the trigger; if it stays, the browser is not a factor: leave it out of the steps and note in the report that you ruled it out. Change two things at once and you cannot tell which one made the difference.

To find candidates, compare a failing case with a working one: two accounts, two requests or two sets of data side by side. The difference is often a single field, such as an empty address line, a long name or another currency. Start with the variables most likely to matter and cheapest to change: data, user, browser, environment.

• Fails for one user only: compare that user’s data, role and settings with a user for whom it works.
• Fails in one browser only: read that browser’s console and repeat in another version of it.
• Fails only after an earlier action: the state that action leaves behind is a variable too.

Try it: JSON Diff Try it: API Response Comparator

Reduce it to a minimal reproduction

The first reproduction usually carries steps that have nothing to do with the failure: you were at step 17 of a long test case. A minimal reproduction is the smallest set of preconditions and steps that still produces the failure. Remove one step or precondition at a time and run again: if the failure still happens, leave it out; if it stops, put it back. Stop when every remaining step is needed.

The result is a report a developer can follow in a minute and a ready-made confirmation test for the fix, often a future regression test case too. State its reliability as a number, such as "5 of 5 attempts", not as "always".

Worked example: an order that fails for one customer

During a regression run on staging, test case TC-214 "Place an order with a saved card" fails: after "Place order" the page shows "Something went wrong" and the network panel shows POST /api/orders returning 500. The tester notes build 4.12.0, account cust-07, a cart of three items with a coupon, the browser and the time, then changes one thing per attempt. Each line below is a round of attempts and what it showed; the report after them is the result.

• Same conditions, twice: fails both times. The failure is reproducible.
• Firefox instead of Chrome: fails. The browser is not a factor.
• Account cust-09 instead of cust-07: the order is created. The account matters, and comparing the two shows one difference: the saved address of cust-07 has an empty postal code.
• Postal code added to cust-07: passes. Removed from cust-09: fails. The empty postal code is the trigger.
• Removing steps one at a time: the coupon, two of the items and the saved card can all go; one item paid by cash on delivery fails too.
• The minimal steps, run 5 times: 5 failures; with a postal code: 0 of 5. The requirement does not say what should happen with an empty postal code, so the report gives both acceptable outcomes and the tester asks the product owner which one is intended.

Title:     Placing an order returns 500 when the saved address has no postal code
Build:     4.12.0, staging        Browsers: Chrome and Firefox (both fail)
Precondition: signed-in customer whose saved address has an empty postal code
Steps:     1. Add any in-stock product to the cart.
           2. Open the cart and select "Checkout".
           3. Keep the saved address, choose "Cash on delivery", select "Place order".
Expected:  the order is created, or the form asks for a postal code
Actual:    "Something went wrong"; POST /api/orders returns 500
Rate:      5 of 5 attempts; with a postal code: 0 of 5
Ruled out: browser, coupon, number of items, payment method (saved card fails too)

Intermittent failures: reproduce by repetition

Some failures appear on only some attempts with the same steps. An intermittent failure still has a cause, a variable you have not found or cannot yet control: timing, two users changing the same data at once, state left by an earlier test, caches, the time of day, or which of several servers handled the request.

An automated test that behaves like this is called a flaky test; the cause can be in the test, the environment or the product, and the same method applies. Do not call it random. Measure it:

• Repeat the same steps a fixed number of times, for example 20, and count the failures: "3 of 20" is a reproduction rate.
• Note the time of each failing attempt so it can be matched with the server logs, and compare the log of a failing attempt with the log of a passing one.
• Change one condition and repeat the same number of times. A condition that moves the rate to 0 of 20, or to 20 of 20, is very likely part of the trigger.
• If it stays unreliable, report it with the evidence, the conditions and the rate, marked as intermittent, and never accept one passing run as proof of a fix.

Try it: Text Diff Try it: Timestamp Converter

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• Eradicating Non-Determinism in Tests (Martin Fowler)martinfowler.com · 2011Why flaky tests destroy a suite's value, their common causes (isolation, asynchrony, remote services, time, resource leaks) and quarantine as a response.