Failed and Blocked Tests

Why it matters

A failed test wrongly reported as a product defect wastes a developer’s time, gets rejected and costs the next report some trust. Dismissing a real failure as "just the environment" is worse: the defect reaches users.

Blocked tests carry a quieter risk: neither passed nor failed, the features behind them are untested. A run with 40 passed and 30 blocked tests says far less about quality than its pass count suggests.

Why a test fails: product, test or environment

A difference between actual and expected result has three possible sources: a product defect (the software does not behave as the test basis says), a test problem (the test case, script, expected result or execution is wrong), or an environment or test data problem (the build, services, configuration or data are not what the test assumes). The last two are false positives, as in the lesson on error, defect and failure.

Rule out the cheaper explanations first, in this order. A failure that gets through all four questions is reported as a product defect; one that gets through the first three but does not repeat is still reported, marked as intermittent.

• Did the test reach the step that checks its expected result? If not, it is blocked, not failed.
• Were the build, services, configuration and data what the test needs? An older build or an already used single-use coupon fails tests the product would pass. The next lesson, on environment and test data problems, covers their signs and how to record them.
• Are the steps and expected result right for the current requirement, and were they followed? A test still expecting "Checkout" after an approved rename to "Pay now" is wrong, not the product.
• Does the failure repeat when you reproduce it, as in the previous lesson? If not, record it as failed, never as passed, and report it as intermittent with the evidence and the rate.

Try it: Text Diff Try it: JSON Diff

Blocked tests: when a test cannot reach its check

A blocked test has produced no result about what it checks. Typical blockers are a defect in an earlier step or in a feature the test depends on, a service that is down, missing data or permissions, and a feature not yet deployed.

The rule: the failure belongs to the test whose own check failed, and every other test that needed that step is blocked by it. If login fails, the login test fails and the checkout tests are blocked. Marking them all failed would show one defect as many failures and hide that those features were never checked. Some teams record them as failed with a link to the defect instead; agree one convention.

• Look for another valid route first, such as creating the order through the API, and note any workaround in the result.
• When the blocker is removed, the blocked tests run for the first time. They are not retests, and may reveal defects the blocker was hiding: the defect masking from the lesson on error, defect and failure.

What to record for each result

A bare "Failed" forces someone to repeat your work.

• Every result: test case ID, run, build, environment, tester and date.
• Failed: the failing step, actual next to expected, evidence (screenshot, log, response), the cause category and the defect ID.
• Blocked: what blocks it and at which step, the blocker’s ticket ID and owner, and since when.
• Test problem: what was wrong, how it was corrected and reviewed, and the corrected result, with a link to any requirement change.
• Retest: the defect ID, the build with the fix, the result and its evidence, as a new execution so the original failure stays in the history.

Worked example: triaging one checkout run

Six checkout tests run on build 4.12.0 in staging, and five do not pass at first. The log shows the results after triage.

• TC-206 checks adding a bundle to the cart, so the HTTP 500 error (a server-side error) is its own failure: Failed, defect DEF-480. TC-203 checks a bundle discount and stops at step 2: Blocked by DEF-480, not a second defect.
• TC-202 reached its check in a healthy environment, its expected total follows the requirement and the failure repeated: Failed, product defect DEF-482.
• TC-204 stopped at payment: the third-party payment sandbox returned 503 (service unavailable) for every card and its health check failed too. Blocked by environment ticket ENV-77, no product defect.
• TC-205 expected "Checkout", but an approved design change had renamed the button "Pay now": a test problem. The test case was corrected against that change and went through the team’s usual review as TC-205 v2; the tester then re-ran it with the change linked: Passed.
• Result: 2 passed, 2 failed, 2 blocked, and two defect reports instead of five.

Run: checkout regression, cycle 3   Build: 4.12.0   Environment: staging

Test    Result   Step  Note                                           Link
TC-201  Passed   -     -                                              -
TC-202  Failed   4     10% coupon: total 115.00, expected 103.50      DEF-482
TC-203  Blocked  2     Cannot add bundle to cart (HTTP 500)           DEF-480
TC-204  Blocked  5     Payment sandbox returns 503 for every card     ENV-77
TC-205  Passed   3     Re-run after test fix: label is now "Pay now"  TC-205 v2
TC-206  Failed   2     Adding a bundle to the cart returns HTTP 500   DEF-480

Try it: API Health Checker Try it: API Response Comparator

Retesting after a fix

Confirmation testing (retesting) checks a reported fix; see the lesson on confirmation and regression testing. When DEF-480 is fixed in build 4.12.1, TC-206 is retested and TC-203 runs for the first time. If TC-203 then fails on the discount, that is a new defect the blocker was hiding, not a reopened DEF-480.

• Check that the build under test contains the fix (see the defect or the release notes); a retest on the old build proves nothing.
• Repeat the original steps, data and conditions plus a few nearby variations, such as other bundle types, and record the retest as a new execution.
• If it passes, update the defect as your lifecycle defines; if it fails the same way, reopen it with the new build and evidence; if it fails differently, report a new defect and link the two.
• Then run the tests the defect was blocking and the regression tests selected for the change.

Try it: Regression Scenario Generator

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISO/IEC/IEEE 29119-2:2021 — Software and systems engineering — Software testing — Part 2: Test processesISO/IEC/IEEE · 2021The test process model: organisational, test management and dynamic test processes, with their activities and outcomes.