Environment and Test Data Problems

Why it matters

In the ISTQB Foundation syllabus, test execution includes analysing each anomaly for its likely cause, and a reported anomaly may turn out to be a false-positive result rather than a real defect. Environment and test data problems are typical causes of such false positives, and they can also produce false negatives, where a test passes although a defect is present.

Repeated setup failures also wear down trust in the results: a team whose runs are often red for setup reasons learns to ignore red.

Environment problems and their signs

Any layer of the environment from the lesson on test environments can break during execution, and each break leaves typical signs.

• Wrong build: an old or partial deployment; a fixed defect is back, or the reported version differs from the run's build.
• Dependency down: a database, queue, payment sandbox or SMS stub is unavailable. Connection errors, timeouts or the HTTP status codes 502 (bad gateway), 503 (service unavailable) and 504 (gateway timeout) appear across unrelated features; with these codes a server says that it, or a service behind it, cannot answer properly right now.
• Configuration drift: a feature flag, environment variable or service address differs from what the tests assume, so a feature is missing or calls the wrong system.
• Expired access: a test account password, an API token (the key one system uses to call another) or the environment's TLS certificate (what lets browsers connect over HTTPS); requests that worked yesterday are rejected with 401 (not authenticated), logins fail, or browsers refuse the connection with a certificate warning.
• Resources and interference: a full disk, exhausted database connections, or someone redeploying the shared environment during your run; failures begin in the middle of a run, or everything fails for a few minutes and then recovers.

Try it: API Health Checker Try it: HTTP Status Code Lookup Try it: JWT Expiration Checker

Test data problems and their signs

Many of a test case's preconditions are about data: customer C-101 exists, the order is paid and not shipped. A data problem means such a precondition was not true when the test ran, although the product may have behaved correctly.

Data can also hide a defect: a test of the first login, which should ask the user to verify their email address, runs on an account already verified in an earlier run, so the verification step never appears and the test passes even if verification is broken. Such false negatives are rarely investigated, because the result is green.

• Missing: the seed script did not run, or the baseline was not restored after a refresh.
• Wrong state: another test or tester used the record, so the order is already refunded or the account is locked.
• Collisions: parallel runs share an account, or "email already registered" appears because a previous run left its user behind.
• Out of date: a hard-coded date, coupon or test card has expired, or a copy of the data made months ago no longer matches the current prices or settings.

Try it: UUID Generator

Telling them apart from product defects

Do not decide from the error message alone. Collect a few facts, cheapest first; together they usually point to one cause.

One trap: a wrong precondition does not excuse wrong behaviour. If an already refunded order produces a server error instead of a clear message, the data was wrong for that test, and the product still has a defect: it must handle that state safely. Fix the data and report the error handling separately.

• Pattern: many unrelated tests failing together point to the environment; one test or feature points to its data or the product.
• Timing: did the failures start with a deployment, a configuration change or a data refresh rather than with the change under test?
• Message: gateway and connection errors point to the environment; a business message such as "already refunded" says: check the data first.
• Baseline: is it the right build, are the dependencies healthy, is the record in the state the test case requires?
• Clean rerun: repeat with fresh data or in a second environment. A failure that persists in a healthy environment with correct data is not an environment or data problem: check the test case as in the previous lesson, and if it is right, report a product defect.

Try it: API Response Comparator

Worked example: 14 failures in one morning

By 09:40, 14 of 40 tests in the checkout regression run on build 4.1.0 have failed. Instead of filing 14 defect reports, the tester groups the failures by symptom and time and checks each group.

The nine API failures hit unrelated endpoints with the same 401 (not authenticated) from the first request at 09:00, which points to access rather than to any one feature. Decoding the test client's API token shows that it expired at midnight: one environment issue, and once the environment owner had renewed the token, all nine reruns passed. The coupon tests used a coupon whose fixed end date has passed. The order used by the refund test had already been refunded by another run, which explains the failure but not the server error (HTTP 500): that is a defect of its own. Only the VAT failure reproduces on fresh data in a healthy environment.

Result: one environment issue whose nine reruns passed after the token was renewed, two data fixes whose reruns passed, and two defect reports instead of 14.

Run: Checkout regression / build 4.1.0 / test environment / 09:00-09:40
Not passed: 14 of 40 (before triage)

API-01..09  401 Unauthorized from cart, address and order endpoints alike
            test client's API token expired at 00:00     -> ENV-31, renewed, reruns passed
CPN-01..03  "Coupon expired"
            coupon in the data has a fixed end date      -> data fixed, rerun passed
RFD-04      500 error when refunding order O-7781
            O-7781 refunded by another run at 08:50      -> data reset, rerun passed
            a refunded order needs a clear message       -> defect DEF-436
TOT-02      VAT calculated on the price before discount
            same result on a fresh order, healthy env    -> defect DEF-437

Prevention: check the setup before the tests

In the ISTQB Foundation syllabus, the test environment is built and verified to be set up correctly during test implementation, before execution, and available environments and test data are among the typical entry criteria; ISO/IEC/IEEE 29119-3 describes readiness reports for both. Martin Fowler's article on non-determinism in tests names lack of isolation, remote services and time among the usual causes of unreliable results. The habits below target them.

• Smoke first: check the dependencies' health and run a short smoke test after every deployment, before planned testing.
• Check the build: compare the version the environment reports with the run's build.
• Own your data: unique records per tester and parallel run, created by the test or a script.
• Make data repeatable: restore a known baseline, and write dates relative to the run date.
• Announce changes: no deployments, restarts or data refreshes in the middle of someone's run.
• Log environment issues with an owner, and treat a cause that returns every week as a risk to fix, not a nuisance to work around.

Try it: Test Data Factory

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISO/IEC/IEEE 29119-3:2021 — Software and systems engineering — Software testing — Part 3: Test documentationISO/IEC/IEEE · 2021Templates and content for test documentation — test policy, strategy, plan, status and completion reports, incident reports.
• Eradicating Non-Determinism in Tests (Martin Fowler)martinfowler.com · 2011Why flaky tests destroy a suite's value, their common causes (isolation, asynchrony, remote services, time, resource leaks) and quarantine as a response.