Test Environments

Why it matters

Teams test outside production for good reasons: customers must not see unfinished work, and tests must not move real money or touch real personal data. But every difference from production is a place where a defect can pass every test and still reach customers, or where a test fails for reasons unrelated to the product.

Knowing the environment tells a tester whether a failed test points to a product defect or to an environment problem, and whether a pass is real evidence. A tester who can say what was tested where, and what differs in production, gives the team an honest picture of the remaining risk.

What a test environment is made of

Think of it as layers around the build under test; each layer is a question you should be able to answer when you plan tests or report a defect.

• Build: the exact version of every component under test, such as web 3.8.0, build 214.
• Infrastructure: servers or containers, operating system, the number of instances (running copies of the application); on the client side, browsers and devices.
• Configuration: settings that change behaviour without changing code, such as feature flags, time zone, locale and limits.
• Dependencies: databases, queues and third-party services such as payment or SMS, each one real, a provider's sandbox (a test version of the service, run by the provider) or a stub.
• Data and access: the test data baseline and accounts, plus the logs you can read.

Dev, test, staging and production

Most teams move a change through a chain of environments, each closer to production than the one before. The names are industry usage, not ISTQB terms, and vary: test may be called QA or SIT, staging pre-production or UAT, and small teams often have one environment between development and production.

In a healthy chain the build is created once and that same build is promoted; only the configuration changes. If each environment gets its own build, a pass on staging says little about production. A fix produces a new build, which goes through the chain again.

• Development: developers' own machines, sometimes also a shared development server; component (unit) tests with mocked dependencies; the fastest feedback and the least like production.
• Test (QA): shared by testers for system and integration testing on synthetic data that can be reset; new builds arrive often.
• Staging: as close to production as practical in versions, topology and data volume; for release and acceptance testing and for rehearsing the deployment.
• Production: real users, data and money; only planned, non-destructive checks after a deployment, with test accounts.
• Many pipelines also create a short-lived environment for each pull request.

Try it: API Health Checker

Configuration, versions and dependencies

Two environments running the same build differ through their configuration, the versions of the software around the build, and the dependencies it calls. Compare these first when something works in one place and not in another, and record them in an environment record or a version page.

Each line of such a record limits what your tests prove: if staging sends SMS to a stub, you proved that the application asks for a message to be sent, not that a customer receives it.

The ISTQB syllabus treats a test environment as a configuration item: configuration management records its parts, their relationships and their versions. Once approved for testing, that record becomes a baseline that changes only through change control, so earlier results can be reproduced.

• Configuration usually lives outside the build, in environment variables or files, so it can drift between environments without anyone noticing.
• Write full versions: PostgreSQL 16 and PostgreSQL 16.4 are different records.
• Check the deployed build before testing: many applications show it on an about page or a version or health endpoint.
• Sandboxes often answer faster and more predictably than the real service.

Try it: JSON Diff Try it: Text Diff

Environment parity problems

Environment parity means the environments differ from production as little as possible, and only in ways you know. Martin Fowler's article on continuous integration recommends testing in a clone of the production environment; the Twelve-Factor App calls the goal dev/prod parity and names three gaps to keep small: time, people and tools.

Gaps cause wrong results in both directions: a false pass (a false negative), where the environment hides a defect, and a false failure (a false positive), where a stale build or a sandbox that is down is reported as a product defect. Full parity is rarely possible, so aim for known differences: each gap that could affect the change is closed, tested another way, or recorded as a risk.

• Time zone: a 20:00 cut-off is tested against the wrong clock.
• One instance in test, several in production: files or sessions kept on one server go missing.
• Small data: a report that is instant on 5,000 rows times out on a million.

Try it: Timezone Converter

Worked example: a receipt upload that passed on staging

Release 3.8.0 lets customers attach a receipt of up to 5 MB to a refund request. All 18 test cases passed on staging. Before sign-off, the tester compares the two environment records.

The build is the same, so what was tested is what will ship. Three differences matter for this change:

• Proxy limit of 2 MB: in production a reverse proxy, a server in front of the application, rejects larger requests. Receipts above 2 MB passed on staging, but production would reject them, typically with HTTP status 413 (Content Too Large), before the application can show its own message. Raise it with DevOps now.
• One instance storing files on its local disk, against three instances sharing object storage behind a load balancer, which spreads requests across them: the storage that production uses was never exercised, and because the upload and the display of a receipt may reach different servers, anything still kept on one server's disk would be missing. Ask for a staging run with two instances and shared storage, or check right after deployment.
• Flag off in production: switching it on is a configuration change with its own risk. Test with the flag off too, and agree who switches it on.
• Payment, SMS and data volume differ too, but the receipt upload does not depend on them: record them as known differences.

Item              staging                  production
----------------  -----------------------  ---------------------------
Build             web 3.8.0+214            web 3.8.0+214
App instances     1                        3, behind a load balancer
Reverse proxy     none                     request body limit 2 MB
File storage      local disk               shared object storage
Database          PostgreSQL 16.4          PostgreSQL 16.4
Time zone         Asia/Riyadh              Asia/Riyadh
refund_receipts   flag on                  flag off until release day
Payment           provider sandbox         live provider
SMS               stub, records messages   live provider
Orders in DB      5,000 synthetic          1.2 million real

Try it: API Response Comparator

Frequently asked questions

References

• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• Continuous Integration (Martin Fowler)martinfowler.com · 2024The widely referenced description of continuous integration: integrate often, keep the build self-testing and fast, and fix a broken build immediately.
• ISO/IEC/IEEE 29119-3:2021 — Software and systems engineering — Software testing — Part 3: Test documentationISO/IEC/IEEE · 2021Templates and content for test documentation — test policy, strategy, plan, status and completion reports, incident reports.