Testware, Test Basis and Test Oracles

Why it matters

As the lesson on error, defect and failure showed, a failure is the system not doing what it should — and "should" needs a source. Without a named test basis, testers test what they assume; without an oracle, a test passes because the screen shows a plausible number; without the exact build of the test object, nobody can repeat the result.

Testware makes testing repeatable and visible: a written test case can be rerun after a fix and in every regression cycle, test logs and reports are the evidence behind a release decision, and a defect report can name the exact requirement, build and expected value.

Test basis, test object and test oracle

The test basis is the body of knowledge used as the basis for test analysis and design: requirements and user stories with acceptance criteria, business rules, designs, API specifications, risk analyses, regulations and, for white-box tests, the code.

The test object is the work product being tested: a function, an API endpoint, a screen, a whole system or a document. Identify it precisely — component, build and environment — because a result only holds for what was tested. Many teams say system under test (SUT); in the ISTQB glossary that is a test object that is a whole system. One item can play both roles: in a requirements review the requirement is the test object; when you test the built feature, it is the test basis.

A test oracle is a source to determine an expected result to compare with the actual result. Often the test basis is the oracle, but not always:

• The specification: an acceptance criterion, a business rule or an API contract states the result.
• An independent calculation by hand or in a spreadsheet — never the system’s own output.
• A previous version or another system: after a refactoring or a data migration, the old output is the expected result.
• Standards and consistency rules: WCAG, the date format of a locale, a cart total that must equal the invoice total.
• A partial oracle when the full result cannot be predicted, such as a search ranking: check what you can (every result contains the search term) and note what remains unjudged. The general difficulty of knowing the correct result is called the test oracle problem.

Try it: Response Contract Comparator Try it: API Response Comparator

Test conditions: deciding what to test

A test condition is a testable aspect of a component or system identified as a basis for testing. It answers "what to test?" and is often narrower than the requirement it comes from. The ISTQB syllabus names acceptance criteria as an example of test conditions; a criterion that covers several behaviours is split into one condition per behaviour.

Test analysis identifies and prioritises conditions; test design turns each into test cases with concrete inputs and expected results. Both get their own lessons later. A good condition:

• Describes an observable outcome, not a wish such as "VAT works correctly".
• Covers one aspect and traces to one item of the basis, such as AC2 of story QA-214 in the worked example below.
• Has a priority, so the riskiest conditions are tested first.

Testware: what each test activity produces

The ISTQB Foundation syllabus v4.0 describes the test process as seven activities: test planning (deciding the test objectives and approach), test monitoring and control (tracking progress against the plan and adjusting), test analysis and test design (what to test and how, as above), test implementation (preparing the scripts, data and environment needed to run the tests), test execution (running the tests and logging the results) and test completion (wrapping up and archiving what is worth keeping). They do not always run in a strict sequence, and the module "The test process (STLC)" covers each one.

The ISTQB syllabus lists testware by the activity that produces it. Not every item is a separate document: in agile teams much of it lives in a test management tool, on the story or in the repository. Some teams say test artefacts; ISO/IEC/IEEE 29119-3 gives templates for many of these documents. Like code, testware is under configuration management, so you know which version of a test ran against which build.

• Test planning: test plan, schedule, risk register, entry and exit criteria.
• Test monitoring and control: test progress reports, records of control decisions and risk information.
• Test analysis: prioritised test conditions, and defect reports on the test basis itself.
• Test design: prioritised test cases, test charters, coverage items, test data requirements and test environment requirements.
• Test implementation: test procedures, automated test scripts, test suites, test data, the test execution schedule and test environment elements such as stubs and drivers.
• Test execution: test logs (what ran, when, on which build, with which result) and defect reports.
• Test completion: the test completion report, lessons learned, improvement actions and change requests; testware that will be useful later is archived or handed over to the teams that need it.

Try it: Test Case Generator Try it: Test Data Factory

Worked example: one story from test basis to test log

A web shop adds VAT to its invoices. The story below is the test basis; the test object is the invoice page, build 3.8.0, on the staging environment.

• Test conditions: VAT is 15% of the subtotal (AC1); a third decimal of 5 rounds up (AC2); the total adds the rounded VAT (AC3).
• Oracle: AC1 and AC2 plus a hand calculation made before the run. For a subtotal of 10.30: 10.30 × 0.15 = 1.545, so the expected VAT is 1.55 and the total 11.85. For 100.00 the expected VAT is 15.00 and the total 115.00.
• Testware: a test case per condition with these expected values, test data with subtotals 10.30 and 100.00, and the test log of the run.
• Result: on build 3.8.0 the rounding case failed, VAT shown 1.54. The defect report cites AC2, the build, and expected against actual; the developer traces it to round-half-to-even ("banker’s rounding"). On screen 1.54 looked plausible; only the oracle exposed the failure.

QA-214  Invoice shows VAT
As a customer, I want my invoice to show the VAT
so that I know how much tax I paid.

AC1  The invoice shows the subtotal, VAT at 15% and the total.
AC2  VAT is rounded to 2 decimal places, half up.
AC3  Total = subtotal + VAT as shown on the invoice.

Try it: Requirements → Test Cases

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISO/IEC/IEEE 29119-3:2021 — Software and systems engineering — Software testing — Part 3: Test documentationISO/IEC/IEEE · 2021Templates and content for test documentation — test policy, strategy, plan, status and completion reports, incident reports.