Test Coverage

Why it matters

Without coverage, "we tested the feature" cannot be checked. Nobody can say which acceptance criteria have no test, which high risks were never tried, or which code no test has run. Those gaps are where defects slip into production.

For a tester, coverage turns effort into a statement others can verify and act on: "all 14 acceptance criteria have passing tests; 2 medium risks were not tested because the payment sandbox was down". It also makes exit criteria measurable and protects the team from false confidence in one large number.

Coverage of what? Requirements, risks and code

A coverage number means nothing until you say what was counted. Three kinds of coverage answer three different questions, and most teams need all three.

• Requirements coverage: which requirements or acceptance criteria have tests, and which have passing tests. The coverage items are the criteria in the test basis, counted through traceability: every test linked to the criterion it checks.
• Risk coverage: which identified product risks have tests that ran, and which have tests that passed, reported by risk level. "Both high risks tested and passing, 2 of 5 medium risks not tested yet" supports a release decision better than any single percentage.
• Code coverage: which parts of the code ran during the tests, measured by a tool. Statement coverage counts executable statements; branch coverage counts branches, the transfers of control in the code, which coverage tools report as decision outcomes, such as both sides of an if. Branch coverage is stronger: 100% branch coverage includes 100% statement coverage, not the other way round.

Coverage items and how to measure them

The ISTQB Foundation syllabus v4.0 measures coverage through coverage items: countable things derived from test conditions by applying a test technique, as in the lesson on the test design activity. Test design identifies them; execution exercises them. The formula never changes: coverage = items exercised ÷ items identified × 100%.

Report the counts with the percentage ("9 of 12 criteria") and the build they apply to. In the ISTQB definition an item is covered once a test has exercised it, whatever the result, so for requirements and risks give two numbers: items with a test that ran on that build, and items whose tests passed. Only the second says the item works: a failed test leaves its item open, and a blocked or unrun test has exercised nothing yet. Requirements and risk coverage come from traceability links in a test management tool or matrix; code coverage comes from a tool that instruments the code (adds counters that record what runs) while the tests run, usually in the CI pipeline. What changes from one view or technique to the next is the item:

• Requirements: acceptance criteria or individual requirements.
• Risks: identified risks, grouped by risk level.
• Equivalence partitioning: partitions. Boundary value analysis: boundary values.
• Decision tables: the columns with feasible combinations of conditions.
• State transitions: states, valid transitions or all transitions.
• Code: executable statements, or branches (in coverage tools, each outcome of each decision).

Try it: Requirements → Test Cases Try it: State Transition Generator

Worked example: four coverage numbers for one feature

A checkout feature applies a discount code. Acceptance criteria: AC1, a valid code reduces the total by its percentage; AC2, an expired code is rejected with a clear message; AC3, codes apply only to totals of 100.00 SAR or more; AC4, one code per order. Product risks: R1 (high) a code is applied twice to one order; R2 (high) an expired code is accepted; R3 (medium) the 100.00 threshold is applied on the wrong side.

The developer wrote one unit test for the discount function; the tester ran three system tests on build 57, with TC-03 checking totals of 99.99 and 100.00. The code and reports are below. What each number says:

• Statement coverage 100% (4 of 4): every statement of applyDiscount ran in the unit test, yet no unit test tried a total under 100.00.
• Branch coverage 50% (1 of 2): the false outcome of total >= 100 never ran. A unit test with a total of 99.99 would cover it.
• Acceptance criteria: 3 of 4 have a test (75%), but only 2 of 4 have a passing test (50%). TC-02 failed and AC4 has no test.
• Risks: 2 of 3 have a test, but only 1 of 3 has a passing test, and neither high risk does: R1 has no test and the test for R2 failed. This is the number the release decision needs.
• Nothing in applyDiscount handles "one code per order". If AC4 was never implemented, no code coverage report can show it: a tool measures only code that exists.

// discount.ts
export function applyDiscount(total: number, percent: number): number {
  let result = total;
  if (total >= 100) {
    result = total - (total * percent) / 100;
  }
  return result;
}

// discount.test.ts (Jest): the only unit test
import { applyDiscount } from './discount';

test('10% off an order of 150', () => {
  expect(applyDiscount(150, 10)).toBe(135);
});

Code coverage (unit tests)   Statements   Branches
discount.ts                  4/4  100%    1/2  50%

Traceability, system tests on build 57
Criterion   Test     Result     Risk
AC1         TC-01    passed     -
AC2         TC-02    failed     R2 (high)
AC3         TC-03    passed     R3 (medium)
AC4         -        no test    R1 (high)

Try it: Requirements → Test Cases Try it: Risk-Based Test Generator

Why 100% coverage does not mean bug-free

Coverage tells you how much of a list was exercised, not whether the list was complete, whether the tests checked the right things, or whether the software is correct. Two of the seven testing principles apply: testing shows the presence of defects, not their absence, and exhaustive testing is impossible.

• The denominator holds only what you identified. A missing requirement or an unlisted risk never appears in the percentage, so 100% of an incomplete list is still incomplete.
• Executing is not checking. A test with no assertion (no check of the result) that calls applyDiscount(150, 10) gives the same 100% statement coverage as one that checks the result is 135.
• Data matters. applyDiscount(150, 110) returns a negative total, and full branch coverage would not reveal it. The syllabus's own example is a division that fails only when the divisor is zero.
• Code coverage cannot see omissions: code that was never written has no statements to miss.
• A target can become a game: people add tests that run code without checking anything. Reviewing assertions, or mutation testing (small deliberate code changes that a good test suite must detect), helps show whether covered code is really checked.

Try it: AI Test Coverage Analyzer

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISO/IEC/IEEE 29119-1:2022 — Software and systems engineering — Software testing — Part 1: General conceptsISO/IEC/IEEE · 2022The international software-testing standard: concepts, processes, documentation and techniques.