Test Monitoring and Test Control

Why it matters

Every test plan is an estimate made before the facts are known: environments arrive late, builds break, defects cluster in one module. Without monitoring, these problems surface on the last day, when the only options left are to release with unknown risk or to delay at the worst moment.

With monitoring and control, when time runs short the low-risk tests are dropped by an explicit decision, not the payment tests by accident. And the tester can say "we are 22 tests behind, here is why, and here are the options" instead of "I am busy" — a message that gets blocked environments fixed.

Test monitoring: gathering information against the plan

Monitoring needs a baseline: the test plan, with its schedule and exit criteria, and the test execution schedule from the lesson on test implementation. A number alone says little; "26 tests executed" becomes information only next to "48 planned by today": 54% of the plan.

In ISTQB CTFL v4.0, test monitoring covers all test activities, not only execution, and checks whether the exit criteria are being met, for example coverage targets for risks or acceptance criteria. Take the data from where the work is recorded — the test management tool, the defect tracker, CI results — and keep the definitions stable so that today compares with yesterday. Typical information:

• Test progress: tests implemented, environment readiness, tests executed, passed, failed and blocked (unable to run because something they need is not available).
• Defects: found, fixed and open, by severity, and where they cluster.
• Coverage: the share of requirements, acceptance criteria or product risks covered by tests that have run, and how many of those tests passed.
• Risk and effort: product risks still untested, and effort spent against the estimate.

Try it: Percentage Calculator Try it: Requirements → Test Cases

Test control: corrective actions

Test control uses the information from monitoring to correct the course of testing. A good control decision answers three questions: what is the variance, which exit criterion or test objective does it threaten, and what is the smallest action that brings testing back on track? Deciding that no action is needed is also a control decision, if it is made on purpose.

Not every action is the tester’s to take. Reordering tests or adding an exploratory session is usually within the test team’s authority; moving a release date, cutting scope or accepting a residual risk (the risk that remains, for example because some tests did not run) belongs to stakeholders such as the product owner, and the tester brings them the facts and the options. Either way, the decision is recorded, the plan is updated and the next round of monitoring checks whether it worked. ISO/IEC/IEEE 29119-2 treats this loop as one test management process, beside test planning and test completion.

Examples of control decisions

CTFL v4.0 lists typical control directives: reprioritising tests when an identified risk becomes an issue (what might go wrong has gone wrong), re-evaluating whether a test item still meets the entry or exit criteria after rework, adjusting the schedule when the test environment is late, and adding resources where needed. In practice they answer signals that recur on almost every project:

• A risk becomes an issue (the payment provider changes its API mid-cycle): run the payment and checkout tests first.
• Defects cluster in one module: test it more deeply, for example with an exploratory session, and ask whether the defects share a root cause.
• The environment or test data is late: adjust the schedule, run what does not depend on it, and escalate with a named owner and a date.
• Rework lands in a tested area: run confirmation and regression tests, and accept the new build only if it meets the entry criteria, such as a passing smoke test.
• A build fails its smoke test: stop executing on it rather than logging dozens of failures with one cause.
• The forecast shows not all tests will run: give stakeholders the options — more time, more people, or leaving low-risk tests unrun with the residual risk recorded.

Try it: Risk-Based Test Generator Try it: Regression Scenario Generator

Worked example: day 4 of a 10-day test cycle

An online shop is testing release 4.2, which adds coupons and a new checkout. The plan: 120 system tests in 10 working days, 12 a day, scheduled feature by feature. Exit criteria: all 30 high-risk tests passed and no open Blocker or Critical defect. The test environment arrived two days late, at the start of day 3.

At the end of day 4 the test lead compares the data with the plan. The total is 22 tests behind, but the high-risk line matters more: 8 of 30 have run and 9 are blocked by a payment sandbox outage. Five of the seven defects are in coupons, one of them Critical. At 13 tests a day, about 16 tests will not run by day 10.

Release 4.2 - system test status, end of day 4 of 10

                          Planned by day 4   Actual   Variance
Tests executed                   48            26        -22
High-risk tests executed         20             8        -12
Tests blocked                     0             9         +9
Defects found                     -             7    (5 in coupons)
Open Critical defects             0             1         +1

Pace since the environment arrived (day 3): 13 tests a day
Forecast: 94 tests left, 6 days x 13 = 78, so about 16 will not run

Worked example: the control decisions

The test lead acts at once on what the test team controls and takes the schedule question to the product owner the same day:

• Reprioritise: the 13 unblocked high-risk tests run first on day 5; low-risk cosmetic checks move to the end.
• Test deeper where defects cluster: a time-boxed exploratory session on coupon combinations.
• Unblock: DevOps names an owner and a date for the sandbox; if it is not back by day 6, stakeholders hear that the high-risk exit criterion is at risk.
• Prepare for rework: when the Critical fix arrives, run confirmation testing and the coupon regression tests.
• Escalate: the product owner chooses between two extra days, a borrowed tester, or leaving 16 low-risk tests unrun with the residual risk recorded.
• Record and re-check: the plan is updated with each decision, its owner and its reason; the day 5 checkpoint checks whether the actions worked.

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISO/IEC/IEEE 29119-2:2021 — Software and systems engineering — Software testing — Part 2: Test processesISO/IEC/IEEE · 2021The test process model: organisational, test management and dynamic test processes, with their activities and outcomes.