Test Suites and Test Runs
A test suite is a set of test cases, procedures or scripts chosen to run together for one purpose: a smoke suite, a checkout regression suite, the tests for this sprint’s stories. A test run is one execution of a suite on a specific version of the software, in a stated environment, with a result recorded for every test. In ISTQB terms, the suite is the set of tests and the run is its execution on one version of the test object.
Example: the checkout regression suite holds 24 test cases. When build 2.4.0-rc2 (release candidate 2 of version 2.4.0) reaches staging, the tester creates a run of that suite for that build. Every test starts as not run and ends as passed, failed, blocked or skipped. When rc3 arrives, a new run starts, and the rc2 results remain the record of what rc2 did.
Why it matters
Without suites, every round of testing starts with "what do we run this time?", and the answer depends on who is asked. Without runs, results float free of builds: a pass recorded on Monday cannot be told apart from a pass on Thursday’s build, which has three more changes, and the team releases on results that describe a different version.
For the tester, the run is the record others read: what is done, what failed, what could not be tested and why. Status reports and release decisions are built on it, so a carelessly set status, such as a blocked test marked failed, misleads everyone downstream.
Test suites: grouping tests for a purpose
In the lesson on test implementation, procedures were grouped into suites. In many test management tools, each test case is stored once in a repository, usually in folders by feature, and a suite selects the cases needed for one purpose. A case can then belong to several suites; the suite refers to it instead of copying it, so a correction to its steps reaches every suite.
Some tools build a suite from a fixed list, others from a query or tag, such as every case tagged checkout with priority High. Automated suites work the same way: a folder, file or tag that the pipeline runs together. Common purposes:
- Smoke: a few checks that show a new build is stable enough to test further.
- Feature: the new tests for the current sprint or release.
- Regression: tests of existing behaviour, run in full or as a risk-based selection after a change.
- Configuration: the same tests for one browser, device or language, where results can differ.
Try it: Test Case Generator Try it: Regression Scenario Generator Try it: Risk-Based Test Generator
Test runs and test cycles
The ISTQB Foundation syllabus describes test execution as running the tests according to the execution schedule, in test runs, comparing actual with expected results and logging the outcome. The run holds that log: a result for each test, with who executed it, when and on which build.
A suite answers "what do we test?" and is reused for months. A run answers "what happened on this build, in this environment?" and is closed when that testing ends, so running the suite on a new build means a new run, not overwritten results. Many teams group the runs for one build or test period into a test cycle, such as "Release 2.4, system test cycle 2" with one run per browser. A run header records what makes the run reproducible:
Run: Checkout regression / 2.4.0-rc2 / Chrome
Suite: Checkout regression (24 test cases)
Cycle: Release 2.4, system test cycle 2
Build: 2.4.0-rc2
Environment: staging, payment sandbox, data baseline CHK-BASE-3
Configuration: Chrome desktop, English UI
Assigned: Sara CHK-01..12, Omar CHK-13..24
Planned: day 1 to day 2Five statuses and what each one means
Each test in a run has one current status. Agree on what each status means before the first run and write it down, or the counts will mean different things to different testers. The common core:
- Not run: in the run’s scope but not executed yet; some tools say "untested" or "to do".
- Passed: executed, and every actual result matched its expected result.
- Failed: executed, and at least one actual result differed from the expected result. The cause still needs analysis: a product defect, a mistake in the test or an environment problem.
- Blocked: could not be started or completed because something outside the test stopped it, such as an environment outage, missing data or access, or a defect in an earlier step. Record the reason and link the blocking issue.
- Skipped: deliberately not executed in this run by a recorded decision, for example because the feature is switched off here or the test does not apply to this configuration. Automated test reports also show skipped for tests disabled in the code, and some frameworks use it when a set-up step failed, which is closer to blocked: check which one before you count it.
| # | From | To | Event or decision |
|---|---|---|---|
| 1 | Not runStart | PassedPass | Executed: every actual result matches its expected result |
| 2 | Not runStart | FailedFail | Executed: at least one actual result differs from the expected result |
| 3 | Not runStart | BlockedCaution | Cannot start or finish: environment down, data or access missing, or an earlier defect in the way; the cause is linkedalternative path |
| 4 | BlockedCaution | Not runStart | Blocker removed: back in the queue, then executedalternative path |
| 5 | Not runStart | Skipped | Recorded decision not to run it in this run, with the reasonalternative path |
| 6 | FailedFail | PassedPass | Retest on a build with the fix passes: a new result; the failed one stays in the historyalternative path |
Executed = Passed + Failed. Blocked, Skipped and Not run are counted separately. Tools name these statuses differently.
Worked example: tracking a regression run
Below is the checkout regression run at the end of day 1. Every status other than passed and not run carries a note.
Executed means passed plus failed: 17 of 24. The pass rate, passed divided by executed, is 15 of 17, about 88%, but on its own it hides 3 blocked and 3 not-run tests still to do and one skipped test, so the summary shows all five numbers. Next steps:
- Failed, CHK-05 and CHK-06: both link to one defect, DEF-512, and are retested (confirmation testing, in ISTQB terms) on the build that contains the fix, each getting a new result with that build recorded.
- Blocked, CHK-02 to CHK-04: all wait on environment issue ENV-77, which has an owner. When the sandbox is back, they return to not run and are executed, never just marked passed.
- Skipped, CHK-07: the reason and who decided are recorded, so the coverage gap shows in the report.
- Not run, CHK-22 to CHK-24: planned for day 2; if time runs out, the test lead decides whether to skip them or extend the run.
Run: Checkout regression / 2.4.0-rc2 / Chrome end of day 1
ID Test case Status Note
CHK-01 Pay cash on delivery, single item Passed
CHK-02 Pay by card, 3-D Secure challenge Blocked ENV-77 payment sandbox down
CHK-03 Pay by card, declined card Blocked ENV-77
CHK-04 Refund to card after cancellation Blocked ENV-77
CHK-05 Apply promo code SAVE10 Failed DEF-512 discount applied twice
CHK-06 Remove promo code Failed DEF-512 (same cause)
CHK-07 Pay with a digital wallet Skipped wallet off on staging (test lead)
CHK-08 .. CHK-21 (14 tests) Passed
CHK-22 .. CHK-24 (3 tests) Not run planned for day 2
Planned 24 Passed 15 Failed 2 Blocked 3 Skipped 1 Not run 3
Executed (passed + failed) 17 Pass rate 15 / 17 = 88%Common mistakes
- Marking a blocked test as failed, which sends developers after a problem that is not theirs.
- Marking a test passed because it passed on the last build, or because the steps that could run were fine.
- Overwriting the previous build’s results instead of starting a new run, so nobody can tell which build a result belongs to.
- Skipping tests silently to finish on time. A skip needs a reason and an owner of the decision, or the report claims coverage that did not happen.
- Reporting only the pass rate, which says nothing about tests that are blocked or not run.
- Building suites around people ("Ahmed’s tests") instead of purposes, so nobody knows what a suite covers once its owner moves on.
The QA perspective
On most teams the test lead creates the runs from the plan: which suites, builds, environments and configurations, and who runs what. Testers keep their part current during the day, because developers, the test lead and the product owner read it to decide what to do next. Link every failure to its defect and every block to its cause, and say in the stand-up what is blocked and who owns it.
Before a run starts, ask which build exactly, which environment and configuration, which suites are in scope, and what your statuses mean here. When a new build arrives, compare the status lists of the two runs, for example with a text diff: a test that passed on rc2 and fails on rc3 is a likely regression.
Practice exercise
Track a run for a login suite
A login suite has 10 test cases, LOG-01 to LOG-10, run on build 5.1.0 on staging in Chrome. LOG-01 to LOG-05 match their expected results. LOG-06 "Log in with a locked account" shows the dashboard instead of the lockout message. LOG-07 and LOG-08 need a one-time code by SMS, and the SMS stub is down. LOG-09 "Log in with SSO" (single sign-on) applies only to the enterprise edition, which is not on staging; the test lead agrees not to run it. LOG-10 has not started when the day ends.
Record the run’s results as you would in a test management tool.
- Assign each test a status, and a note where one is needed.
- Count each status, then executed and the pass rate.
- Write the next action for every test that did not pass.
Show the model answer
Statuses: LOG-01 to LOG-05 passed. LOG-06 failed, linked to a new defect report; a locked account that can log in bypasses a security control. LOG-07 and LOG-08 blocked, note: SMS stub down, linked to the environment issue. LOG-09 skipped, note: enterprise edition not on staging, decided by the test lead. LOG-10 not run.
Summary: passed 5, failed 1, blocked 2, skipped 1, not run 1. Executed = 6. Pass rate = 5 / 6, about 83%. Still to execute: 3 tests (LOG-07, LOG-08 and LOG-10), plus the retest of LOG-06 after the fix.
Next: LOG-06 is retested on the build with the fix and gets a new result. LOG-07 and LOG-08 return to not run when the stub is back, then are executed. LOG-09 goes into a run on an environment with the enterprise edition, or the gap is stated in the report. LOG-10 runs first the next day.
Frequently asked questions
What is the difference between a test suite and a test run?
A suite is a reusable set of tests chosen for a purpose, such as smoke or checkout regression. A run is one execution of a suite, or part of one, on a specific build in a specific environment, with a result for each test. One suite usually has many runs over its life.
Is a blocked test the same as a failed test?
No. A failed test was executed and produced a result different from the expected one. A blocked test could not be executed or finished, so it says nothing yet about whether the feature works. Blocked tests are counted separately and need a reason and an owner for the blocker.
Should a retest after a fix go into the same run or a new one?
It depends on the team’s convention and its tool. Many teams add the retest as a new result in the same run, with the build that contains the fix; others create a run for the new build. Either way, keep the failed result in the history and record which build the new result belongs to.
What is a test cycle?
Teams use "test cycle" for a round of test execution, often all the runs for one build or one test period, such as "system test cycle 2". Tools use the words run, execution and cycle in different ways, so learn what your team’s tool means by each.
References
- ISTQB Certified Tester Foundation Level (CTFL) v4.0The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
- ISTQB GlossaryThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
Practise with Mutqan tools
Generate structured test cases (id, title, preconditions, steps, expected result, priority) from a feature description and field list.
Generate regression scenarios from a change description and a list of affected and related modules.
Score features by likelihood and impact, then generate a prioritised risk-based test plan.
Calculate percentages, percentage change, increase/decrease and what percent X is of Y.
Compare two texts line by line or word by word and highlight differences.
Go deeper
Your progress on this topic
Not started
Mark the topic complete when you have studied it and done the exercise.
Sign in to track your progress
Mark topics complete, continue where you left off and unlock specialisation exams.