Pairwise Testing

Why it matters

Many failures appear only when two settings meet: a layout that breaks only for Arabic in Firefox, a price that is wrong only for yearly billing in US dollars. Testing each value once misses these interactions, and testing every combination is rarely possible, because the count multiplies with every parameter.

Pairwise testing is a systematic middle ground. Instead of "the configurations we usually try", the team gets a small set whose coverage can be stated and checked, such as "all 27 value pairs in 9 tests".

Why testing all combinations explodes

A parameter is anything that varies independently: browser, operating system, interface language, user role, payment method, feature flag. The number of combinations is the product of the value counts, so every new parameter multiplies the total.

At the other extreme is each choice coverage from the lesson on equivalence partitioning: every value appears in at least one test, so three tests are enough when every parameter has three values. It is cheap, but it never deliberately puts two particular values together.

• 3 browsers × 3 operating systems × 3 languages = 27 combinations.
• Add screen size (phone, tablet, desktop): 27 × 3 = 81.
• Add two user roles (guest, signed-in): 81 × 2 = 162, or 27 hours at 10 minutes per manual run.
• Ten parameters with three values each: 3^10 = 59,049 combinations.

What all-pairs coverage means

A value pair is one value of one parameter together with one value of another, for example Browser = Chrome with Language = Arabic; the ISTQB glossary calls it a pair of parameter-value pairs. Value pairs are the coverage items of this technique: pairwise coverage is the percentage of value pairs exercised by at least one test, and a pairwise set reaches 100%.

The set stays small because one test covers several pairs at once: Chrome · Windows · Arabic covers Chrome–Windows, Chrome–Arabic and Windows–Arabic. The more parameters a test has, the more pairs it covers, so a pairwise set grows slowly as parameters are added.

No set can be smaller than the value count of the largest parameter times that of the second largest: 4 browsers and 3 operating systems need at least 12 tests. Most tools use greedy algorithms that find a small set, not always the smallest; what matters is 100% pair coverage.

Parameters:        Browser (3), OS (3), Language (3)

All combinations:  3 × 3 × 3                  = 27 tests

Value pairs:       Browser–OS         3 × 3  =  9
                   Browser–Language   3 × 3  =  9
                   OS–Language        3 × 3  =  9
                   total                     = 27 pairs

Pairs in one test: 3 (one per pair of parameters)
Fewest tests:      3 × 3 (two largest)       =  9 tests
                   27 pairs / 3 per test     =  9, so each pair appears once

Try it: Pairwise Testing Generator

Worked example: browsers × operating systems × languages

A checkout page must work in Chrome, Firefox and Edge, on Windows, macOS and Linux, in English, Arabic (right to left) and Hindi. Safari is left out because it runs only on Apple platforms, so Safari–Windows is impossible; the rules that exclude such pairs, called constraints, are the subject of the lesson on combinatorial testing.

Build the set by hand. Draw a grid with the browsers as rows and the operating systems as columns: its 9 cells are the 9 browser–OS pairs, one test each. Write a language in each cell so that every language appears once in every row and once in every column. Every row then holds all browser–language pairs, and every column all OS–language pairs.

• Each row is a configuration, not a test case: run the same checkout scenario, with its steps and expected results, in all nine.
• The Pairwise Testing Generator may return 10 tests for this input; any set with 100% pair coverage is correct.
• Chrome on Windows in Arabic is not in the set, although each of its pairs is. If analytics show that most customers use it, add it as a tenth test: pairwise testing treats all pairs as equally important, risk does not.

#  Browser  OS       Language
1  Chrome   Windows  English
2  Chrome   macOS    Arabic
3  Chrome   Linux    Hindi
4  Firefox  Windows  Arabic
5  Firefox  macOS    Hindi
6  Firefox  Linux    English
7  Edge     Windows  Hindi
8  Edge     macOS    English
9  Edge     Linux    Arabic

9 tests instead of 27 combinations; 27 of 27 value pairs covered

Try it: Pairwise Testing Generator

Where pairwise testing stops

Pairwise testing assumes that an interaction failure needs at most two particular values to appear. It guarantees every pair, not every triple: a failure that needs Chrome, Windows and Arabic together is found only if that triple happens to be in the set.

• Interactions of three or more values can be missed. For high-risk features, use 3-wise testing, which covers every triple of values instead of every pair; the lesson on combinatorial testing shows how.
• It does not choose the values: a partition or boundary missing from the value list is never tested, so select values with equivalence partitioning and boundary value analysis first.
• It knows nothing about impossible or important combinations: impossible pairs need constraints, and key configurations must be added deliberately.
• Invalid values do not belong in the combined set: a test with two invalid values usually stops at the first error and hides the second, which is called masking. Test each invalid value on its own.
• It does not fit every problem: use a decision table when a business rule decides the outcome from specific combinations, and state transition testing when the order of events matters.

Frequently asked questions

References

• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• ISO/IEC/IEEE 29119-1:2022 — Software and systems engineering — Software testing — Part 1: General conceptsISO/IEC/IEEE · 2022The international software-testing standard: concepts, processes, documentation and techniques.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.