Combinatorial Testing
Combinatorial testing is a black-box test technique that designs test cases to exercise chosen combinations of values of several parameters. Instead of running every combination, you choose a strength t: for any t parameters, every combination of their values appears in at least one test. A checkout with five parameters of four values each has 1,024 combinations; 16 tests can cover every pair of values.
Pairwise testing is combinatorial testing at strength 2. This lesson adds choosing the strength, constraints between parameters, the classification tree method, and when pairs are not enough.
Why it matters
Many failures appear only when particular values meet: express delivery, a fixed-amount coupon and a business account each work on their own and in any pair, yet a business customer who chooses express delivery with a fixed-amount coupon is charged the wrong fee. A hand-picked list of combinations leaves gaps nobody can see; a combinatorial set has a stated, checkable goal, such as "every allowed triple of delivery, customer type and coupon", so what is left out is a decision, not an accident.
For the tester, it turns a test count into an argument: what the set guarantees, what it does not, and what more strength would cost.
Strength: how many parameters must meet
Strength 1 is the each choice coverage you met in equivalence partitioning: every value appears at least once. Strength 2 (pairwise) puts every pair of values of any two parameters together in some test, strength 3 every triple, and when t equals the number of parameters you are back to all combinations. The ISTQB glossary calls this n-wise testing; NIST and many tools say t-way. A test set that reaches a given strength is called a covering array.
No t-way set can be smaller than the product of the t largest numbers of values: two parameters with four values form 16 pairs and one test holds only one of them, so pairwise needs at least 16 tests. For the checkout, sets of exactly 16, 64 and 256 tests exist. Each step up multiplies the minimum by roughly the number of values, so strength is chosen by risk, not maximised by default.
Minimum sizes; for this model, sets of exactly these sizes exist. Constraints and uneven numbers of values change the numbers, and a generator may return more rows than the minimum.
Text version
Checkout: 5 parameters x 4 values each
Strength Every test set must contain Minimum
1-way (each choice) every value at least once 4
2-way (pairwise) every pair of values, any 2 parameters 16
3-way every triple of values, any 3 parameters 64
4-way every combination of any 4 parameters 256
5-way (all combinations) 4 x 4 x 4 x 4 x 4 1,024Constraints between parameters
Some combinations are impossible: the shop offers Apple Pay only on the web and in the iOS app. If the generator does not know this, some rows cannot be executed, and deleting them afterwards also deletes the other pairs they carried. Declare constraints before generating, so every allowed combination is covered and no forbidden one appears. The notation below is generic; each tool has its own. The Pairwise Testing Generator takes a forbidden combination as Payment = Apple Pay AND Device = Android app and accepts the implication below as written.
- Forbidden combination:
NOT (Payment = Apple Pay AND Device = Android app). - Implication:
IF Customer = guest THEN Payment <> bank transfer(bank transfer needs an account, so the page does not offer it to guests). - Conditional parameter: card scheme matters only for card payments; give it a "not applicable" value or model it as a refinement in a classification tree.
- Not a constraint: a combination users can reach but the system must refuse. Keep it as a test with its expected result; invalid values, as in pairwise testing, get their own negative tests.
The classification tree method
The classification tree method, described by Grochtmann and Grimm in 1993, draws the model so the team can review it. The test object is the root; below it are classifications, the aspects that may change the result; each is split into classes, which are equivalence partitions. A class can be refined by another classification, such as card scheme under "card".
Under the tree, a combination table holds one test case per row, marking one class of each classification that applies. A coverage rule decides the number of rows: every class once, every pair, a higher strength, or all combinations.
Checkout (test object)
|-- Payment
| |-- card --> Card scheme: Visa | Mastercard | mada
| |-- Apple Pay
| |-- bank transfer
| `-- cash on delivery
|-- Delivery: standard | express | same-day | store pickup
|-- Customer: guest | new account | returning | business
|-- Device: desktop web | mobile web | Android app | iOS app
`-- Coupon: none | percentage | fixed amount | free shipping
Combination table (one row = one test case, one class per classification)
TC Payment Scheme Delivery Customer Device Coupon
1 card mada express business iOS app fixed amount
2 Apple Pay - store pickup guest desktop web free shipping
3 bank transfer - same-day returning Android app noneWhen to go beyond pairs
Pairwise is a sensible default: work at NIST reports that most failures in the software it studied were triggered by one parameter or by two interacting, and progressively fewer by three or more. That is a tendency, not a guarantee.
Going beyond pairs rarely means raising t for the whole model. Use mixed strength: keep 2-way for all parameters and raise t only for the group that interacts, as the worked example below does for the delivery fee. If the logic is written as explicit rules, a decision table may test it better. Consider 3-way or higher for a group of parameters when:
- One rule combines three or more parameters: fees, prices, eligibility, permissions.
- A failure would be expensive: money, health, legal duties or lost data.
- A past defect involved three parameters together.
- The tests are automated and cheap, so 64 rows cost little more than 16.
Worked example: planning a checkout test set
The checkout has five parameters with four classes each from equivalence partitioning; card scheme is left out because, as a developer confirmed, the payment provider handles all schemes alike. A requirement says the delivery fee depends on delivery option, customer type and coupon together. The product owner confirmed three constraints: Apple Pay is not offered in the Android app, cash on delivery is not offered for store pickup, and guests are not offered bank transfer.
The plan runs pairwise through the user interface, where tests are slow, and all 64 fee triples as data-driven API tests, where they are cheap. Without constraints 16 tests would cover every pair; with them 16 is out of reach. Each payment method and each device must meet all four coupons, so in 16 rows each would appear exactly four times. Apple Pay's four rows can then use only three devices and the Android app's four rows only three payment methods, so two Payment–Device pairs repeat and the 15 allowed pairs need at least 17 rows. The Pairwise Testing Generator returns 22 rows for this model. Store pickup with a free-shipping coupon is possible, so it is not a constraint but one of those triples, with the expected result the product owner defined. In total that is about 8% of the 1,024 combinations, with a stated guarantee for each part.
Parameters (4 values each)
Payment card | Apple Pay | bank transfer | cash on delivery
Delivery standard | express | same-day | store pickup
Customer guest | new account | returning | business
Device desktop web | mobile web | Android app | iOS app
Coupon none | percentage | fixed amount | free shipping
Constraints
NOT (Payment = Apple Pay AND Device = Android app)
NOT (Payment = cash on delivery AND Delivery = store pickup)
IF Customer = guest THEN Payment <> bank transfer
Plan
UI 2-way, all 5 parameters at least 17 tests (generator: 22)
API 3-way, Delivery x Customer x Coupon 64 tests, every fee triple
Not planned: all combinations 1,024 (832 allowed)Common mistakes
- Raising the strength for every parameter "to be safe", which multiplies the test count; raise it only where parameters interact.
- Declaring a combination a constraint because the system must refuse it, so its error handling is never tested.
- Writing a constraint wider than the rule, such as excluding Apple Pay everywhere instead of only in the Android app, which removes valid combinations without anyone noticing.
- Drawing a classification tree whose classes overlap or leave gaps, so an input fits two classes or none.
- Choosing the strength once and never revisiting it when new rules or incidents show that more parameters interact.
- Treating t-way coverage as proof: it shows which combinations ran, not that every result was checked or that the model is complete.
The QA perspective
Build the model during test analysis and design with the product owner, who confirms which combinations are impossible and which must show an error, and with a developer, who knows which parameters the code really combines. A function that computes a fee from three inputs is a strong hint for 3-way.
Record the strength of each group of parameters and the reason for it next to the model, so reviewers see why there are 22 or 86 tests. Revisit it after incidents: if a production failure needed three values together, move those parameters into a 3-way group and note why.
Practice exercise
Plan a combinatorial test set for a hotel booking form
A hotel booking form has the six parameters and three rules below. Model the form, sort the rules into constraints and tests, and plan the strength for each part. Then check the pairwise part with the Pairwise Testing Generator.
Hotel booking form (web and app)
Room standard | deluxe | suite
Guests 1 | 2 | 3
Board room only | breakfast | half board
Payment pay now by card | pay at hotel
Member loyalty member | not a member
Channel web | app
Rule A: A standard room holds at most 2 guests. The guest selector does not
offer 3 when a standard room is selected.
Rule B: Pay at hotel is only for loyalty members. If a non-member chooses it,
the page shows "Sign in to pay at hotel".
Rule C: The price is calculated from room, guests and board together.- Draw a classification tree with one classification per parameter.
- Decide whether Rule A and Rule B are constraints or tests.
- Calculate all combinations, the allowed ones, and the minimum sizes of a 1-way and a 2-way set.
- Use Rule C to decide where 3-way is needed and its minimum number of tests.
- Generate a pairwise set with Rule A declared, then check its size, its pair coverage and every row against Rule A.
Show the model answer
Tree: the booking form is the root, with one classification per parameter and the listed values as its classes.
Rule A is a constraint, because the form makes the combination impossible: NOT (Room = standard AND Guests = 3). Rule B is not: a non-member can choose pay at hotel, so it stays a test whose expected result is the sign-in message.
All combinations: 3 × 3 × 3 × 2 × 2 × 2 = 216; Rule A removes 1 × 1 × 3 × 2 × 2 × 2 = 24, leaving 192. A 1-way set needs at least 3 tests and a 2-way set at least 9, because Room × Board alone has 9 pairs. With Rule A, 9 is not reachable. Nine rows would have to hold each Room–Board and each Guests–Board pair exactly once, so every room type and every number of guests would appear exactly three times. The three standard-room rows can use only 1 or 2 guests and the three 3-guest rows only deluxe or suite, so two Room–Guests pairs repeat and the 8 allowed Room–Guests pairs need 10 rows. A 10-row set exists, so 10 is the constrained minimum.
Rule C combines Room, Guests and Board: 27 triples minus the 3 that Rule A forbids leaves 24, so at least 24 tests, with the other parameters spread so that their pairs stay covered. That is mixed strength: 3-way for these three parameters, 2-way for the rest.
Enter each parameter on its own line, such as Room: standard, deluxe, suite, and declare Rule A as Room = standard AND Guests = 3, which the tool treats as a forbidden combination. It should report 92 value pairs (93 minus the forbidden one), all covered. The generator may return 11 rows instead of the minimum of 10; any set that covers every allowed pair is correct. If a row pairs a standard room with 3 guests, the constraint was not applied: regenerate with it declared instead of deleting the row.
Frequently asked questions
Is combinatorial testing the same as pairwise testing?
Pairwise testing is combinatorial testing at strength 2. Combinatorial testing is the general technique: any strength, mixed strengths for different groups of parameters, and constraints that remove impossible combinations.
Which strength should I choose?
Start with 2-way for all parameters. Raise it to 3-way for the parameters that one rule combines or where a failure is costly, and consider 4-way or more only for small, critical parts whose tests are automated. The strength is a risk decision, so record why you chose it.
Do I need a tool to build the test set?
Beyond a handful of parameters, yes: building covering arrays by hand is slow and easy to get wrong. The model is still your work. Check the output: the number of rows, that no forbidden combination appears, and, where the tool reports it, that coverage at the chosen strength is complete.
Does a classification tree replace equivalence partitioning?
No, it builds on it. The classes in the tree are equivalence partitions, and boundary values can be classes too. The tree organises the partitions of several aspects in one picture, and the combination table chooses which of them are tested together.
References
- ISTQB GlossaryThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
- ISO/IEC/IEEE 29119-1:2022 — Software and systems engineering — Software testing — Part 1: General conceptsThe international software-testing standard: concepts, processes, documentation and techniques.
- ISTQB Certified Tester Foundation Level (CTFL) v4.0The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
Practise with Mutqan tools
Generate a minimal pairwise (all-pairs) test set from parameters and their values.
Derive valid and invalid equivalence classes for each input and pick representative test values.
Build decision tables from conditions and actions, generate all rule combinations and collapse impossible ones.
Go deeper
Your progress on this topic
Not started
Mark the topic complete when you have studied it and done the exercise.
Sign in to track your progress
Mark topics complete, continue where you left off and unlock specialisation exams.