What to Automate (and What Not To)

Why it matters

Automating the wrong checks costs a product twice: time goes into writing and repairing scripts for checks that rarely run or keep changing, and important checks stay manual because the budget is spent. CTFL v4.0 names both causes among the risks of test automation: using a tool where manual testing is more appropriate, and inaccurate estimates of the effort needed to maintain test scripts.

Testers know which checks are repeated every release, which need judgement and which features are still moving, so the decision is part of the job. With a clear rule and honest numbers you can argue for what to automate first, and protect time for the exploratory testing that no script does.

What is worth automating

A good candidate is a check that a person would otherwise repeat many times, with the same steps and a precise expected result. Typical candidates:

• Regression checks that must pass on every release: sign-in, checkout and the business rules behind them.
• Smoke checks after each deployment, which confirm the build is worth testing at all.
• API and contract checks against a specification: usually faster and more stable than the same check through the UI.
• Data-driven checks: the same steps over many rows of boundary values, currencies, locales, roles or browsers.
• Work no person can do by hand, such as generating load from hundreds of concurrent users.
• Repetitive preparation: creating test data and resetting environments, which saves manual testers time too.

Try it: Regression Scenario Generator Try it: OpenAPI → Test Cases

What not to automate, or not yet

These checks are usually better done by a person. Some of them are "not yet" rather than "never": a feature that settles, or a dependency that gets a stub (a simple stand-in that replaces the real service and returns fixed answers), becomes a candidate later.

• Exploratory testing and the first pass over a new feature, while the expected behaviour is still being discovered. A script checks only what someone anticipated.
• Judgement: whether wording is clear, whether an Arabic screen reads naturally right to left, whether a flow is confusing. Tools can measure parts such as colour contrast; the verdict stays human.
• One-off checks, such as verifying last weekend's data migration: run it once by hand or with a throwaway query.
• Features that change every sprint: a screen under redesign, an A/B experiment, a prototype.
• What the test cannot control: a CAPTCHA, which exists to stop automation, or a real SMS reaching a real phone. Ask for a test configuration or a stub, and check the real thing by hand from time to time.

The rule: frequency, determinism, stability

Ask three questions about every candidate, in this order, and automate only when all three answers are yes.

Frequency: how many times will the check run during its useful life? Multiply releases by data rows, browsers and languages: 40 rows at 26 releases a year is 1,040 executions. A check that runs once does not repay the build and upkeep of an automated test; if it is too large to do by hand, a throwaway query or script is enough.

Determinism: can a script decide pass or fail on its own, and does the same input give the same result on every run? "The total is 115.00 SAR" is a definite expected result; "the page looks good" is not. A result that depends on the clock, random data, a shared account or a live third-party service is not repeatable yet: control the time, let the test create its own data or stub the service, then automate. Martin Fowler's article on non-determinism explains the cost of skipping this: once tests fail for no reason, a team learns to ignore failures.

Stability: will the behaviour, and the interface the check uses, stay the same for the next few months? Business rules change less often than layouts. If not, test it by hand now and keep it on a candidate list.

• Three yes answers make a candidate, not an order: rank candidates by risk, as in risk-based testing.
• Automate at the lowest level that can express the check, for example a price rule through the API rather than the UI. The lesson on the test automation pyramid develops this.
• Then run the numbers: a check whose upkeep per run is close to its manual cost never pays back.

Try it: Risk-Based Test Generator

Automation ROI, calculated honestly

ROI compares what automation saves with what it costs over the same period. For a regression check the measurable saving is the manual execution time no longer spent; the cost is everything the automated check consumes, not only writing it.

• Count only manual runs that would really happen. If regression is run by hand once per release, that is the baseline, even if the script will run on every commit.
• Report the other benefits in their own units: feedback within minutes of a commit, consistent results, testers freed for exploratory work. Do not convert them into invented hours.
• Include upkeep from day one: every product change that touches the check, every failed run someone must read, every flaky test someone must fix.
• Compare break-even with the check's expected life: if the screen will be redesigned in four months, a six-month break-even never arrives.
• State your assumptions, and replace the estimates with actual hours after a few months.

Period: one year. Run: one regression cycle, for example one release.

Manual cost      = manual hours per run x runs a person would really do
Automation cost  = build hours              (write, review, test data, setup)
                 + upkeep hours per run x runs
                   upkeep = fixes after product changes, reading failed runs,
                            fixing flaky tests, CI time and licences in hours
ROI              = (manual cost - automation cost) / automation cost
Break-even       = build hours / (manual hours per run - upkeep hours per run)
                   no break-even when upkeep per run >= manual hours per run

Try it: Percentage Calculator

Worked example: three checks from a web shop

A web shop releases every two weeks and runs its manual regression list before each release: 26 runs a year. The team estimates three candidates; upkeep includes maintenance and reading failed runs.

• A passes all three questions: automate it through the API. It will also run on every commit; that faster feedback is real, but it is not part of the 52 hours. From year two, with no build cost, it saves 39 hours a year.
• B fails stability, and the numbers agree: upkeep per release is four times the manual check. It stays on the manual release checklist.
• C first failed determinism, because the code went to a real phone. The developers added an SMS stub to the test environment, disabled in production, and the check now passes all three questions. A tester still signs in with a real phone after each production release, because the stub cannot prove that real SMS delivery works; that check stays on the manual release checklist and is not part of the 25 hours the script replaces.

Release every 2 weeks: 26 manual regression runs a year

A  Cart price and VAT rules, 40 data rows, checked through the API
   Manual       2.0 h x 26                  = 52 h a year
   Automated    16 h build + 0.5 h x 26     = 29 h in year 1
   Break-even   16 / (2.0 - 0.5) = 10.7     -> release 11, about 5 months
   ROI year 1   (52 - 29) / 29              = 79 %

B  Home page promo banner, redesigned every month, checked in the UI
   Manual       0.25 h x 26                 = 6.5 h a year
   Automated    6 h build + 1.0 h x 26      = 32 h in year 1
   Break-even   none: upkeep 1.0 h per run > manual 0.25 h per run

C  Sign-in with an SMS one-time code, using an SMS stub in the test environment;
   today checked by hand after each of ~100 staging deploys a year
   Manual       0.25 h x 100                = 25 h a year
   Automated    10 h build + 0.05 h x 100   = 15 h in year 1
   Break-even   10 / (0.25 - 0.05) = 50     -> deploy 50, about 6 months
   ROI year 1   (25 - 15) / 15              = 67 %

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• Certified Tester Advanced Level Test Automation Engineering (CTAL-TAE) v2.0ISTQB · 2024ISTQB's advanced syllabus on designing, implementing and maintaining test automation solutions and their architecture.
• Eradicating Non-Determinism in Tests (Martin Fowler)martinfowler.com · 2011Why flaky tests destroy a suite's value, their common causes (isolation, asynchrony, remote services, time, resource leaks) and quarantine as a response.
• The Practical Test Pyramid (Ham Vocke)martinfowler.com · 2018A widely cited explanation of how to balance unit, integration, contract and UI tests in an automated test portfolio.