Smoke vs Sanity vs Regression Testing
Smoke, sanity and regression testing are three answers to three different questions after a change. Smoke testing asks whether a build is stable enough to test at all. Sanity testing, in its most common meaning, asks whether a specific change or fix looks right before investing in broader testing. Regression testing asks whether the change broke anything that used to work. Mixing them up leads to either wasted effort — a two-hour regression run on a build that cannot log in — or false confidence — a five-minute smoke run treated as release sign-off.
This guide compares the three side by side, is honest about the fact that “sanity testing” is used inconsistently in the industry, explains how to choose which regression tests to run for a given change, and shows where each one sits in a CI/CD pipeline, with a worked release example.
Smoke, sanity and regression at a glance
- Question answered — Smoke: can we test this build? Sanity: does this change look right? Regression: did this change break something else?
- Scope — Smoke: broad and shallow, the critical paths across the whole product. Sanity: narrow and deeper, the changed area only. Regression: broad and deep, everything the change could affect, up to the whole suite.
- When — Smoke: immediately after every build or deployment. Sanity: after a fix or small change, before wider testing. Regression: after any change, before release, and on a schedule.
- Duration — Smoke: minutes. Sanity: minutes to an hour. Regression: from minutes (a selected subset) to hours (the full suite).
- Scripted or not — Smoke: scripted and ideally fully automated. Sanity: often partly unscripted, guided by the change. Regression: scripted, and the main target of automation.
- Outcome of failure — Smoke: build rejected, no further testing. Sanity: change sent back, broader testing postponed. Regression: defect raised; release blocked if the gate says so.
Smoke
Can we test this build?
- Scope:
- Broad and shallow: the critical paths across the whole product
- When:
- Immediately after every build or deployment
- Duration:
- Minutes
- Scripted or not:
- Scripted, ideally fully automated
- Outcome of failure:
- Build rejected, no further testing
Sanitymost common meaning
Does this change look right?
- Scope:
- Narrow and deeper: the changed area only
- When:
- After a fix or small change, before wider testing
- Duration:
- Minutes to an hour
- Scripted or not:
- Often partly unscripted, guided by the change
- Outcome of failure:
- Change sent back, broader testing postponed
Regression
Did this change break something else?
- Scope:
- Broad and deep: everything the change could affect, up to the whole suite
- When:
- After any change, before release, and on a schedule
- Duration:
- Minutes (a selected subset) to hours (the full suite)
- Scripted or not:
- Scripted, and the main target of automation
- Outcome of failure:
- Defect raised; release blocked if the gate says so
The scope shapes are schematic: width is how much of the product is covered, height how deeply. Sanity in its most common meaning; some teams use the word as another name for smoke.
Smoke testing: is the build testable?
A smoke test is a small set of checks that cover the most important functions of the system end to end, run on every new build or deployment, to decide whether the build is stable enough for further testing. The name comes from hardware: switch the device on and see whether it smokes. In software, a smoke failure means something fundamental is wrong — the application does not start, a service cannot reach its database, login is broken, a configuration value is missing — and every later test would fail for the same reason.
A good smoke suite is short, stable and critical-path only. It touches each major component once, does not try to find subtle defects, and never depends on data that other tests change. Because it runs so often, a flaky smoke test is especially harmful; fix or remove it immediately. The same suite, pointed at production after a release, is the core of post-deployment verification.
Smoke suite: web shop, build under test on staging (example)
[ ] GET /health returns 200 for web, orders, pricing, payments services
[ ] Home page loads; no console errors; main navigation renders
[ ] Log in with the smoke test account (smoke.user@example.com)
[ ] Search returns results for a known product
[ ] Add a known product to the cart; cart total is shown
[ ] POST /api/v1/orders with a fixed payload returns 201
[ ] Payment sandbox call returns an authorised result
[ ] Invoice endpoint returns 200 with Content-Type application/pdf
Rule: any failure -> build rejected, deeper testing does not start.
Budget: under 5 minutes, fully automated, runs on every deployment.Sanity testing: a term used in more than one way
Sanity testing is the least consistently defined of the three, and it is better to say so than to pretend otherwise. In the most common usage it means a quick, focused check of a specific change — a bug fix, a small feature, a configuration change — on a new build, to confirm that the change behaves reasonably before the team commits time to broader testing. In that sense it is narrow and deep where smoke testing is broad and shallow, and it often includes the confirmation test of the fixed defect.
Other teams and some older glossaries use “sanity test” as a synonym for smoke test. Still others use it informally for any quick plausibility check, such as checking that a report total is in a believable range. None of these is wrong in its own context, but mixing them inside one team causes real problems: a manager hears “sanity passed” and assumes more was checked than was. The practical fix is to define the term in your test strategy or test plan — what it covers, who runs it, how long it takes — and use it consistently. If the word causes confusion, use a descriptive name instead, such as “fix verification”.
- Common meaning: focused check of the changed area on a new build, before broader testing.
- Alternative meaning: another name for smoke testing.
- Informal meaning: a quick plausibility check of an output.
- Recommendation: write down your team’s definition and stick to it.
Regression testing: did the change break something else?
Regression testing re-runs existing tests after a change to detect whether defects have been introduced or exposed in parts of the software that were not meant to change. The ISTQB Foundation syllabus distinguishes it from confirmation testing: confirmation testing checks that a reported defect is actually fixed; regression testing checks that the fix, or any other change, has caused no adverse consequences — in the changed component, in other components or in connected systems. Both are needed after every fix.
Changes that trigger regression testing include new features, bug fixes, refactoring, dependency and framework upgrades, configuration changes, infrastructure and database migrations. Because regression tests are repeated many times and their expected results are known, they are the natural candidates for automation, and most regression suites live in the CI pipeline. What to automate at which level — unit, API or UI — follows the test pyramid and is covered in the manual versus automated testing guide.
- Confirmation test (re-test) — the steps of the original defect now pass.
- Regression test — tests around and beyond the change still pass.
- A regression suite grows with the product; without maintenance it becomes slow, duplicated and flaky.
Regression selection: choosing what to run
Running every test on every change is the safest option and, for a small suite, the right one. As the suite grows past the time budget of the pipeline stage, teams have to select. Selection is a risk decision, so it should be explicit and reviewable rather than a tester’s memory of “what usually breaks”.
- Retest all — run the full suite. Highest confidence, highest cost; the usual choice for nightly runs and release candidates.
- Impact-based selection — start from what changed (files, modules, APIs, database tables, configuration) and include the tests that exercise those parts and their direct dependents. Test impact analysis tools automate this by mapping tests to code, usually from coverage data.
- Risk-based selection — add tests for areas with high business impact or a history of defects even if the change seems unrelated: payments, authentication, data export.
- Prioritisation — order the selected tests so the most valuable run first: recently failed tests, tests on changed code, critical journeys. A failure then arrives early.
- Always-include core — smoke checks and a handful of critical end-to-end journeys run on every change regardless of selection.
- Safety net — whatever the per-change selection skips, the scheduled full run covers, so a selection mistake is caught within a day, not at release.
Try it: Regression Scenario Generator Try it: AI Regression Test Generator Try it: Risk-Based Test Generator
Where each one runs in a CI/CD pipeline
In a pipeline the three are not separate phases performed by different people; they are suites attached to different triggers. The following layout is common and works for most web products.
- Pull request — commit-stage checks plus a selected regression subset (impact-based and risk-based) at unit and API level; the critical E2E journeys.
- Merge to main — the full unit and API regression suites; E2E suite in parallel shards.
- Every deployment to a test environment — the smoke suite first; nothing else starts until it passes.
- After a fix is deployed — sanity (fix verification) by the tester who reported the defect, including the confirmation test.
- Nightly — the full regression suite across all levels, plus longer non-functional runs.
- Release candidate — full regression green, then smoke against production immediately after the release.
Worked example: a VAT rounding fix
For example, a customer reports that invoices with a discount show VAT one halala off. Developers fix the rounding in the pricing service; the fix touches VAT calculation, which is also used by refunds, credit notes and order history. The plan below shows how the three kinds of testing divide the work on build 5.8.1.
Change: fix VAT rounding on invoices that have a discount (pricing service, invoice PDF)
Build: 5.8.1 on staging
1. SMOKE (automated, about 3 min): can this build be tested at all?
[x] /health 200 for web, pricing and invoice services
[x] log in with the test customer, add an item, reach checkout
[x] create order via API -> 201
[x] invoice PDF endpoint -> 200, application/pdf
2. SANITY (focused, about 20 min): is the fix itself right?
[x] 100.00 SAR, 10% discount -> subtotal 90.00, VAT 15% 13.50, total 103.50
[x] 3 x 33.33 SAR, 5% discount -> totals follow the rounding rule in the spec
[x] invoice PDF shows the same VAT as the order API response
[x] steps of the original bug report no longer reproduce (confirmation test)
3. REGRESSION (selected): did anything else break?
must pricing unit + API suites (all), invoice API suite, checkout E2E journeys
should refunds and credit notes, promo codes, order-history totals (all use VAT)
skip profile, notifications, search (no dependency on pricing)
later full regression suite in the nightly run, as every nightCommon mistakes
- Treating a green smoke run as release sign-off. Smoke says the build is testable, not that it is correct.
- Letting the smoke suite grow until it takes half an hour; it stops being run on every deployment.
- Reporting “sanity passed” without saying what was checked.
- Closing a defect after the confirmation test alone, without any regression around the fix.
- Selecting regression tests from memory instead of from the change and a written risk list.
- Never pruning the regression suite: duplicated tests, tests for removed features and permanently skipped tests cost time on every run.
- Skipping the scheduled full run because per-change selection “has been fine so far”.
Frequently asked questions
What is the main difference between smoke and sanity testing?
In the most common usage, smoke testing is broad and shallow — it checks that the critical functions of the whole build work so testing can begin — while sanity testing is narrow and deeper, checking that a specific change or fix behaves correctly. Some teams use the two terms as synonyms, so agree on a definition in your test strategy.
Is sanity testing a subset of regression testing?
It is often described that way, because it re-checks an area after a change. In practice it is usually a quick, focused check of the change itself, done before regression testing starts, and it includes the confirmation test of a fixed defect.
Should smoke tests be automated?
Yes. They run on every build and deployment, they must be fast and consistent, and their result decides whether further testing starts. A manual smoke checklist is acceptable only where automation is not yet possible, and should be an early automation target.
How many tests should a smoke suite have?
As few as needed to touch each critical path and each major component once — often between ten and a few dozen checks — and short enough to finish in a few minutes. If it takes much longer, some of it belongs in the regression suite.
Do I need to run the full regression suite for every change?
Not necessarily. For small suites, yes; for larger ones, run a selected subset on each change based on impact and risk, and run the full suite on a schedule and before release so anything the selection missed is caught quickly.
What is the difference between regression testing and re-testing?
Re-testing, or confirmation testing, checks that a specific defect has been fixed by repeating its steps. Regression testing checks that the fix, or any other change, has not broken other functionality.
References
- ISTQB Certified Tester Foundation Level (CTFL) v4.0The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
- ISTQB GlossaryThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
- The Practical Test Pyramid (Ham Vocke)A widely cited explanation of how to balance unit, integration, contract and UI tests in an automated test portfolio.
- Continuous Integration (Martin Fowler)The widely referenced description of continuous integration: integrate often, keep the build self-testing and fast, and fix a broken build immediately.
Practice exercise
Plan smoke, sanity and regression for a transfer-limit fix
You test the web app of a retail bank. Build 3.4.2 has just been deployed to staging with the fix for defect BUG-1182. The material below gives the defect, the developer’s note, the team’s risk list and eleven candidate checks. Your team uses “sanity testing” in the guide’s most common meaning: a focused verification of the fix on the new build, including its confirmation test.
Turn the candidate checks into a test plan for this build, using the groups of the guide’s worked example: smoke, sanity, regression (must and should), skip and later. Give a one-line reason for every regression, skip and later decision. Two sanity checks are missing from the list: add them. Then answer the two questions at the end of the material.
Retail banking web app - build 3.4.2 deployed to staging
Defect BUG-1182 (fixed in this build)
A local transfer to a saved beneficiary is rejected with "Daily limit exceeded"
when the amount equals the remaining daily limit exactly.
Example: daily limit 10,000.00 SAR, 7,500.00 SAR already sent today,
transfer of 2,500.00 SAR -> rejected. Expected: accepted.
Developer note
The limit check in the limits service now uses amount <= remaining
(it was amount < remaining). Local transfers, bill payments and the
scheduled-transfer job call this check. International transfers use
their own limit in the FX service. Nothing else changed in this build.
Team risk list: international transfers (high business impact,
history of production defects).
Candidate checks
C1 GET /health returns 200 for web, accounts, transfers and limits services
C2 Log in with the test customer; the accounts overview shows balances
C3 Local transfer of 10.00 SAR to a saved beneficiary through the API -> 201
C4 Repeat the steps of BUG-1182 -> the 2,500.00 SAR transfer is accepted
C5 Bill payments: within the limit accepted, above the remaining limit rejected
C6 Scheduled-transfer job: within the limit executes, above it is skipped
C7 All automated unit and API tests of the limits service
C8 International transfer within its own limit -> accepted
C9 Change the profile picture
C10 Download last month’s account statement as PDF
C11 Full regression suite, all levels
Questions
Q1 The first smoke run fails at C1: the limits service returns 503.
What happens to the rest of the plan?
Q2 Your test lead asks: "Did sanity pass?" Write a one-line answer
that says what was checked.- Read the defect and the developer note. Write down what changed and which features call the changed code.
- Pick the smoke checks: the ones that only show whether build 3.4.2 can be tested at all.
- Pick the sanity checks for the fix, mark the confirmation test, and add the two missing checks on either side of the limit.
- Select the regression to run now and write the reason for each pick (impact or risk); put the rest under skip or later, with a reason.
- Write the execution order, then answer Q1 and Q2.
Show the model answer
Smoke (automated, first, a few minutes): C1, C2 and C3. Together they show whether build 3.4.2 can be tested at all: the services are up, a customer can log in, and the transfer path works end to end. C3 runs through the changed check, but it is not sanity: 10.00 SAR is far from the limit, so it says nothing about the fix. Rule: if any smoke check fails, the build is rejected and nothing else starts.
Sanity, in the team’s meaning of fix verification (narrow and deeper, right after smoke): C4 is the confirmation test, because it repeats the steps of BUG-1182. The two missing checks sit on either side of the limit, with the same starting state (limit 10,000.00 SAR, 7,500.00 SAR sent today). S1: a transfer of 2,500.01 SAR is rejected with “Daily limit exceeded”, which shows the fix does not let amounts above the remaining limit through. S2: a transfer of 2,499.99 SAR is accepted, which shows amounts below the limit still work. Together, C4, S1 and S2 test the remaining limit of 2,500.00 SAR itself and one step of 0.01 SAR on either side of it: the “exactly the limit and one beyond” check from the lesson on edge and corner cases, applied on both sides. The test design module calls this 3-value boundary value analysis. Each of them needs exactly 7,500.00 SAR sent today: C4 uses up the remaining limit (0.00 after it), and a customer used for C3 has already sent 10.00 SAR, so give each check its own test customer or reset the data before it. A useful extra: after C4, a further transfer of 0.01 SAR is rejected.
Regression, must (impact-based: they exercise the changed check or call it directly): C7 first, because the unit and API tests of the limits service are fast and cover the changed code; then C5 (bill payments) and C6 (the scheduled-transfer job), the other two callers named in the developer note. Regression, should (risk-based): C8. International transfers do not call the changed check, but they are on the team risk list, so one journey is a cheap safeguard. Listing C8 under must is also acceptable if your team always runs its risk list; what matters is that the reason is written down.
Skip: C9 and C10. The profile picture and the statement PDF do not call the limit check; the statement only reads transactions that already exist. Later: C11, the full regression suite, runs in the nightly run as every night and must be green for the release candidate. It is the safety net for anything this selection missed, not the gate for this change.
Order: C1 to C3, then C4, S1 and S2, then C7, C5, C6 and C8. C9 and C10 are not run for this change; the nightly full run (C11) still covers them. If S1 fails (2,500.01 SAR is accepted), the fix has introduced a more serious defect: customers could exceed their daily limit. Report it with the exact amounts, send the fix back and postpone the regression. Do not close BUG-1182 after the confirmation test alone; close it once the selected regression has passed.
Q1: when C1 fails, build 3.4.2 is rejected for testing. Do not start sanity or regression: the fix and every caller of the changed check depend on the limits service, so those checks would fail for the same reason and say nothing about the fix, and a failed confirmation test could wrongly reopen BUG-1182. Tell the developers at once and record the problem as your team’s process requires, with the build number, the endpoint, the 503 status and the time. When a corrected deployment arrives, start again from smoke.
Q2, for example: “Fix verification of BUG-1182 on build 3.4.2 passed: the exact-limit transfer of 2,500.00 SAR is accepted (confirmation test), 2,500.01 is rejected and 2,499.99 is accepted. Selected regression (C5 to C8) is in progress.” The bare words “sanity passed” would hide what was checked and what was not.
Open question for the product owner: bill payments and scheduled transfers call the same check, so a payment of exactly the remaining limit is now accepted there too. Is that intended? Record the answer in the defect or in the test report.
Your progress on this topic
Not started
Mark the topic complete when you have studied it and done the exercise.
Sign in to track your progress
Mark topics complete, continue where you left off and unlock specialisation exams.
Part of the QA Roadmap
Before this topic
Next topics
Tools mentioned in this guide
Generate regression scenarios from a change description and a list of affected and related modules.
Generate a regression test suite from a change description, diff summary or release notes with Mutqan AI.
Score features by likelihood and impact, then generate a prioritised risk-based test plan.
Check several API health endpoints at once and get status, latency and a pass/fail summary.
Generate a structured test-case matrix (positive, negative, auth, boundary) from an OpenAPI / Swagger specification.
Find duplicate and overlapping test cases in a suite and get merge recommendations from Mutqan AI.