Flaky Tests

What a flaky test is, and why it matters

A test is flaky when repeated runs against the same commit and the same environment configuration give different results. Run it ten times and you might get eight passes and two failures. Because the outcome is non-deterministic, neither result on its own tells you much about the product.

The cost is out of proportion to the frequency. Each flaky failure costs someone an investigation or a rerun; after a few, the team starts treating every failure as noise. Martin Fowler’s article on non-determinism in tests makes the point directly: a suite that fails randomly loses its value as a signal. And a flaky test is not always a bad test — sometimes the product contains the race condition, and the test is the only thing that noticed.

• The same test fails on one run and passes on the rerun, with no code change in between.
• It fails only in CI, only in parallel, only at certain times of day, or only in one browser.
• The failure message changes between runs — a timeout in one, “element not found” in the next.

Common causes of flaky UI tests

• Timing and asynchrony — acting or asserting before the page is ready: clicking a button that is still disabled, reading text before the API response arrives, or a fixed sleep that is sometimes too short.
• Shared state and test order — tests that depend on data or browser state another test created or changed. They pass in one order and fail in another, or fail when run alone.
• Test data collisions — parallel tests using the same account, the same email or the same product stock; uniqueness constraints that fail when two runs pick the same “random” value.
• Environment — an overloaded CI runner, a shared staging environment redeployed mid-run, slow container start-up, or a different screen size, locale or time zone on the runner.
• Third-party and remote services — payment sandboxes, SMS gateways, map tiles, feature-flag services and CDNs that are slow, unavailable or rate-limited.
• Animations and transitions — clicking an element while it slides into place, or asserting on a toast message that disappears after a few seconds.
• Time and dates — tests that break around midnight, month ends and daylight-saving changes, or when the runner’s time zone differs from the application’s.
• Unordered data — asserting on the first row of a list the server returns in no guaranteed order.
• Resource leaks — browsers, sessions or database connections not closed, so later tests in a long run slow down or fail.
• Real product defects — race conditions, double submissions, caches that are not invalidated. These are bugs to report, not test noise to quarantine.

Symptoms and root causes: worked examples

The table pairs typical flaky symptoms, written the way they appear in a defect tracker, with the root cause found on investigation and the fix that removed it. Notice that none of the fixes is “add a longer sleep” or “turn on retries”.

For example (fictional):

Symptom (as a bug title)                                         Root cause                                  Fix
checkout.spec: "Place order" click ignored in ~1 of 10 runs      Button enabled only after price recalculated  Assert the enabled state; rely on auto-wait
orders.spec fails only when run after refunds.spec               refunds.spec cancels the shared order         Each test creates its own order via the API
signup.spec: "email already registered" in parallel CI runs      Two workers generate the same email           Add run/worker id + UUID to generated data
report.spec fails between 00:00 and 03:00 Riyadh time            Runner uses UTC, so "today" differs           Control the clock or compute dates in one zone
profile.spec: "Saved" toast not found on CI only                 Toast disappears before the slow assertion    Assert the saved state, not the transient toast
map.spec times out under parallel load                           Third-party tile service rate-limits the run  Stub map responses; keep one contract check

Diagnosing a flaky test

Treat a flaky test like any intermittent bug: reproduce it, collect evidence from the failing runs, and change one variable at a time until the failure rate moves.

• Reproduce by repetition — run the single test many times (Playwright has --repeat-each; with other runners use the runner’s repeat option or a shell loop). A failure rate turns “it sometimes fails” into a number you can compare before and after a fix.
• Vary one condition — alone vs in the full suite, serial vs parallel, a different order, local vs CI, throttled CPU or network, another browser. The condition that changes the failure rate points at the cause.
• Read the evidence from the failing run — the screenshot at failure, the video and the trace. Playwright’s trace viewer shows each action with DOM snapshots, network requests and console messages; Cypress can capture screenshots and videos of runs; with Selenium, capture a screenshot, browser logs and the page source in your runner’s failure hook.
• Correlate with the back end — log the request ids and timestamps the test produced and look up the same window in the server logs. Watch time zones: CI runners often use UTC while the application logs in local time.
• Check the history — test reports and CI dashboards show when the test started flaking; compare that date with commits, dependency upgrades and environment changes.

// playwright.config.ts — keep evidence from failing runs
export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    trace: "on-first-retry",
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
});

# reproduce locally: run one spec 50 times with 4 workers
npx playwright test tests/checkout.spec.ts --repeat-each=50 --workers=4

# open the trace from a failed CI run
npx playwright show-trace trace.zip

Fixing the common causes

• Timing — replace sleeps with condition-based waits: explicit waits in Selenium, auto-waiting actions and web-first assertions in Playwright, retried queries with .should() in Cypress. Wait for the application’s own signal — a response or a state change — not for time.
• Shared state — make every test create its own data and browser state, start from a fresh browser context, and never rely on another test having run first.
• Data collisions — generate unique values per test and per worker; retire fixed “test user 1” accounts from parallel runs.
• Environment — pin browser and dependency versions, give the suite a dedicated environment or freeze deployments during the run, and size CI runners for the number of workers.
• Third-party services — stub them in most UI tests (page.route in Playwright, cy.intercept in Cypress, or a mock server) and keep a small, separately reported check against the real sandbox.
• Animations — assert the final state rather than a transient one; where the application honours the reduced-motion preference, emulate it in tests to shorten or remove transitions.
• Time — control the clock (Playwright’s clock API, cy.clock in Cypress) or compute expected dates in the application’s time zone.
• Order — sort or filter explicitly before asserting; never assume the first row is the one your test created.

// Flaky: guesses how long the save takes, then reads the text once
await page.getByRole("button", { name: "Save" }).click();
await page.waitForTimeout(2000);
expect(await page.getByTestId("status").textContent()).toBe("Saved");

// Stable: waits for the app’s own signal and retries the assertion
const saved = page.waitForResponse(
  (r) => r.url().includes("/api/profile") && r.request().method() === "PUT");
await page.getByRole("button", { name: "Save" }).click();
expect((await saved).ok()).toBeTruthy();
await expect(page.getByTestId("status")).toHaveText("Saved");

Try it: Fake User Generator Try it: Test Data Factory Try it: Mock API Response Generator Try it: API Request Builder

Retries and quarantine: containment, not cure

Automatic retries — Playwright’s retries setting, Cypress’s retries option, and rerun features in runners such as TestNG and Maven Surefire — turn a pipeline green when a flaky test passes on a later attempt. That is acceptable as short-term containment and harmful as a policy: retries hide real intermittent defects, slow the pipeline, and let the flaky rate grow unseen. If you use them, make them visible; Playwright, for example, reports a test that failed and then passed on retry as “flaky” rather than “passed”.

Quarantine is the other containment tool: move a known-flaky test out of the blocking suite into a separate, non-blocking job, where it keeps running and reporting without stopping merges. Fowler describes quarantine as a way to keep the main suite trustworthy — with the warning that quarantined tests must be fixed quickly, or they stop protecting anything.

• Allow at most one retry in CI and none locally, and report “passed on retry” as flaky.
• Check first whether the failure is a product bug; never quarantine a real defect.
• Give every quarantined test an owner, a ticket and a due date, and review the list weekly.
• When a test passes its due date in quarantine, fix it, rewrite it at a lower level, or delete it. A permanently skipped test is dead code that looks like coverage.
• Cap the quarantine. If it keeps growing, stop adding to it and fix the shared cause — usually the environment or the test data strategy.

Flaky test policy (example)
- Detection:  a test that fails and passes on the same commit is flagged flaky by CI
- Retries:    at most 1 in CI, 0 locally; "passed on retry" is reported as flaky, not passed
- Triage:     within 1 working day: product bug, test bug or environment?
- Quarantine: non-blocking job, with ticket + owner + due date (max 10 working days)
- Exit:       fixed and 50 consecutive passes with --repeat-each, or rewritten / deleted
- Review:     weekly flaky list in the QA sync; flaky-rate trend per suite

Ownership and measurement

Flaky tests stay flaky when nobody owns them. Assign ownership by product area — the team that owns checkout owns the checkout tests — rather than to one “automation person”, and make flaky-test triage part of the normal definition of done instead of a side project.

A few simple measures show whether the situation is improving. Track them per suite and review the trend, not a single week.

• Flaky rate — the share of test runs that failed and then passed on retry, or failed without a code change.
• Top offenders — the handful of tests responsible for most flaky failures; fixing them usually gives the largest improvement.
• Time to fix — how long a test stays flagged or quarantined.
• Rerun cost — pipeline minutes and people-hours spent on retries and manual reruns.

Preventing flakiness when writing tests

• No fixed sleeps; every wait is a condition with a timeout.
• Retrying assertions on user-visible outcomes, not one-off reads.
• Resilient locators based on role, label, text or test ids — see the guide on locator strategies.
• Each test creates and owns its data, with unique values per run and per worker.
• State is set up through the API, not by clicking through unrelated screens.
• Third-party services are stubbed; the clock and time zone are controlled where dates matter.
• New tests are run many times — for example 20 to 50 repetitions — before they join the blocking suite.
• The suite runs in parallel and, from time to time, in random order, so hidden dependencies surface early.

Frequently asked questions

References

• Eradicating Non-Determinism in Tests (Martin Fowler)martinfowler.com · 2011Why flaky tests destroy a suite's value, their common causes (isolation, asynchrony, remote services, time, resource leaks) and quarantine as a response.
• Trace viewer — PlaywrightMicrosoft (Playwright)Recording and replaying a test run step by step — DOM snapshots, network, console — to diagnose failures, especially from CI.
• Auto-waiting — PlaywrightMicrosoft (Playwright)The actionability checks Playwright runs before each action — visible, stable, enabled, receiving events — which remove most hand-written waits.
• Waiting Strategies — SeleniumSelenium ProjectWhy browser tests race the page, and how implicit and explicit waits synchronise a WebDriver script with the application.
• Best Practices — CypressCypress.ioCypress guidance on selecting elements with dedicated attributes, avoiding fixed waits and keeping tests independent.
• Best Practices — PlaywrightMicrosoft (Playwright)Test user-visible behaviour, keep tests isolated, prefer resilient locators and web-first assertions, and debug with traces.