What Is Test Automation?

Why it matters

Every release adds regression checks, and re-running them all by hand takes longer each time, so teams either slow down or stop re-checking, and regressions reach users. Automation keeps yesterday's behaviour confirmed while the team builds today's. Done badly, it is worse than none: a suite that is always red gets ignored, and a suite that is always green because it checks nothing gives confidence the product has not earned.

Even if you never build a framework, automation changes your daily work: you read run results, decide whether a red test is a product defect, a test defect or an environment problem, and review the scripts your colleagues write.

What an automated test is made of

An automated test has the parts of a good manual test case, written as code. It arranges the preconditions, acts by performing the steps, asserts by comparing actual with expected results, and cleans up so the next test starts from a known state. The assertion is the heart of it: without a precise expected result, a script proves only that nothing crashed. Some practitioners call an automated test a "check" to stress that it verifies only what someone decided in advance.

Below, the sign-in case as a browser test written with Playwright; Selenium WebDriver and Cypress tests have the same shape. You do not need every keyword yet: read the comments, the steps and the assertion. A line such as page.getByLabel('Email') is a locator, which tells the test how to find an element on the page, here by its visible label. The script is only one part of what the ISTQB Advanced Level Test Automation Engineering syllabus calls a test automation solution: the scripts plus everything needed to run them and report the results. Every part can break:

• Test scripts: one behaviour per test, named after the rule it checks.
• A framework and runner that discovers the tests, runs them and records each result.
• The interface the test drives: a function (unit level), an HTTP API or a user interface. Lower levels are faster and break less often, as the lesson on the test automation pyramid shows.
• Test data and an environment: the accounts, products and settings the test needs, and the system it runs against.
• A trigger and a report: usually a CI pipeline that runs the suite on every change, and a result that someone reads.

import { test, expect } from '@playwright/test';
import { testUser } from './test-data';

test('registered user can sign in', async ({ page }) => {
  // Arrange: start on the sign-in page with a known test account
  await page.goto('/login'); // resolved against baseURL in playwright.config.ts

  // Act: the steps of the manual test case
  await page.getByLabel('Email').fill(testUser.email);
  await page.getByLabel('Password').fill(testUser.password);
  await page.getByRole('button', { name: 'Sign in' }).click();

  // Assert: the expected result, decided before the test ran
  await expect(page.getByRole('heading', { name: 'My account' })).toBeVisible();

  // Clean up: nothing to undo, signing in changes no data
});

Try it: REST API Tester Try it: REST Request Generator

What automation gives a team

On a working team the benefits look like this. Each one holds only if the expected results are right, because a script repeats a wrong expected result as faithfully as a right one.

• Fast regression feedback: the suite runs on every change, so a regression is reported while the developer still remembers the code.
• Exact, repeatable comparison: every field of a response and every total to the last halala, on every run, where a person repeating a case for the twentieth time starts to skim.
• Scale: the same steps over hundreds of data rows, in several browsers, every night.
• A quality gate: an automated check in the pipeline can stop a broken build on every change, before it reaches staging, without waiting for a person.
• Living documentation: a test named "orders of 300.00 SAR or more ship free" states a rule and shows on every run that it still holds.
• Tester time: hours once spent re-executing scripted cases go to exploratory testing and to designing tests for new features.

Try it: Regression Scenario Generator Try it: Test Case Generator

What goes wrong once the scripts exist

The risks in the lesson on tool support for testing concern the decision to automate: unrealistic expectations, underestimated effort, the wrong tool. Once a suite exists, a second group appears, and most of it comes from the scripts themselves:

• False confidence: assertions too weak to fail. A test that checks only that the cart page loaded stays green while the total on it is wrong.
• Flaky tests: tests that pass and fail with no change to the product or to the test, usually because of timing, shared test data or an unstable environment. After a few false alarms the team stops reading red results, and real regressions slip through with them.
• Brittle scripts: tests tied to details that change often, such as a CSS class or the position of a button, break on every redesign although the behaviour is the same.
• Slow suites: when a run takes longer than people are willing to wait, they merge without it and the fast feedback is gone.
• Worn-out tests: a suite re-checks only what someone anticipated, so it finds regressions but rarely new defects, and it does not replace exploratory testing.
• A single owner: if only one person understands the framework, maintenance stops when that person moves on.

Automation code is software and needs maintenance

A test suite is a program that tests another program. It has requirements (the test cases it implements), a design, dependencies and defects of its own, such as a wrong expected result, a missing wait or a test that passes only after another test has run. It lives as long as the product and changes whenever the product does.

Maintenance comes from four directions: the product changes (a renamed button, a new mandatory field, a changed API response), the tools change (browser, framework and runtime upgrades), the data and environments change, and the tests' own defects surface. Teams whose suites stay useful treat test code like product code:

• Version control next to the product code; a behaviour change and its test change go in together.
• Code review for tests too: names, assertions, duplication, fixed waits.
• Each step written once, in a shared helper (later in this module, a page object), so one UI change means one edit.
• Data, URLs and secrets kept in configuration; never a real password in a script.
• Every failure triaged and recorded as a product defect, a test defect or an environment problem, never re-run until green.
• Maintenance time planned in every sprint, and tests that no longer protect anything deleted.

Worked example: one redesign, 37 red tests

A team has 120 browser tests. Designers split the sign-in page into two steps, the email and a "Continue" button first, then the password, and rename the button "Sign in" to "Log in". The nightly run shows 37 failures. The tester triages before anyone re-runs anything: the failures are exactly the 37 tests that sign in, all stop at the sign-in step, and each of those test files has its own copy of the sign-in lines. The product works; the tests are out of date.

Patching 37 files would take most of a day and would recur at the next redesign, so the team moves the steps into one helper, changed and reviewed once. After the fix, 36 tests pass. One still fails further on: "Remember me" no longer keeps the user signed in after a browser restart. The tester reproduces it by hand and reports a product defect, which is exactly what the suite is for.

// Before: these three lines were copied into 37 test files
//   await page.getByLabel('Email').fill(testUser.email);
//   await page.getByLabel('Password').fill(testUser.password);
//   await page.getByRole('button', { name: 'Sign in' }).click();

// After: one helper (sign-in.ts) that every test calls; the next redesign means one edit
import type { Page } from '@playwright/test';
import type { TestUser } from './test-data';

export async function signIn(page: Page, user: TestUser) {
  // Step 1: email, then "Continue". Step 2: password, then the renamed button.
  await page.getByLabel('Email').fill(user.email);
  await page.getByRole('button', { name: 'Continue' }).click();
  await page.getByLabel('Password').fill(user.password);
  await page.getByRole('button', { name: 'Log in' }).click();
}

// In each test: await signIn(page, testUser);

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• Certified Tester Advanced Level Test Automation Engineering (CTAL-TAE) v2.0ISTQB · 2024ISTQB's advanced syllabus on designing, implementing and maintaining test automation solutions and their architecture.
• Best Practices — PlaywrightMicrosoft (Playwright)Test user-visible behaviour, keep tests isolated, prefer resilient locators and web-first assertions, and debug with traces.