The Command Line and Test Runners

Why it matters

In the previous lesson you read a test file; nothing happens until something runs it. A tester who cannot run the suite from the terminal depends on others to learn whether the product is broken, cannot reproduce a red pipeline on their own machine and cannot narrow a two-hour run down to the one test that matters.

The terminal is also the honest version of the truth: a green badge on a dashboard means only that a command returned exit code 0. Once you see the chain of command, runner, tests, report and exit code, you can tell a product defect from a broken installation, a wrong folder or a filter that matched nothing, and give developers the exact command that reproduces a failure.

The terminal: where you are and what you run

Every command runs inside a working directory, the folder the terminal is currently in. pwd prints it, ls lists its contents (dir in the old Windows command prompt), cd tests moves into a sub-folder and cd .. moves one level up. A path is relative to that folder (tests/login.spec.ts) or absolute from the root of the disk (/home/sara/shop-tests, C:\Users\sara\shop-tests). Many "command not found" and "no tests found" surprises are just a terminal sitting in the wrong folder.

A command is a program name followed by arguments and options: in npx playwright test tests/login.spec.ts --headed, npx is the program, the file is an argument and --headed an option. The shell, the program that reads what you type, finds programs through the PATH environment variable, a list of folders it searches for a program with that name. Environment variables are named values that every program receives from the shell that started it; a value you set lasts only for that terminal window. Test projects keep everything that changes between machines in them, such as the URL under test or a password, so the test code stays the same.

• bash, zsh and Git Bash: export BASE_URL=https://staging.shop.example, or for one command only: BASE_URL=https://staging.shop.example npx playwright test.
• PowerShell: $env:BASE_URL = 'https://staging.shop.example'; command prompt: set BASE_URL=https://staging.shop.example.
• In Node.js the value arrives as process.env.BASE_URL; when it is missing it is undefined, so the project should check it at the start and stop with a clear message instead of failing later on a broken address such as undefined/login.
• Ctrl+C stops a running command; the Tab key completes file names.

$ pwd                        # where am I?  (PowerShell: pwd works too; cmd.exe: cd)
/home/sara/shop-tests
$ ls                         # what is here?  (cmd.exe: dir)
node_modules  package.json  package-lock.json  playwright.config.ts  tests
$ cd tests && ls
checkout.spec.ts  login.spec.ts
$ cd ..
$ export BASE_URL=https://staging.shop.example   # PowerShell: $env:BASE_URL = '...'
$ npx playwright test

Node.js, npm, npx and the two package files

JavaScript runners such as Playwright Test, Jest and Vitest run on Node.js. Install the current LTS (long-term support) release from nodejs.org and check with node --version and npm --version. npm, the package manager that ships with Node, downloads the libraries a project depends on into node_modules. npx runs a command-line program that one of those packages provides, here playwright, without a global installation; if the package is not in the project, npx downloads it into a cache first, asking for confirmation when the terminal is interactive.

Two files describe the project. package.json is written by people: scripts are short names for long commands, so npm test runs whatever the test script says, and devDependencies lists packages with version ranges, where ^1.47.0 means any 1.x release from 1.47.0 upwards. package-lock.json is written by npm: the exact version of every package and sub-package that was installed. npm install treats the lock file as a starting point: when package.json has changed or there is no lock file, it resolves the ranges again, can pick newer versions and rewrites the lock file. npm ci installs exactly what the lock file says, removes an existing node_modules first and stops with an error when the two files disagree. That is why CI, and anyone setting up the project fresh, should run npm ci. It does not download the browsers Playwright drives: npx playwright install does that after Playwright is installed or upgraded, and on a fresh Linux CI machine npx playwright install --with-deps also adds the system libraries they need.

{
  "name": "shop-tests",
  "scripts": {
    "test": "playwright test",
    "test:smoke": "playwright test --grep @smoke"
  },
  "devDependencies": {
    "@playwright/test": "^1.47.0"
  }
}

Try it: JSON Validator

Worked example: one login suite, several ways to run it

The project shop-tests has tests/login.spec.ts with three tests, two of them tagged @smoke, and tests/checkout.spec.ts with four more. These are the Playwright Test commands; other runners have the same shapes with different spellings.

-g (--grep) takes a regular expression, a text pattern in which a plain word is the simplest case, and matches it against the full title, describe names included, so -g Login runs the whole group; a tag is a word such as @smoke filtered the same way. In PowerShell write the tag in quotes, --grep '@smoke': a bare @smoke is read as a PowerShell variable and the filter is lost. Headless, the default, runs the browser without a window. --headed shows the window, --debug opens the Playwright Inspector with the test paused so you can step through it action by action, and --ui lets you watch and re-run. All three are for your machine; CI has no screen.

Goal                           Command
All tests                      npx playwright test
One file                       npx playwright test tests/login.spec.ts
By title (regular expression)  npx playwright test -g "wrong password"
By tag                         npx playwright test --grep @smoke
Headed: watch the browser      npx playwright test --headed
Debug: step through actions    npx playwright test tests/login.spec.ts --debug
One browser project only       npx playwright test --project=chromium
Through package.json scripts   npm test          npm run test:smoke

Try it: Regex Tester

How the runner finds tests and what it prints

The runner does not run "the project"; it discovers files. Playwright Test looks in the configured testDir for files whose names end, by default, in suffixes such as .spec.ts, .spec.js, .test.ts or .test.js, loads them and collects every test() call, grouped by test.describe() blocks; Jest and Vitest use the same structure with describe and it or test, Mocha with describe and it. A file that does not match the pattern is silently ignored, a common reason a new test "did not run".

Read the output top to bottom: the header counts the tests found and the workers, the parallel processes that run them; each line is one test with its status, file, line number and title; failures are repeated at the end with the assertion message and a stack trace; the last lines count failed, flaky (failed, then passed on a retry), skipped and passed. The html reporter, when the configuration turns it on, writes to playwright-report/ (open it with npx playwright show-report), screenshots and traces of failed tests, when the configuration turns them on, go to test-results/, and the junit reporter writes XML, usually to a file named in the configuration, that CI servers turn into a per-test view.

// tests/login.spec.ts — what the runner collects
import { test, expect } from '@playwright/test';

test.describe('Login', () => {
  test('valid login reaches dashboard', { tag: '@smoke' }, async ({ page }) => {
    // arrange, act, assert
  });
  test('wrong password shows error', { tag: '@smoke' }, async ({ page }) => {
    // ...
  });
  test('locked account is refused', async ({ page }) => { /* ... */ });
});

// npx playwright test tests/login.spec.ts — the list reporter
Running 3 tests using 1 worker

  ✓  1 [chromium] › login.spec.ts:4:7 › Login › valid login reaches dashboard (2.1s)
  ✘  2 [chromium] › login.spec.ts:7:7 › Login › wrong password shows error (1.4s)
  ✓  3 [chromium] › login.spec.ts:10:7 › Login › locked account is refused (1.2s)

  1) [chromium] › login.spec.ts:7:7 › Login › wrong password shows error

    Error: expect(locator).toHaveText(expected) failed
    Expected string: "Wrong e-mail or password"
    Received string: "Invalid credentials"

  1 failed
    [chromium] › login.spec.ts:7:7 › Login › wrong password shows error
  2 passed (5.0s)

Try it: Text Diff

Exit codes: how CI knows the result

Every program ends with an exit code, a number the operating system hands back to whoever started it. By convention 0 means success and anything else means failure. A test runner exits with 0 only when no test failed; a failed test, an error in the configuration file, a filter that matched nothing or a browser that is not installed all give a non-zero code. Right after a command, echo $? prints it in bash, $LASTEXITCODE in PowerShell and %ERRORLEVEL% in the command prompt.

CI servers such as GitHub Actions, GitLab CI and Jenkins do not read colours: they run each step as a command and mark the step failed when its exit code is non-zero, which stops the job, turns it red and, where the team has made the check required, blocks the merge. That is the whole contract. It also means a runner that swallows failures and exits 0 gives a green job for a broken product, and a test that fails only now and then turns the job red even when the product works. The steps themselves live in a pipeline file in the repository, a YAML workflow in .github/workflows/ for GitHub Actions or .gitlab-ci.yml for GitLab CI (Jenkins uses a Jenkinsfile), where every run: or script: entry is a command like the ones in this lesson.

• Python: pip install -r requirements.txt installs the dependencies; pytest tests/test_login.py -k wrong_password runs one file filtered by a keyword in the test name and -m smoke selects a marker; pytest exits 0 when the selected tests ran and none failed, 1 when a test failed and 5 when no test was collected.
• Java: Maven reads pom.xml; mvn test runs the suite and mvn -Dtest=LoginTest test runs one class; a failing test fails the build with a non-zero exit code.

$ npx playwright test --grep @smoke
  1 failed
  2 passed (6.8s)
$ echo $?                    # PowerShell: $LASTEXITCODE    cmd.exe: echo %ERRORLEVEL%
1

Try it: YAML Formatter

Frequently asked questions

References

• Playwright DocumentationMicrosoft (Playwright)Official documentation for Playwright, an end-to-end test framework that drives Chromium, WebKit and Firefox.
• Continuous Integration (Martin Fowler)martinfowler.com · 2024The widely referenced description of continuous integration: integrate often, keep the build self-testing and fast, and fix a broken build immediately.
• GitHub Actions documentationGitHubHow workflows, jobs, runners and artifacts are defined in GitHub Actions — where automated tests run on each push or pull request.
• Certified Tester Advanced Level Test Automation Engineering (CTAL-TAE) v2.0ISTQB · 2024ISTQB's advanced syllabus on designing, implementing and maintaining test automation solutions and their architecture.