Running Tests in CI Pipelines

Why it matters

Most tests in a CI team are run by the pipeline, so the workflow decides what evidence exists. A check that runs the wrong set, or a browser suite that only ever runs in Chromium, still shows green; a check that takes 40 minutes is one people stop waiting for.

Testers usually decide what the suite covers, so they need to change the workflow themselves: add a browser, move a slow suite to the nightly run, keep a failed job's report.

Anatomy of a pipeline file

YAML has three building blocks. A mapping is key: value; a list is lines starting with - , or the short form [chromium, firefox]; nesting is shown only by indentation with spaces. Tabs are not allowed in indentation, and one space too many or too few can move a key into another block. Text after # is a comment, and a value starting with * must be quoted, as in the cron '*/30 * * * *'.

The on key lists the triggers: pull_request when a pull request is opened, reopened or gets new commits, push filtered to main, and schedule with a cron expression. Its five fields are minute, hour, day of month, month and day of week, so '0 1 * * *' means 01:00 every day; schedules run on the default branch, in UTC unless the entry sets a timezone. Each job under jobs runs on a runner, the machine named by runs-on; a GitHub-hosted runner is a fresh virtual machine per job, so nothing survives between jobs except what you cache or upload. A job's steps run in order in one workspace: uses runs a published action, run a shell command, and a non-zero exit code fails the job.

# .github/workflows/e2e.yml
name: e2e

Part 1:1on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: '0 1 * * *'          # every night, 01:00 UTC

jobs:
  smoke:
    Part 2:2if: github.event_name != 'schedule'
    Part 3:3runs-on: ubuntu-latest
    timeout-minutes: 15
    Part 4:4steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          Part 5:5cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --project=chromium Part 6:6--grep @smoke
      - uses: actions/upload-artifact@v7
        Part 7:7if: always()
        with:
          name: smoke-results
          path: results/
          retention-days: 14

  nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    timeout-minutes: 45
    Part 8:8strategy:
      fail-fast: false
      matrix:
        browser: [chromium, firefox, webkit]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps ${{ matrix.browser }}
      - run: npx playwright test --project=${{ matrix.browser }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          Part 9:9name: nightly-${{ matrix.browser }}
          path: results/
          retention-days: 14

# .github/workflows/e2e.yml
name: e2e

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: '0 1 * * *'          # every night, 01:00 UTC

jobs:
  smoke:
    if: github.event_name != 'schedule'
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --project=chromium --grep @smoke
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: smoke-results
          path: results/
          retention-days: 14

  nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    timeout-minutes: 45
    strategy:
      fail-fast: false
      matrix:
        browser: [chromium, firefox, webkit]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps ${{ matrix.browser }}
      - run: npx playwright test --project=${{ matrix.browser }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: nightly-${{ matrix.browser }}
          path: results/
          retention-days: 14

Try it: YAML Formatter Try it: YAML to JSON Try it: Cron Expression Explainer

Dependency caching and a browser matrix

Because each job starts clean, npm ci would download every package on every run. A dependency cache stores those downloads under a key computed from the lock file and restores them next time; when package-lock.json changes, so does the key. cache: npm on actions/setup-node caches npm's download cache, not node_modules, so npm ci still installs exactly what the lock file says, only faster, and must give the same result on a cache miss. Playwright's CI documentation advises against caching browser binaries: restoring them takes about as long as downloading them.

A matrix turns one job definition into one job per combination of values. browser: [chromium, firefox, webkit] creates three parallel jobs, each passing its value to --project, which must match a project name in the Playwright configuration. Adding shard: [1, 2] makes six jobs, so add a dimension only for a real risk, such as rendering differences between engines. fail-fast: false lets every browser finish, showing whether a failure belongs to one engine or all.

Worked example: pull request or nightly

A shop team has 1,240 unit, 310 API and 180 browser tests, 25 tagged @smoke for sign-in, search, cart and payment. The browser suite takes about 35 minutes per browser; pull requests have a 15-minute budget. The team sorts each suite by speed, stability, the risk it covers and who acts on a failure. The workflow above implements the browser rows.

• Pull requests get fast, stable checks on the key journeys; the author waits for them, so they gate the merge.
• Pushes to main repeat that set: two pull requests that passed alone can break together.
• The nightly run carries the slow, broad suites; a named owner reads it each morning and checks the commits since the last green night.
• Quarantined flaky tests leave the pull request set and run only at night, reporting without blocking anyone. In the nightly job, the main test step adds --grep-invert @quarantine, which skips tests with that tag; a second step runs --grep @quarantine with continue-on-error: true, so its failure does not fail the job, and if: always(), so it still runs after the main step has failed.

Suite                 Tests    Time*   Pull request  Push to main  Nightly 01:00 UTC
Lint + unit           1,240    3 min   gate          run           run
API tests             310      6 min   gate          run           run
E2E @smoke, Chromium  25       4 min   gate          run           -
E2E full, 3 browsers  3 x 180  35 min  -             -             run, 3 parallel jobs

gate: must pass before the merge
*  wall-clock time per job; illustrative figures

Try it: Cron Expression Generator

Test reports, artifacts and parallel jobs

Configure the test runner to write results for machines and people, then make the job keep them. The excerpt below writes JUnit XML for the CI server, an HTML report for people, and traces and screenshots of failed tests. GitLab reads the XML through artifacts:reports:junit, Jenkins through the JUnit plugin's junit step; elsewhere, check the platform's documentation.

Artifacts are files a job uploads so they outlive its runner. Upload them with if: always(): by default a step after a failed step is skipped, and failed runs are when you need them. GitHub Actions needs unique artifact names within a run, hence nightly-${{ matrix.browser }}; retention-days sets how long they stay. Keep personal data and secrets out of them.

Jobs without needs run in parallel, as do matrix jobs, as far as runners allow; max-parallel caps a matrix. Inside a job, Playwright's CI guidance prefers one worker for stable runs, scaling out with shards instead. Merge the per-job reports, for example with Playwright's blob reporter and npx playwright merge-reports, so the gate sees one result.

// playwright.config.ts (excerpt)
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  workers: process.env.CI ? 1 : undefined,
  outputDir: 'results/artifacts',                       // traces, screenshots
  reporter: [
    ['junit', { outputFile: 'results/junit.xml' }],                // for the CI server
    ['html', { outputFolder: 'results/html', open: 'never' }],     // for people
  ],
  use: { trace: 'retain-on-failure', screenshot: 'only-on-failure' },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit', use: { ...devices['Desktop Safari'] } },
  ],
});

Try it: XML Formatter

Frequently asked questions

References

• GitHub Actions documentationGitHubHow workflows, jobs, runners and artifacts are defined in GitHub Actions — where automated tests run on each push or pull request.
• Playwright DocumentationMicrosoft (Playwright)Official documentation for Playwright, an end-to-end test framework that drives Chromium, WebKit and Firefox.
• YAML Ain't Markup Language (YAML™) version 1.2.2YAML.org · 2021The YAML 1.2 data model and syntax, and its relationship to JSON (every JSON document is valid YAML 1.2).
• Unit test reports — GitLab CI/CDGitLabPublishing JUnit XML results as pipeline artifacts so failed tests appear in the merge request and can be compared between runs.