Debugging Pipeline Failures

Why it matters

A red pipeline blocks every merge and release behind it, and a wrong classification costs more than slow debugging. Calling a product defect "flaky" lets it ship; calling an environment problem a product bug sends developers hunting for a defect that is not in the code. Re-running until green does both: it hides the evidence and teaches the team that red means nothing.

For a tester, this is where trust is earned. A report that names the failing test, the first error, the artifacts, a reproduction command and the first bad commit gets fixed quickly; "the pipeline is red, can someone look?" does not.

Start with the log and the artifacts

As in the lesson on automation reporting and debugging, start from the first error and its evidence; in a pipeline, that means first finding the first failed job and the first failed step inside it. Search the log for the test name or the error marker (##[error] in a raw GitHub Actions log) instead of scrolling, and remember that runner timestamps are usually in UTC.

Then download the job’s artifacts before their retention period ends, such as the JUnit XML report, the Playwright trace and the application logs, and read them as that lesson showed. Compare them with the job’s last green run: a new runner image, a changed dependency or fewer tests executed is a clue before you read any code.

# GitHub CLI: logs of the failed steps only, then the artifacts of one job
gh run view 9876543 --log-failed
gh run download 9876543 -n e2e-results-3

# open the trace recorded for the failing test
npx playwright show-trace test-results/invoice-order-date/trace.zip

Try it: XML Formatter Try it: Text Diff

Reproduce the failure locally

A failure you can reproduce on your own machine can be studied with a debugger, extra logging and as many runs as you need. Reproduction is a matter of parity: match the CI job in everything that can change the result, then change one thing at a time. If the test still passes locally, the remaining difference is your best clue.

• Commit: the exact SHA from the job log. A pull-request run on GitHub Actions tests a temporary merge into the target branch, not your branch head; that commit is on no branch, so fetch it with git fetch origin pull/<number>/merge and check that the SHA matches the log.
• Image and versions: the job’s container image if it has one, dependencies from the lockfile (npm ci, not npm install) and the browser builds your Playwright version pins.
• Environment variables: the non-secret values the job sets, including CI=true, which many test configurations read; your own test credentials, never production secrets.
• Time zone, locale and seed: runners often use UTC and an English locale, and tests that generate random data should log their seed so the data can be replayed.
• Subset and order: the same shard and worker count, then the failing test alone and repeated, to tell "fails alone" from "fails only next to other tests".

git checkout 9f8e7d6                     # the SHA the job tested
npm ci && npx playwright install --with-deps chromium
export CI=true TEST_SEED=48151           # non-secret values copied from the job
npx playwright test --shard=3/4 --workers=2
npx playwright test invoice.spec.ts --repeat-each=10

Timeouts, fail-fast and re-runs

A timeout means something the test waited for never happened: an element, a response, a service. Raising the limit rarely fixes it; the call log or the trace shows what was awaited. Layer the limits: a test timeout (30 seconds by default in Playwright) fails one named test and the report is still written, while a job timeout (timeout-minutes on GitHub Actions, 360 minutes if unset) cancels the whole job and may leave no report. Keep test timeouts well below the job timeout.

Fail-fast stops at the first failure (in Playwright, -x, short for --max-failures=1); run-all collects every failure. Fail fast in cheap early stages such as linting and unit tests. In expensive stages and nightly runs, run everything: one failing test and forty failing tests tell different stories, and forty usually point to a shared cause such as the environment. A GitHub Actions matrix cancels its other jobs when one fails unless you set fail-fast: false.

Re-run as narrowly as you can: only the failed jobs (gh run rerun <run-id> --failed, where the run ID is the long number in the run’s URL, not its # number) or only the failing test. A GitHub Actions re-run uses the same commit, so a pass proves non-determinism, not a fix; record it as evidence.

Retries and quarantine, in brief: an automatic retry turns a run green when a flaky test passes on a later attempt, and quarantine moves a known-flaky test into a non-blocking job where it keeps running. Both are containment, not a cure: allow at most one retry, report "passed on retry" as flaky, and give every quarantined test an owner, a ticket and a deadline. The flaky tests guide covers the policy, and the CI/CD testing guide’s triage flow shows where a re-run fits.

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 30      # job limit: cuts off a hang
    strategy:
      fail-fast: false       # one failing shard does not cancel the others
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      # checkout and install steps omitted
      - run: npx playwright test --shard=${{ matrix.shard }}/4

Try it: YAML Formatter

Bisecting a regression across pipeline runs

When a scheduled run on main turns red after many merges, the pipeline history gives you the last green and the first red run, each with its commit. First check the test’s history in the CI test reports: a test that has failed now and then for weeks is a flaky candidate, not a new regression. If it fails every time from one run on, binary search finds the change: test the middle commit, keep the half where pass turns into fail, and repeat. Sixteen candidate commits need at most four test runs.

git bisect run automates the search: its command exits with 0 for a good commit, 125 to skip a commit that cannot be tested, and another code from 1 to 127 for a bad one. Trust the result only if the test fails every time on a bad commit, so repeat it at each step, and if the environment stays fixed. If the last green commit now fails too, the cause is outside the code, such as a runner image, an external service or expired test data.

# last green nightly: a1b2c3d     first red nightly: 9f8e7d6
git bisect start 9f8e7d6 a1b2c3d
# TZ=UTC gives the browser the runner's time zone (Linux and macOS)
git bisect run sh -c 'npm ci || exit 125
  TZ=UTC npx playwright test -g "order date" --repeat-each=3'
# git prints "<sha> is the first bad commit"
git bisect reset

Worked example: a red nightly run

Nightly run 412 on main is red; run 411 was green, with nine commits in between. Fail-fast is off, so all four shards finished, and shard 3 has one failure in invoice.spec.ts, "invoice shows the order date": expected 1 Oct 2026, received 30 Sep 2026. It failed on its first attempt and on its retry, so the report lists it as failed, not flaky. The trace shows that the API returned createdAt as 2026-09-30T22:40:00Z, which is 01:40 on 1 October in Riyadh, the store’s time zone: the data is right and the display is wrong.

On the tester’s laptop in Riyadh the test passes. With the job’s commit, lockfile, browser build and seed, one difference remains: the runner’s browser uses UTC. With Playwright’s timezoneId set to UTC, the test fails 10 times out of 10. With TZ=UTC in the bisect command, so that every commit runs in the runner’s time zone, git bisect run names c41d9a0, "Format dates with Intl.DateTimeFormat", which dropped the store’s time zone. The tester files a product defect with the run link, the test, the trace, the reproduction setting and the first bad commit, and proposes pinning timezoneId in the test configuration so that local runs behave like CI.

Try it: Timezone Converter

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• GitHub Actions documentationGitHubHow workflows, jobs, runners and artifacts are defined in GitHub Actions — where automated tests run on each push or pull request.
• Reference — GitGit projectThe official reference for every Git command and its options, grouped by task — setup, branching, sharing, patching, inspection and debugging.
• Playwright DocumentationMicrosoft (Playwright)Official documentation for Playwright, an end-to-end test framework that drives Chromium, WebKit and Firefox.