Automation Reporting and Debugging

Why it matters

A suite is only as useful as the time it takes to act on a red result. If a failure leaves only expected true, received false, someone must re-run the test and guess the CI conditions; teams then re-run until the build is green, and real regressions pass with the noise.

Good evidence turns a failure into a short investigation: the screenshot and the trace usually show whether to fix the test, call the environment owner or report a defect. It also makes a bug report actionable, with the failing request, the response and the trace attached instead of “checkout test failed”.

Reports: one run, several readers

One run has several readers. Whoever started it reads the console. Whoever investigates a failure needs an HTML report that opens each test into its steps, error, retries and attachments. The CI server needs a machine-readable file, usually JUnit XML, to show results per test and keep their history. In Playwright, list the formats you need under reporter.

A good report answers, without asking anyone, what failed, where, in which environment and with what evidence. Name tests after the behaviour they check, wrap business-level actions in test.step() so the report reads like a test case, and attach data the screen does not show.

• Run context: build or commit, browser, base URL and start time, to match a failure with deployments and server logs.
• Flaky tests (failed, then passed on retry) counted apart from passed ones; otherwise the report hides them.

// playwright.config.ts (excerpt): one run, three readers
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['list'],                                        // the terminal
    ['html', { open: 'never' }],                     // people: playwright-report/
    ['junit', { outputFile: 'results/junit.xml' }],  // the CI server
  ],
});

// checkout.spec.ts: steps and attachments make the report read like a test case
import { test, expect } from '@playwright/test';

test('VAT is added to the order total', async ({ page }, testInfo) => {
  const cart = await test.step('Create a cart with one 150.00 SAR book', () =>
    createCart(page, { sku: 'BOOK-01', qty: 1 }));     // createCart: a project helper
  await testInfo.attach('cart', {
    body: JSON.stringify(cart), contentType: 'application/json',
  });
  await test.step('Open checkout and check the total', async () => {
    await page.goto('/checkout/' + cart.id);
    await expect(page.getByTestId('order-total')).toHaveText('172.50 SAR');
  });
});

Try it: XML Formatter

Screenshots, video and traces: what each one shows

Keep evidence for failed tests only: each kind answers a different question and costs run time and storage.

• Screenshot: the page at the moment of failure, answering “what did the user see?”: an error banner, a spinner, a dialog over the button. Playwright: screenshot: 'only-on-failure'; Cypress takes one automatically when a test fails in cypress run; Selenium: call takeScreenshot() (JavaScript) or getScreenshotAs (Java) in the runner’s failure hook.
• Video: the whole test, showing order and movement such as a dialog that flashed, but no DOM, network or console data. Files are large: in Playwright use video: 'retain-on-failure'; Cypress records only when its video option is enabled (off by default).
• Trace: a recording you step through. A Playwright trace holds each action with DOM snapshots before and after it, the call log, console, network requests and responses, and the source line. Open it with npx playwright show-trace, from the HTML report, or at trace.playwright.dev, which loads it in your browser without uploading it.
• Logs: runner output, browser console and the server logs of the same time window, matched by timestamp or request id.
• Evidence is data: a trace can hold request headers, cookies and everything the test typed, so use synthetic data and let artifacts expire.

// playwright.config.ts (excerpt): keep evidence for failures only
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'on-first-retry',   // needs retries; with none, use 'retain-on-failure'
  },
});

// Then, in a terminal, open a trace downloaded from a CI run:
//   npx playwright show-trace trace.zip

Debugging a failed test: from the first error to the cause

ISTQB separates testing, which shows failures, from debugging, which finds, analyses and removes their causes. For a failed automated test, the tester’s part is failure analysis: is the defect in the product, the test code or the environment? The tester debugs the test code; a developer debugs the product, starting from the evidence.

Read the first error, not the last; later errors are usually consequences. Its type tells you which artifact to open. A setup error means the test never reached its check: read the hook’s error and the server log. A timed-out action means an element could not be found or used: open the DOM snapshot before that action and its call log. A failed assertion means the page showed something else: compare the screenshot with the expected value, then check in the network tab whether the data arrived wrong or late.

• Reproduce locally: npx playwright test --debug steps through the test in the Playwright Inspector, --ui opens UI mode with a time-travel view of each run, and await page.pause() stops at a chosen line in a headed run. Change one thing at a time, as in the lesson on reproducing failures.
• Do not raise a timeout before you know why the time ran out: a slower response can itself be the defect.

Try it: Text Diff Try it: Timestamp Converter

Worked example: a wrong order total in the nightly run

The nightly UI run on staging, build 512, reports one failure; its retry failed too, so it is failed, not flaky. The report and trace excerpts are below, shortened.

The first error is an assertion: the element was found but showed the wrong value. The screenshot shows VAT 0.00 in the cart, so the test read what the page displayed. In the trace, the response to GET /api/cart/8812 already contains "vatRate": 0: the browser rendered what the server sent. The last green run, build 509, kept no trace, but its report still holds the cart JSON the test attaches in every run, with "vatRate": 0.15; a JSON diff of the two shows that only vatRate and total changed.

So the cause is a back-end change between builds 509 and 512 or a tax setting changed on staging; one question to the developer or the environment owner settles which. The test is right and stays as it is. The report “Cart API returns vatRate 0 for a Saudi cart since build 512” carries both responses, the trace and the build range.

✘ checkout.spec.ts:42 › checkout › VAT is added to the order total
  failed (retry #1 failed too) · chromium · staging · build 512

  Error: expect(locator).toHaveText(expected)
  Locator:  getByTestId('order-total')
  Expected: "172.50 SAR"
  Received: "150.00 SAR"
    at checkout.spec.ts:51

  Attachments: cart · screenshot · video · trace

Build 512 › trace › Network › GET /api/cart/8812 → 200
  { "subtotal": 150.00, "vatRate": 0,    "total": 150.00 }

Build 509 (last green run, no trace kept) › HTML report › attachment "cart"
  { "subtotal": 150.00, "vatRate": 0.15, "total": 172.50 }

Try it: JSON Diff

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• Certified Tester Advanced Level Test Automation Engineering (CTAL-TAE) v2.0ISTQB · 2024ISTQB's advanced syllabus on designing, implementing and maintaining test automation solutions and their architecture.
• Trace viewer — PlaywrightMicrosoft (Playwright)Recording and replaying a test run step by step — DOM snapshots, network, console — to diagnose failures, especially from CI.
• Cypress DocumentationCypress.ioOfficial documentation for end-to-end and component testing in the browser with Cypress.