Reporting Product Quality to Leadership

Why it matters

Leaders decide budgets, priorities and release dates. If quality reaches them only as test case counts or angry customer tickets, they decide on instinct: they cut testing when the numbers look good, or demand "more testing" after an incident without knowing where the gap is.

For a tester moving into a lead role, this report is where testing gains influence: evidence linked to outcomes, risk stated plainly and a specific ask turn QA from a cost line into a source of decisions.

Three reports, three audiences

The "test summary report" many templates ask for is the test completion report from the lesson on test reporting; the 2018 ISTQB Foundation syllabus used the old name. It closes one stage of testing against its exit criteria, for whoever accepts that release.

Leadership asks other questions over a longer period: is quality improving, where are customers hurt, what risk do we carry, what do you need? So use three reports:

• Test completion report: per test stage, for whoever accepts the release; what was tested and what is still open.
• Residual-risk statement: per release, for the go/no-go decision owner; what could still hurt users, and who accepts it.
• Quality review: a monthly dashboard and a quarterly meeting for leadership; quality across releases and teams, causes, risk carried forward and decisions needed.

Combining test results with production signals

Test results show what you checked; production shows what users experienced. A 99% pass rate says nothing about untested paths, and a quiet quarter may only mean that few people used the new feature. Put four signals side by side, per team:

• Evidence before release: leading indicators from your KPI set, such as high-risk tests passed at release.
• Escape rate: defects found in production within an agreed window, as a share of all defects found for those releases (the complement of DDP from the lesson on defect metrics). Give counts and window: "9% (4 of 44, 30 days)".
• Incidents: unplanned events in production that harm users or the business, from the incident log, by severity level (often Sev1 for the most serious; each company defines its own levels) and user impact. Label each one: capacity, configuration and third-party failures are not escaped defects.
• Change fail rate: DORA's share of deployments that need immediate intervention, such as a rollback or a hotfix. It catches deployment failures that tests of the code never see.

Try it: Percentage Calculator

Worked example: a quarterly quality review

The food-delivery app from the previous lesson has three teams: checkout, ordering and the courier app. Each quarter the QA lead sends leadership this one-page review two days before a 45-minute meeting, built from data the teams already keep and checked first by each team lead, so nobody is surprised.

Part 1:1QUARTERLY QUALITY REVIEW · Food-delivery app · Q3 · data as of 2 Oct
Readers: CTO, head of product, engineering managers, support lead
Sources: test management tool, defect tracker, incident and deployment logs

Part 2:2HEADLINE  Checkout improved and met its targets. Courier app quality fell:
          2 of the 3 Critical escapes this quarter came from it.
          Decision needed by 15 Oct: test devices for the courier app.

                               Checkout          Ordering          Courier app
Part 3:3BEFORE RELEASE (Q3)
  High-risk tests passed       100%              96%               71%
Part 4:4IN PRODUCTION (Q2 -> Q3)
  Escape rate, 30 days         18% -> 9% (4/44)  12% -> 11% (3/27) 10% -> 21% (7/33)
  Critical escapes             2 -> 0            0 -> 1            0 -> 2
  Change fail rate             14% -> 8% (4/50)  10% -> 10% (3/30) 8% -> 25% (3/12)
  Sev1 incidents               1 -> 0            0 -> 0            0 -> 1
  Sev2 incidents               3 -> 1            1 -> 2            1 -> 3
Part 5:5TARGETS  escape rate <= 10% · no Critical escapes · change fail rate <= 10%

Part 6:6WHY  Courier: 10 of 35 high-risk tests could not run (no older Android
     devices), and 5 of its 7 escapes occur only on those versions. The Sev1
     (live tracking down for 3 hours) followed an Android permission change.
     Ordering: the Critical escape was a stale menu price after an admin edit.
     Checkout: contract tests added in Q2 stopped 6 breaking payment-API
     changes before release.

Part 7:7RESIDUAL RISK CARRIED INTO Q4
  R1  Courier app on older Android versions: untested until devices arrive
  R2  New payment provider in November: no load test possible before go-live
  R3  14 Minor defects deferred, each with a workaround (list attached)

Part 8:8ASKS  1. Approve renting 6 older Android test devices (quote attached),
         by 15 Oct
      2. Head of product: accept R2 at the November go/no-go, or move the date

• Headline first: the trend and the decision needed, with a deadline. Every rate shows its counts, the previous quarter and the target; each team is read against its own past, never ranked. Escape rates count only releases whose 30-day window had closed by the data date; later releases join the next review.
• Evidence sits above outcomes so that readers compare them. For the courier app they agree: only 71% of high-risk tests passed and 21% of defects escaped, so the leading indicator warned before release; next time, act on that warning before shipping. Green tests with rising escapes usually mean the tests miss what users do; stable escapes with a rising change fail rate point at deployment or infrastructure.
• The "why" lines come from escape analysis (why each escaped defect got through) and blameless incident reviews, or postmortems, which look for the conditions behind a failure rather than someone to blame. They speak of user impact, so the meeting can spend its time on the asks.

QUARTERLY QUALITY REVIEW · Food-delivery app · Q3 · data as of 2 Oct
Readers: CTO, head of product, engineering managers, support lead
Sources: test management tool, defect tracker, incident and deployment logs

HEADLINE  Checkout improved and met its targets. Courier app quality fell:
          2 of the 3 Critical escapes this quarter came from it.
          Decision needed by 15 Oct: test devices for the courier app.

                               Checkout          Ordering          Courier app
BEFORE RELEASE (Q3)
  High-risk tests passed       100%              96%               71%
IN PRODUCTION (Q2 -> Q3)
  Escape rate, 30 days         18% -> 9% (4/44)  12% -> 11% (3/27) 10% -> 21% (7/33)
  Critical escapes             2 -> 0            0 -> 1            0 -> 2
  Change fail rate             14% -> 8% (4/50)  10% -> 10% (3/30) 8% -> 25% (3/12)
  Sev1 incidents               1 -> 0            0 -> 0            0 -> 1
  Sev2 incidents               3 -> 1            1 -> 2            1 -> 3
TARGETS  escape rate <= 10% · no Critical escapes · change fail rate <= 10%

WHY  Courier: 10 of 35 high-risk tests could not run (no older Android
     devices), and 5 of its 7 escapes occur only on those versions. The Sev1
     (live tracking down for 3 hours) followed an Android permission change.
     Ordering: the Critical escape was a stale menu price after an admin edit.
     Checkout: contract tests added in Q2 stopped 6 breaking payment-API
     changes before release.

RESIDUAL RISK CARRIED INTO Q4
  R1  Courier app on older Android versions: untested until devices arrive
  R2  New payment provider in November: no load test possible before go-live
  R3  14 Minor defects deferred, each with a workaround (list attached)

ASKS  1. Approve renting 6 older Android test devices (quote attached),
         by 15 Oct
      2. Head of product: accept R2 at the November go/no-go, or move the date

Try it: Markdown Formatter Try it: Markdown to PDF

Stating residual risk for a go/no-go audience

Before each release the question narrows: what could still hurt users if we ship, and who accepts it? Sort what remains as the risk-based testing guide does (reduced, partial, open, not tested, out of scope), then rewrite each item that matters for readers who know neither risk IDs nor test counts. Here is R2 from the review in five parts, for the November go/no-go; the meeting itself belongs to the lesson on release readiness.

• Impact, in users' terms: if the new payment provider slows at the Friday evening peak, customers wait at checkout and some abandon their orders.
• Evidence: payment tests passed on the provider's rate-limited test environment, so peak load was never tested.
• Likelihood, reasoned and never an invented percentage: unknown, with no evidence either way, which is the risk.
• Containment: 10% of payments go to the new provider in week one, an alert fires when a payment takes over 5 seconds, and a rehearsed switch back takes under 15 minutes.
• Acceptance, with your recommendation labelled as yours: the head of product decides; QA recommends the 10% start.

Try it: Risk-Based Test Generator

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• Certified Tester Advanced Level Test Management (CTAL-TM) v3.0ISTQB · 2024ISTQB's advanced syllabus on managing testing: test planning and estimation, risk-based testing, monitoring, metrics and reporting.
• DORA's software delivery performance metricsDORA (Google Cloud)The DORA delivery metrics — change lead time, deployment frequency, failed deployment recovery time, change fail rate and related measures — grouped by throughput and instability.
• Postmortem Culture: Learning from Failure — Site Reliability Engineering (John Lunney and Sue Lueder)Google / O'Reilly Media · 2016Blameless postmortems: which incidents call for one, what it records — impact, root causes and follow-up actions — and why blame is kept out of it.