Module 25 · Professional QA & test management

QA Metrics and KPIs

By the Mutqan Soft QA team · · 19 min · Advanced

A metric is a measurement with a defined method: what is counted, how, and from which data. A key performance indicator (KPI) is one of the few metrics chosen to show progress toward a business goal, and it comes with an owner, a target, thresholds that trigger action and a fixed review. "Pass rate of last night’s regression run" is a metric for the team; "customer-found Critical defects per release" can be a KPI for the whole QA function.

This lesson shows how to build a small KPI set from a business goal, present it to leadership, compare teams fairly, and explain the value of testing with the cost-of-quality model instead of invented return-on-investment figures.

Why it matters

Without agreed KPIs, quality discussions fall back on opinions or on the easiest number to collect, usually a count of test cases. Leadership then funds testing on instinct and learns about quality from customer complaints.

For a tester moving into a lead role, KPIs and the cost of quality are the language of budget meetings and release decisions. "Escaped defects cost us three hotfixes and about 90 support hours last quarter; here is what we propose" carries more weight than "we need more testers", as long as every number traces back to real data.

Metrics, KPIs, leading and lagging indicators

The ISTQB Foundation syllabus groups the metrics used in testing into project progress, test progress, product quality, defect, risk, coverage and cost metrics. A team may track dozens of them to steer its daily work.

The term KPI comes from general management practice and is used loosely. A useful working definition: a metric becomes a KPI when it tracks a business goal, has a named owner, a target and thresholds with agreed actions, and is reviewed on a fixed cadence by the people who decide. Every KPI is a metric; most metrics should never become KPIs.

A metric measures one thing with a defined method; a KPI is one of a few metrics tied to a business goal, with an owner, a target, thresholds and a fixed review.

Metric

A measurement with a defined method: what is counted, how, and from which data

Question it answers:
What is the value of this measure now?
Example:
Pass rate of last night’s regression run
Target and thresholds:
Often none; read in context and as a trend
Owner and reader:
The team that collects and uses it
Review cadence:
As often as useful, often daily
How many:
As many as the team needs, often dozens

KPI

A metric chosen to track a business goal, with an owner, a target and thresholds

Question it answers:
Are we on track for a business goal?
Example:
Customer-found Critical defects per release
Target and thresholds:
Always: a target, and thresholds with an agreed action
Owner and reader:
A named owner; reviewed by the people who decide
Review cadence:
Fixed: per sprint, month or release; the set itself each quarter
How many:
Three to five per business goal

Every KPI is a metric; most metrics never become KPIs.

  • Lagging indicator — measures an outcome after it happened: customer-found defects, production incidents, change fail rate (the share of deployments that need immediate intervention such as a rollback or hotfix). It is trustworthy, but it arrives too late to change the release it describes.
  • Leading indicator — moves earlier and is expected to predict the outcome: high risks with passing tests before release, stories whose acceptance criteria were reviewed before development, the flaky rate of the regression suite. You can act on it in time, but its link to the outcome is an assumption.
  • Pair them: each lagging KPI gets one or two leading indicators the team can move this sprint, and each leading indicator is checked against its KPI over several releases. If risk coverage rises but escapes do not fall, the leading indicator measures the wrong thing.

Building a KPI set: from business goal to indicators

Two ideas from the test metrics guide matter even more for KPIs. Start from the decision, not the dashboard: a number nobody will act on does not belong in the set. And Goodhart’s law: once a measure becomes a target people are judged by, they improve the number instead of the quality, so counts of test cases or bugs per tester make poor KPIs. Give every KPI a written definition and a counter-metric that exposes gaming.

A KPI tree makes the chain explicit: one business goal, the quality objective that supports it, one or two lagging KPIs, the leading indicators the team can move, and the data that feeds them. The example is a food-delivery app; its baselines come from its own last three releases.

  • Targets and thresholds — the target is where you aim; each threshold band has an agreed action (amber: discuss at the weekly review; red: the release owner decides explicitly). Set them from your own baseline, not from other companies’ benchmarks.
  • Review cadence — leading indicators every sprint with the team, lagging KPIs monthly or per release with leadership, and the set itself each quarter: retire indicators that no longer drive decisions.
  • Definitions — write what is counted and what is excluded, the data source, the counting window (for example 30 days after release) and the owner, so that next quarter’s number means the same thing.
BUSINESS GOAL      More repeat orders
QUALITY OBJECTIVE  Customers rarely meet a checkout defect in production

LAGGING KPIs (per release, reviewed with leadership)
  K1  Customer-found Critical/Major checkout defects, 30-day window
      baseline 5 · target <= 2 · amber 3 · red >= 4
  K2  Change fail rate of the checkout service (DORA definition)
      baseline 14% · target <= 10% · amber > 10% and <= 15% · red > 15%

LEADING INDICATORS (per sprint, reviewed by the team)
  L1  High checkout risks covered by passing tests at release     -> K1
      target 100% · red < 95% (release owner decides explicitly)
  L2  Checkout stories with acceptance criteria reviewed by QA
      before development                                          -> K1
      target >= 90% · red < 75%
  L3  Flaky rate of the checkout regression suite                 -> K2
      target < 2% · red > 5%

DATA  CI test results · defect tracker · deployment log ·
      support tickets tagged "checkout"

Try it: Risk-Based Test Generator Try it: Percentage Calculator

Leadership dashboards and comparing teams fairly

A leadership dashboard answers three questions in under a minute: are we on track, is it getting better or worse, and is a decision needed? Comparing teams is where KPIs do the most harm: different products, risk levels and counting rules produce different numbers for reasons unrelated to how well a team tests.

  • Show three to five KPIs per goal, each with its trend over six periods, the target, the band in words (on track, at risk, off track) rather than colour alone, the date and source of the data, and one sentence of interpretation.
  • End with the ask: the decision or support needed, and by when.
  • Compare fairly: the same written definitions and counting window, rates instead of raw counts (escaped defects per release or per 100 changes), and each team against its own baseline first.
  • Use cross-team views to find practices worth sharing, never to rank teams or people; once a ranking decides bonuses, the numbers stop being honest.

Try it: Markdown to PDF

The cost of quality: four categories

The cost-of-quality model from general quality management sorts what an organisation spends because of quality into four categories: the first two are spent to get quality right, the last two are the price of defects. Some testing texts call appraisal "detection". Reviews find defects in work products, so most teams count them as appraisal, while agreeing acceptance criteria before anything is built changes how the work is done and counts as prevention. Wherever you draw the line, draw it the same way every quarter.

  • Prevention — secure-coding training, agreeing acceptance criteria before development, coding standards, fixing a root cause found in a retrospective.
  • Appraisal — designing and running tests, reviews, static analysis, test environments and test data, test automation upkeep.
  • Internal failure — defects found before release: investigating, fixing and retesting them, and the build that waits for the fix.
  • External failure — defects found by customers: hotfixes, support calls, refunds, incident handling, contractual penalties, and lost customers who never appear in any ledger.

Showing value honestly: budgets and go/no-go

Quantitative value comes from your own records: defects found and fixed before release, known defects documented with workarounds, and the measured cost of the defects that escaped. Qualitative value is real but not countable: confidence to release, reputation, compliance with regulations and contracts. Report both and label which is which.

Invented ROI figures mislead. "Testing returned 900%" usually multiplies the defects found by a cost each one is assumed to have caused in production. Nobody knows which of them would have reached users or what they would have cost, and one challenged number discredits the whole report. Use measured costs, stated assumptions and ranges.

  • Budget request — start from measured external failure ("three checkout hotfixes and the support calls they caused took about 120 hours last quarter"), propose a specific prevention or appraisal investment, name the KPIs that should move, and agree a review date. Present the effect as an expectation to check, not a promised return.
  • Go/no-go — weigh the cost of more appraisal now (extra test days, delayed revenue) against the likely external failure cost of the open defects and untested risks. Give the release owner the residual risk in plain words and the estimate as a range; the decision is theirs.

Common mistakes

  • Turning every available metric into a KPI, so the dashboard shows thirty numbers without owners or targets.
  • Choosing only lagging KPIs, so the team learns about problems after customers do.
  • Copying targets from other companies instead of setting them from your own baseline.
  • Ranking teams or testers on raw counts such as defects found, which rewards gaming and punishes honest reporting.
  • Presenting an invented ROI percentage or a generic cost multiplier as fact.
  • Counting the cost-of-quality categories differently each quarter, so the trend reflects the bookkeeping, not the product.

The QA perspective

A QA lead builds the KPI set with the product owner and the engineering manager, because the goal and the decisions are theirs. Start small: one goal, two lagging KPIs, two or three leading indicators and a one-page definition sheet, with the thresholds and their actions agreed before the first report.

Keep a quarterly cost-of-quality log from data you already have (time tracking, the defect tracker, the incident log, support tickets). When someone asks what testing is worth, show it next to the KPI trends and say plainly what the numbers do not show.

Practice exercise

Classify a quarter’s quality costs and draft a KPI set

Your task

A mobile banking team wants fewer customer complaints about failed transfers next year. Below is its last quarter from time tracking and the defect tracker, and a slide a vendor suggested for the budget meeting. Classify the costs, draft a minimal KPI set for the goal, and decide what to do with the slide.

Material
Mobile banking app · transfers · last quarter · hours from time tracking

  A  Secure-coding training for developers                         24
  B  Root-cause workshop on the previous quarter's incidents;
     new review checklist adopted                                  12
  C  Review of transfer requirements and acceptance criteria       30
  D  Designing and running transfer regression tests              210
  E  Maintaining the test environment and test data                70
  F  Fixing and retesting 42 defects found in system testing      160
  G  Hotfix 7.4.1: transfer shown twice in the history             48
  H  Call-centre time on failed-transfer complaints                95

Production: 6 transfer defects reported by customers (2 Critical, 4 Major)

Vendor slide: "Testing ROI = 900%. Every defect found in testing
               saves 10x its fix cost in production."
Steps
  1. Assign lines A–H to prevention, appraisal, internal failure or external failure; total the hours and each category’s share (the percentage calculator helps).
  2. Write one lagging KPI for the goal with its definition, baseline, target and thresholds.
  3. Add two leading indicators the team can move within a sprint, and a review cadence.
  4. Say in two sentences what to do with the vendor’s ROI figure.
Show the model answer
Model answer

Prevention: A 24 + B 12 = 36 hours (5.5%). Appraisal: C 30 + D 210 + E 70 = 310 hours (47.8%). Internal failure: F 160 hours (24.7%). External failure: G 48 + H 95 = 143 hours (22.0%). Total: 649 hours. The requirements review (C) finds defects in a work product, so it is appraisal; the root-cause workshop (B) changes how the work is done, so it is prevention. External failure is nearly four times prevention before complaints and lost customers are even counted.

Lagging KPI: customer-reported Critical or Major transfer defects per quarter, duplicates excluded. Baseline 6 (2 Critical); target at most 3 and no Critical; at risk at 4 or 5; off track at 6 or more or any Critical, which triggers a review with the product owner.

Leading indicators: High-risk transfer scenarios with passing tests before each release (target 100%, off track below 95%) and transfer stories whose acceptance criteria QA reviewed before development (target 90%, off track below 75%). Review them every sprint, the KPI monthly with the product owner and quarterly with leadership, and the set itself each quarter.

Vendor slide: the 900% appears to be the assumed 10× multiplier restated (a saving of 10 for a cost of 1 is a net gain of 9, or 900%), and it also assumes every defect found would have reached customers; neither is measured here, and one challenged number discredits the rest. Show the measured 143 hours of external failure and the 6 customer-found defects, the proposed investment, the KPIs that should move, and a review date.

Frequently asked questions

What is the difference between a metric and a KPI?

A metric is any measurement with a defined method, such as pass rate or defect age. A KPI is one of a few metrics chosen to show progress toward a business goal, with an owner, a target, thresholds that trigger action and a fixed review. Every KPI is a metric; most metrics are not KPIs.

How many KPIs should a QA function have?

Few enough that leadership reads all of them: usually one or two lagging KPIs per business goal plus two or three leading indicators, three to five per goal in total. If a KPI has not changed a decision in two quarters, retire it.

Can I calculate the ROI of testing?

Only partly. You can measure costs (appraisal, internal and external failure) and track how external failure changes after an investment. You cannot measure what the defects testing found would have cost if they had escaped, so any single ROI percentage depends on assumptions. Present measured costs, stated assumptions and ranges instead.

Are the DORA metrics QA KPIs?

They measure the delivery system as a whole, not the test team, but change fail rate works well as a lagging quality KPI in a QA set, as long as its definition follows DORA’s and its target comes from your own baseline. Testing influences all of them.

References

Tools

Practise with Mutqan tools

Percentage Calculator

Calculate percentages, percentage change, increase/decrease and what percent X is of Y.

Utilities

Risk-Based Test Generator

Score features by likelihood and impact, then generate a prioritised risk-based test plan.

Test Design

Markdown to PDF

Convert Markdown (.md) files to PDF in your browser: headings, lists, tables, code blocks, quotes, links and images typeset with embedded fonts.

PDF Conversion

Go deeper

Your progress on this topic

Part of the QA Roadmap