Defect Metrics

Why it matters

A tracker with hundreds of defects tells nobody anything until it is summarised. A few well-defined numbers answer the questions a team has to decide on: is the release ready, which component needs more testing or a redesign, are fixes done properly, and is testing finding the important defects before users do?

For a tester they matter twice. You produce the raw data, because every report, severity and status change feeds these numbers, and careless records give wrong metrics. And you are often asked to explain them: "the escape rate rose because the new SMS provider was tested only against its sandbox" helps a team far more than "20%".

The core defect metrics and their formulas

Each metric is a ratio, a time span or a count over fields you already record: creation and closure dates, severity, component, status history, and where the defect was found (a test level or production). Count valid defects only: leave out duplicates and reports rejected as "not a defect".

• Defect density = defects ÷ size of the work product, such as thousand lines of code (KLOC), function points or stories. It compares areas of different size: 20 defects in a small module can be worse than 30 in a large one.
• Defect detection percentage (DDP) = defects found by testing ÷ (found by testing + found afterwards) × 100, with an agreed window for "afterwards", such as 30 days in production. The ISTQB glossary defines it per test level, so it also works for system testing alone.
• Escape rate = defects found afterwards ÷ the same total; it is the complement of DDP. Some teams call it defect leakage, and some divide by the defects found in testing instead of the total, so check the formula before you compare.
• Defect age = time from report to closure, or to today for an open defect. Report the median (the middle value of the sorted ages) and the oldest defect per severity, not only the average.
• Reopen rate = defects reopened ÷ defects resolved in the same period. A defect is reopened when confirmation testing shows the fix did not work, or when the failure comes back.
• Distribution = the split of defects by one attribute: severity, component, where found or root cause. It shows where to act; a single total never does.

Try it: Percentage Calculator Try it: Date Difference Calculator

Worked example: one release of a clinic booking app

Thirty days after release 3.4 of a clinic appointment-booking app went live, the QA lead prepares the retrospective. Of 55 reports from testing, 4 were duplicates and 3 were rejected as "works as designed", leaving 48 valid defects; users and support reported 12 more. The block below holds the cleaned data and the results; this is how the lead reads them.

• DDP is 80%, which looks acceptable, but one escape is Critical: two patients could book the same slot (fixed in a hotfix). The useful question is why system testing missed concurrent bookings, not whether 80% is good.
• Booking has the most defects (27), but Notifications, new code for SMS reminders, has almost twice its density: 8.4 against 4.5 per KLOC. It deserves extra attention in the next release.
• The reopen rate is 12%, and 4 of the 6 reopened defects are in Payments, a sign that fixes there are confirmed on one path only; the team adds refunds to its confirmation tests.
• 17 of the 60 defects trace back to requirements: an argument for earlier requirement reviews, not for more test cases.
• Ten defects are still open, none of them Blocker or Critical. Mean age 16.6 days, median 4.5: most are new, and two old ones pull the mean up. What matters is a Major open for 60 days, waiting for the SMS provider, and a Trivial open for 75 days that should be fixed or deliberately deferred.

Release 3.4, clinic booking app: tracker export 30 days after go-live

Found before release   55 reports - 4 duplicates - 3 rejected = 48 defects
Found in production    first 30 days                         = 12 defects
                       (1 Critical, 3 Major, 8 Minor)

DDP           48 / (48 + 12)            = 80%
Escape rate   12 / 60                   = 20%
Reopen rate   6 reopened / 50 resolved  = 12%   (4 of the 6 in Payments)

Component       Defects   Changed KLOC   Density
Booking              27            6.0   4.5 per KLOC
Payments             12            3.0   4.0 per KLOC
Notifications        21            2.5   8.4 per KLOC

Root cause (60)     requirements 17 | code 31 | environment 8 | data 4

Open defects (10)   Major 3 | Minor 5 | Trivial 2   (no Blocker or Critical)
Age in days         Major 1, 4, 60 | Minor 2, 2, 3, 6, 8 | Trivial 5, 75
                    mean 16.6 | median 4.5

Reading defect metrics carefully

Before you compare two numbers, check that they were counted the same way, and ask what else could explain them.

• Fixed rules: the same definition of a valid defect, the same severity scale and classification (IEEE 1044 is a standard scheme for classifying software anomalies) and the same window. An escape rate measured after 7 days cannot be compared with one measured after 90.
• Counts next to percentages: small numbers swing hard. 2 escapes out of 10 is 20%; one more makes it 3 of 11, or 27%.
• Trends, not snapshots: compare a component or a team with its own previous releases, not with another product.
• Context: a low count can mean good code or little testing; a high density can mean weak code or thorough testing. As the first testing principle says, testing shows the presence of defects, not their absence.
• Pairs: read DDP with the severity of what escaped, defect age with severity, and density with how much testing that area received.

How defect metrics are misused

Defect metrics describe a product and a process. They go wrong when they become targets for people: once a number decides praise or blame, people manage the number instead of the quality, an effect usually called Goodhart’s law.

• Bugs per tester as a performance score: one problem is split into several reports, hard investigations are avoided, and testers compete instead of helping developers.
• Defects per developer: every report becomes an argument, and problems are fixed quietly in chat without a record.
• A target such as "defect age under 10 days": defects are closed as "cannot reproduce" or deferred without agreement, and the age improves while users wait.
• A target on escape or reopen rate: production defects are relabelled as change requests, and reopened defects are filed as new ones.
• League tables across teams: different products, risks and reporting habits make the comparison meaningless, and teams learn to report less.

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• Certified Tester Advanced Level Test Management (CTAL-TM) v3.0ISTQB · 2024ISTQB's advanced syllabus on managing testing: test planning and estimation, risk-based testing, monitoring, metrics and reporting.
• IEEE 1044-2009 — IEEE Standard Classification for Software AnomaliesIEEE Standards Association · 2009A standard vocabulary and attributes for classifying software anomalies — defects, failures and their impact (listed by IEEE as inactive-reserved).