Learning from Production Incidents

Why it matters

A production incident is the most expensive way to find a defect: users are harmed, support and finance spend time on it, and the team works under pressure. If the only outcome is a hotfix, the conditions that let the defect escape stay in place, and a similar incident follows.

For a tester, every incident is evidence about the test approach that no review can provide: it shows exactly where the test basis, test levels, environments or oracles did not match reality. Using that evidence without blame keeps the regression suite tied to real risks rather than guesses.

From incident to prevention: the feedback loop

This lesson starts where detection ends. The order matters: restore the service first, investigate second, and change tests and process last, once the facts are known.

In ISTQB terms, what users experience in an incident is a failure: the system does not do what it is required to do. The cause may be a code defect, but also configuration, data, capacity or a third-party service, so establish what failed and why before deciding that a fix and a new test are the answer.

• Detect and declare: an alert, a synthetic check or a customer report opens an incident record with time, impact and owner.
• Mitigate: stop the harm with a rollback, a feature flag or a configuration change. Mitigation buys time; it is not the fix.
• Reconstruct the timeline: deployments, configuration changes, alerts, logs and decisions, all in one time zone.
• Hold a blameless postmortem: impact, timeline, contributing causes and action items.
• Analyse the test gap: which test, check or review could have caught it, at which level, and why it did not.
• Prevent and verify: add regression tests and other actions, each with an owner and a due date, and confirm they are done.

Blameless postmortems

The lesson on quality culture introduced the postmortem as a written record of an incident: its impact, detection, mitigation, causes and follow-up actions. Here you run one. The Google SRE book recommends agreeing the triggers in advance, for example user-visible downtime or degradation beyond a threshold, any data loss, a manual intervention such as a rollback, a long resolution time, or a failure that monitoring did not detect.

Blameless means the review looks for contributing causes in the system and the process, not for a person to punish, and assumes everyone acted in good faith with the information they had. This is practical, not polite: people who fear blame leave out details, and the review misses the real causes. Blameless does not mean unaccountable; every action item has a named owner.

• Blaming: "The developer forgot the timeout case." Blameless: "The retry path had no test, and the code review checklist has nothing on slow responses."
• Blaming: "QA missed it." Blameless: "The payment mock in the test environment always answers in under a second, so the timeout path never ran."
• "Human error" is never a root cause on its own: ask what made the error easy to make and hard to notice.

From postmortem to test gaps and regression tests

The tester's main contribution to a postmortem is the test gap analysis. Trace the failure back through the test process: was there a requirement, a test condition and a test case for this behaviour, and did the test run against this build in an environment like production? Each "no" marks a gap, and one incident often has several. Each kind of gap calls for a different action.

• Missing test condition or test case: neither the requirement nor the risk analysis named the situation, or the test design never reached it, for example no boundary values at the edge. Add the condition, the acceptance criterion or the test case.
• Wrong test level: only an end-to-end happy-path test existed; a unit or API test reaches the faulty branch more easily.
• Unrealistic data or environment: test data, mocks, time zone, configuration or volumes differ from production.
• Weak oracle: the test ran but did not assert what went wrong, for example it checked the HTTP status but not the number of charges.
• Test not run: skipped, quarantined as flaky, or left out of the regression selection for this change.
• Detection gap: users found the problem before monitoring did. That action belongs to monitoring, not to the test suite.

Try it: Regression Scenario Generator Try it: Test Case Generator

Worked example: a double charge after a release

A fictional incident in an online store. Release 7.3 lowered the payment client's timeout from 30 to 10 seconds. Under load the provider sometimes answered after more than 10 seconds; the client then retried with a new payment reference, which the provider treated as a new payment, so some customers were charged twice.

In the postmortem excerpt below, customers rather than monitoring detected the problem, 67 minutes after the first duplicate charge, and most action items are tests and checks, not the code fix.

• Once the incident was declared, mitigation took twenty minutes; prevention needed a requirement, two tests, a new test-environment capability and an alert.
• Before the fix, the API test was run against release 7.3 and failed with two charges; with the fix it passed. A regression test that has never failed against the defect proves nothing about it.
• The tests target the condition, a slow provider followed by a retry, not the exact orders from the incident, so they also protect every other flow that uses the same payment client.

INC-0412  Customers charged twice for one order after release 7.3 (fictional)

Impact     37 orders charged twice; all refunded the same day; no data lost
Timeline   all times UTC+3
  09:40  Release 7.3 deployed: payment client timeout 30 s -> 10 s
  09:58  First duplicate charge (found later in the payment logs)
  11:05  First customer complaint to support
  11:30  Incident declared; payments team and QA join
  11:50  Mitigated: timeout back to 30 s by configuration; no new duplicates
  15:00  All duplicate charges refunded
Detection  by customers, 67 minutes after the first duplicate; no alert fired

Contributing causes (no individuals named)
  - A retry after a timeout sends a new payment reference, so the provider
    cannot recognise it as the same payment.
  - The payment requirement said nothing about retries.
  - The payment mock in every test environment answers in under 1 s,
    so the timeout-and-retry path never ran in any test.
  - No monitoring signal for "more than one successful charge per order".

Action items                                                      owner  due
  1  Acceptance criterion: a retried payment never charges twice  PO     7.4
  2  Unit test: a retry reuses the original payment reference     dev    7.4
  3  API test: the mock answers the first call after 12 s, the
     retry at once; exactly one charge, order status PAID         QA     7.4
  4  Payment mock simulates slow answers and, like the provider,
     treats a repeated payment reference as the same payment      QA     7.4
  5  Alert on any order with more than one successful charge      ops    7.4

Try it: Timestamp Converter Try it: Date Difference Calculator

Frequently asked questions

References

• Postmortem Culture: Learning from Failure — Site Reliability Engineering (John Lunney and Sue Lueder)Google / O'Reilly Media · 2016Blameless postmortems: which incidents call for one, what it records — impact, root causes and follow-up actions — and why blame is kept out of it.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.