Resilience Testing

Resilience testing defined: failure and recovery

The pillar guide uses resilience in its broad sense — behaviour under load, under failure and under attack. This guide covers the failure-and-recovery part: what happens when a piece of the system breaks while it carries normal traffic.

A resilient system is not one that never fails; it is one whose failures stay small and short. Resilience testing turns that idea into checks: during a named fault the service must stay within a written limit (for example, 99% of checkouts still succeed), switch to a named reduced mode if it cannot (prices served from a cache, orders queued), and be back to normal within a stated time after the fault ends — without anyone restarting something by hand.

ISO/IEC 25010:2023 places these properties under reliability, which it divides into faultlessness, availability, fault tolerance and recoverability. Resilience testing exercises the last three directly: is the service there when it is needed, does it operate as intended despite hardware or software faults, and after an interruption does it recover the affected data and return to the desired state? Chaos engineering is the experimental form of the same work; its principles favour running experiments on production traffic once that can be done safely.

In performance testing teams, resilience testing is usually the step after the stress test: the same environment, the same workload model and the same load generator, with a fault added. The Stress Testing lesson in the QA Roadmap covers recovery after overload; resilience testing covers recovery after a failure, under normal load.

• Continuity — the share of user requests that still succeed while the fault is present, compared with the steady state before it.
• Containment — the fault stays in the part of the system it hit: a slow payment provider does not slow down product search.
• Degradation — when full service is impossible, the product switches to the reduced mode its requirements name, and tells the user.
• Detection — monitoring notices the fault and alerts the right people within the agreed time.
• Recovery — after the fault ends, the service returns to its steady state on its own, within the target, with no lost or duplicated data.

Resilience testing vs performance testing

The two share tools and environments, which is why they are often confused. The difference is what changes during the test. A performance test — load, stress, spike or soak — changes the amount of traffic and keeps every component healthy, to find out how fast the system is and how much it can carry. A resilience test keeps the traffic at a realistic, steady level and changes the health of one component, to find out what users experience when that component misbehaves and how the system recovers.

The ISTQB performance testing syllabus lists load, stress, scalability, spike, endurance, concurrency and capacity testing, and ISTQB's definition of stress testing also covers resources available below the specified minimum, such as less memory than required — which is where the two practices meet. In day-to-day work they run together: performance tests set the baseline, the capacity and the timeouts that make sense, and resilience tests check how the system behaves at that baseline when things fail. A system can pass every load test and still go down with the first slow dependency, because a load test never makes a dependency slow.

                    Performance testing                      Resilience testing
Question            How fast is it, and how much load can    What do users get when one part fails,
                    it carry before it breaks?               and how fast does it recover?
What changes        The load: users, request rate, data      The health of one component: latency,
                    volume, duration                         errors, crashes, network, full disk
What stays fixed    Every component is healthy               A realistic, steady load
Inputs              Workload model, load profile,            Steady-state baseline, hypothesis,
                    test data                                fault, blast radius, abort conditions
Pass criteria       p95/p99 latency, throughput and error    Success rate during the fault, degraded
                    rate at a stated load; capacity limit    mode shown, alert fired, recovery time
                                                             vs RTO/RPO, no lost or duplicated data
Typical tools       JMeter, k6, Locust                       Fault proxies (Toxiproxy), HTTP mocks,
                                                             Chaos Mesh, LitmusChaos, AWS FIS,
                                                             Azure Chaos Studio + a load generator
Output              Latency curves, capacity figure          Experiment record: expected vs observed,
                                                             findings, follow-up actions

Resilience requirements: RTO, RPO, SLOs and degraded modes

A resilience test needs what a load test needs: a target written as a number before the run. Five kinds of requirement cover most systems.

The sheet below is an example for a fictional shop's checkout; every line in it is a test.

• Recovery time objective (RTO) — NIST SP 800-34 defines it as the overall length of time an information system's components can be in the recovery phase before the organisation's mission or business processes are harmed. A resilience test measures the actual recovery time, from the fault to normal service, and compares it with the RTO.
• Recovery point objective (RPO) — in the same publication, the point in time to which data must be recovered after an outage; in other words, how much recent data the business can afford to lose. A failover test measures it by comparing the last records written before the fault with what exists after recovery.
• Service level objective (SLO) and error budget — Google's SRE book defines an SLO as a target value or range of values for a service level measured by an indicator (SLI), such as the share of successful requests. The error budget is the failure the SLO leaves room for: with a 99.5% success SLO over 30 days and 2,000,000 requests, 10,000 may fail. Resilience tests check that the SLO holds while a fault is contained, and production experiments are planned to spend only a small, agreed part of the budget. A degraded mode is not free either: requests it serves outside the SLO still spend budget, and the requirements should say how much.
• Degraded modes — what the product does instead of failing: read-only mode (customers can browse and see their orders but not place new ones), cached data (prices up to 10 minutes old), queueing (the order is accepted and the confirmation e-mail sent later), or switching off a non-essential feature (recommendations hidden). Each mode needs a trigger, a user message and an exit condition that can be tested.
• Detection and recovery time — how quickly an alert must fire, and how quickly the service must return to its steady state after the fault ends, with no manual step.

Resilience requirements - Bustan Market checkout (fictional example)

SLO          99.5% of checkout requests succeed within 3 s, rolling 30 days
             error budget 0.5%: about 10,000 of 2,000,000 requests a month may fail
Payments     slow (answers in up to 3 s): checkout completes, p95 <= 4 s, errors <= 1%;
             checkouts over 3 s still count against the SLO: at about 2,800 an hour,
             10% over 3 s spends ~280 (about 3% of the monthly budget) per hour
             failing: "Card payment is unavailable - your basket is saved",
             no HTTP 500, no duplicate charge; breaker opens, retries stop
             recovered: normal checkout within 60 s, no restart
Price cache  lost: prices read from the database, at most 200 queries/s, p95 <= 1.5 s
Order DB     zone loss: failover to the replica, RTO 15 min, RPO 1 min
E-mail       service down: orders accepted, confirmations queued, sent within 1 h
Detection    alert "checkout errors" to on-call within 5 min of user errors above 5%

Designing a resilience experiment

Every resilience test is an experiment with a written expectation. The Principles of Chaos Engineering describe its shape: define the steady state as a measurable output of the system that indicates normal behaviour, hypothesise that it will continue, introduce variables that reflect real-world events, and try to disprove the hypothesis by looking for a difference. The steady state is about outputs users feel — throughput, error rate, latency percentiles — not internal measures such as CPU. Of the four golden signals in the SRE book, latency, traffic and errors make the steady-state criteria; saturation, the fourth, is the diagnostic that explains them.

Write the hypothesis so that it can fail: "When the payment API answers 2 s slower, checkout keeps succeeding for at least 99% of requests with p95 under 4 s, because the client timeout is 3 s and the connection pool is sized for it." A hypothesis that names the mechanism tells you where to look when it fails.

• One variable at a time. Inject one fault of one size, so that a change in the result has one cause. Combine faults only after each has passed on its own.
• Blast radius. Decide in advance which environment, which share of instances or traffic, which users (test accounts only, or a small share of real traffic) and for how long. Start with the smallest and widen only after a pass.
• Abort conditions. Write the limits that end the experiment at once — for example, user-facing errors above 5% for two minutes, or any alert from a team outside the experiment — and who calls the stop. A load generator can enforce part of it: k6 thresholds marked abortOnFail stop the run as soon as they are crossed.
• A tested way out. The command that removes the fault must work before the fault goes in; try it once on a healthy system.
• Observability before injection. Check that the dashboards show the steady-state metrics and the fault itself (the dependency's latency, the breaker's state). If you cannot see the fault, you cannot see its effect; if monitoring misses an injected fault, that is the first finding.
• Baseline, then fault, then recovery. Record at least 10–15 minutes of steady state before injecting, keep the fault long enough for the metrics to settle, and keep measuring after removing it until the steady state is back.

RESILIENCE EXPERIMENT RX-<nnn>          Owner: <name>          Date: <yyyy-mm-dd>

Service -> dependency   <service> -> <dependency>
Steady state            <metric = value>, measured for <n> min before injection
Hypothesis              When <fault> hits <dependency>, <service> will <behaviour>
                        because <mechanism>; <metric> stays <limit>.
Fault (one variable)    <type>, <size>, <duration>, injected with <tool>
Blast radius            <environment>; <share of instances or traffic>; <which users>
Traffic                 <synthetic rate, or share of real traffic>
Observe                 <dashboards, logs, traces, alert names>
Abort if (any one)      <user-facing errors > x% for y min> | <p95 > z s>
                        | <alert from a team outside the experiment>
Abort method            <command or button>, last tried on <date>
Pass if                 <criteria compared with the steady state>
Recovery check          steady state back within <t> after removal, no manual step,
                        data reconciled (no lost or duplicated records)
Result                  pass | fail | aborted     Findings: <ids>     Follow-ups: <tickets>

Try it: API Health Checker Try it: Response Time Analyzer

A fault catalogue: what to inject and what to watch

Choose faults from your own architecture and incident history, highest risk first; the list below covers the ones most systems meet. The second half of each item is what to observe — the evidence that decides pass or fail.

• Dependency latency — add 500 ms, then a delay just under and just over the caller's timeout. Watch: the call ends at the configured timeout, connection or thread pools do not fill up, and endpoints that do not use the dependency stay as fast as before.
• Error responses — make the dependency return 500, 503 with a Retry-After header, or 429. Watch: retries happen only for retryable errors, Retry-After is respected, and the user sees a specific message rather than a raw 500.
• Timeouts and hangs — the dependency accepts the connection and never answers. Watch: your own timeout ends the call (not the operating system default, minutes later), and the resources the call held are released.
• Crashes and restarts — kill a process, a container or an instance while traffic flows. Watch: health checks take it out of rotation, in-flight requests are retried safely or fail cleanly, queued work is not lost, and it rejoins after the restart.
• Network partition and packet loss — drop 5–10% of packets, or cut the link between two components. Watch: loss shows up as latency from retransmission, clustered data stores do not accept conflicting writes on both sides of a partition, and connections are re-established once it heals.
• Disk full — fill the volume used for logs, temporary files or the database. Watch: an alert before it is full, writes that fail with a clear error instead of corrupting data, and recovery once space is freed.
• Clock skew — move one node's clock a few minutes ahead or behind. Watch: token validation (RFC 7519 allows only a small leeway for exp and nbf, usually no more than a few minutes), certificate validity checks, scheduled jobs, and anything that orders events by timestamp.
• Expired certificates and credentials — let a test certificate expire, or revoke an API key or secret. Watch: a clear error that names the cause, an alert days before expiry, and rotation that takes effect without a redeploy.
• Zone or region loss — stop every instance in one availability zone, or fail the database over to its replica. Watch: the remaining capacity carries the load, failover time against the RTO, data lost against the RPO, and DNS or load-balancer changes.
• Retry storms — take a dependency down, then bring it back while many clients are retrying. Watch: how many calls the dependency receives per user request, whether backoff with jitter spreads the retries out, and whether the dependency recovers instead of being knocked over again.
• Cache failure — stop the cache, or flush it so that every request misses. Watch: database load during the cold start, latency, whether the application limits how many requests rebuild the same entry, and whether it serves stale data where the requirements allow it.
• Message-queue backlog — stop the consumers for 30 minutes, then start them. Watch: queue-depth and message-age alerts, what producers do when the queue is full, drain time, and that each message is processed once (consumers must be idempotent, because redelivery happens).

Try it: Mock Error Response Generator Try it: SSL Certificate Checker Try it: JWT Expiration Checker

How to test each resilience pattern

Michael Nygard's Release It! describes stability patterns that keep a failure local — among them timeouts, circuit breakers, bulkheads, fail fast and shed load. The list below adds retries with backoff, idempotency and graceful degradation, which come from the SRE book and RFC 9110 rather than from that catalogue but are tested the same way. Knowing that a pattern is in the code is not the same as knowing that it works with your settings; each one has a test that proves it.

• Timeouts — inject latency just below and just above the timeout and check that the call ends at the configured value. In a chain of calls, each layer's timeout must be shorter than its caller's; the SRE book recommends propagating one deadline through the whole request, so that a downstream service stops working on a request the client has already given up on.
• Retries with exponential backoff and jitter — count the calls the dependency receives for one user request. Retries multiply across layers: in the SRE book's example, three layers that each retry three times turn one user action into 64 attempts on the database. Check the cap on attempts and on delay, that the delays are randomised (retries from many clients must not arrive in waves), that only retryable failures are retried (a timeout or a 503, never a 400 or a 422), and that Retry-After on 429 and 503 is honoured.
• Circuit breakers — raise the failure rate above the breaker's threshold and check that it opens (calls fail in milliseconds instead of waiting for timeouts), that after its reset timeout it lets a trial call through (half-open), and that it closes again when the dependency is healthy. The state changes should be visible in metrics or logs.
• Bulkheads — exhaust the connection or thread pool of one dependency and check that endpoints which use other dependencies keep their normal latency. This is the containment check: one slow partner must not take the whole service down.
• Idempotency — send the same request twice, and again after a timeout on the first attempt, and check that it has one effect: one order, one charge. RFC 9110 defines PUT and DELETE as idempotent and POST as not, so a POST that may be retried needs an idempotency key that the server deduplicates on.
• Graceful degradation — switch the dependency off and check that the degraded mode in the requirements appears, that the user is told, and that the product returns to full mode by itself after recovery. Degraded modes are rarely exercised in normal operation, which is exactly why they need a test.
• Load shedding and rate limiting — push traffic above capacity and check that the excess is rejected early and cheaply (429 or 503, ideally with Retry-After, in milliseconds), that important traffic is protected ahead of less important traffic, and that the requests still admitted meet their SLO. RFC 6585 defines 429 Too Many Requests.

Try it: API Status Code Simulator Try it: Mock Error Response Generator Try it: Concurrent Request Test Generator

How to run a resilience testing session (game day)

A resilience testing session — usually called a game day — is a scheduled, time-boxed exercise in which a team injects planned faults into a system and practises its response, so that the people, the runbooks and the alerts are tested along with the software. AWS's Well-Architected guidance describes game days as simulating events in production-like environments to test systems, processes and team responses, with the same teams who would handle the real event, run on a regular schedule.

Keep the first session small: one service, two to four scenarios, half a day, in a staging environment that matches production. Its value is the comparison between what the team expected and what happened — including what nobody noticed.

• Preparation (one to two weeks before): choose the service and the scenarios from the risk list and recent incidents; write a hypothesis and an abort condition for each; get the service owner's approval; tell on-call, customer support and dependent teams the date and the scope; check dashboards and alerts; rehearse removing each fault.
• Roles: a facilitator who runs the agenda and decides when to move on; one operator who injects and removes faults, so nobody else changes the system; responders — the on-call engineers — who handle what they see as they would a real incident; observers who watch dashboards and report; and a scribe who keeps a timestamped timeline.
• Runbook: for each scenario, the injection command, the expected observations, the removal command and the abort condition, in the order they will be used. Responders work from the team's normal incident runbooks, so gaps in those runbooks surface as findings.
• Safety: anyone may call stop; on stop, the operator removes every fault and the team confirms the steady state before doing anything else. Do not start during a real incident, a release or a peak period.
• Environment: a test environment with a fault proxy for the first runs; staging with production-like topology and synthetic traffic for most sessions; production only for experiments that already passed in staging, with a small blast radius, error budget to spare, off-peak timing and stop conditions in place.
• Debrief (the same day): go through each scenario — expected, observed, detected when and by whom, recovered when, manual steps needed. Keep it blameless, as the SRE book's postmortem chapter describes: the subject is the system and the process, not the person. Every finding becomes a ticket with an owner and a date, and a scenario that failed is run again after the fix.

RESILIENCE GAME DAY - <service> - <date>, <start>-<end> <time zone>

Goal          <the question this session answers>
Environment   <test | staging | production>    Traffic: <synthetic rate | real share>
Scope         in: <services>    out: <services>
Approved by   <service owner>, <on-call lead>
Notified      on-call, customer support, <dependent teams>, <provider if needed>

Roles
  Facilitator  <name>   runs the agenda, decides when to move on
  Operator     <name>   the only person who injects and removes faults
  Responders   <names>  on-call engineers, respond as in a real incident
  Observers    <names>  watch dashboards and report what they see
  Scribe       <name>   keeps the timeline: time, event, who

Stop rule     anyone may call STOP; the operator removes all faults (<command>)
Abort if      <user-facing errors > x% for y min> | <SLO burn > z> | <real incident>

Agenda
  09:00  Briefing: goal, scenarios, roles, stop rule; baseline dashboards checked
  09:20  Scenario 1  <fault>     hypothesis: <...>     time box 30 min
  10:00  Scenario 2  <fault>     hypothesis: <...>     time box 30 min
  10:40  Scenario 3  <fault>     hypothesis: <...>     time box 30 min
  11:20  All faults removed; steady state confirmed
  11:30  Debrief (blameless): expected vs observed, surprises, runbook and alert gaps
  12:15  Close: findings recorded, follow-up tickets with owners and dates

Record per scenario
  Injected / removed      <hh:mm:ss> / <hh:mm:ss>
  Detected by / at        <alert or person> / <hh:mm:ss>
  Steady state held       yes | no  (<values>)
  Recovered at            <hh:mm:ss>    manual steps: <none | list>
  Findings                <ids>

Try it: Website Uptime Checker

Resilience testing tools, and where each kind runs

Resilience testing tools fall into a few categories, mostly by where the fault is injected. The descriptions below come from each project's own documentation; which one fits depends on your platform, not on a ranking.

The script after the list is an example of a check that runs on every merge, written for the client settings of the worked example below (timeout 3 s, one retry, 5 s checkout deadline). Toxiproxy makes the payment call slower than the timeout; the script checks that checkout gives up with a clear 503 within its deadline after exactly one retry, then removes the fault and checks that the next checkout succeeds without a restart.

• Fault proxies sit between a service and its dependency. Toxiproxy, a TCP proxy for simulating network and system conditions in chaos and resiliency testing, adds "toxics" such as latency with jitter, bandwidth limits, timeouts, connection resets and sliced data, and is controlled through an HTTP API from the test code. It works at the TCP level, so it cannot return an HTTP status; pair it with an HTTP mock for 500, 503 and 429.
• HTTP mocks and stubs return chosen status codes, bodies and delays in place of a real dependency — the cheapest way to test error handling and retries in CI. The API Status Code Simulator on this site returns any status from 200 to 599 with an optional delay of up to 10 s for manual checks; it is rate-limited, so use a local mock for automated runs at volume.
• Kubernetes chaos platforms inject faults into pods, nodes and the network around them. Chaos Mesh, a CNCF incubating project built on Kubernetes custom resources, documents pod failures, network latency and packet loss, DNS and HTTP faults, CPU and memory stress, file I/O faults, clock jumps and JVM faults, and chains experiments into workflows. LitmusChaos, also CNCF incubating and Kubernetes-native, declares experiments as custom resources and judges them with resilience probes — HTTP, command, Kubernetes and Prometheus checks whose success is part of the verdict.
• Cloud fault-injection services act on the provider's own resources. AWS Fault Injection Service runs experiments from templates of actions, targets and stop conditions, where a stop condition is a CloudWatch alarm that halts the experiment; AWS recommends a planning phase and pre-production runs before production. Azure Chaos Studio runs service-direct faults through a resource's management API (shutting down a virtual machine, failing over an Azure Cosmos DB account) and agent-based faults inside virtual machines (CPU pressure, process kills); its newer Workspaces with preconfigured outage Scenarios were in public preview when this guide was written.
• A load generator supplies the steady traffic and part of the abort logic: JMeter, or k6 with thresholds on error rate and p95 latency.
• Where they run: fault proxies and mocks on every merge in CI (a handful of fast, deterministic checks); Kubernetes and cloud tools in staging and pre-release sessions; production experiments only with stop conditions, a small blast radius and an agreed share of the error budget. AWS's chaos-engineering guidance adds that an experiment the system has passed should be kept as an automated regression test in the pipeline.
• A resilience testing strategy puts these together in order: requirements written first; fast fault tests in CI on every merge; staging experiments and a game day before each major release or infrastructure change; guarded production experiments only for what has passed; and the whole set repeated on a schedule, because dependencies change even when the code does not.

#!/usr/bin/env bash
# Resilience check on every merge (example). docker compose starts the app with
# PAYMENTS_URL=http://toxiproxy:21212, a payments stub on :8080 and Toxiproxy (API on :8474).
set -euo pipefail
TOXI=http://localhost:8474
APP=http://localhost:3000
STUB=http://localhost:8080   # the example stub counts the calls it receives at /__calls

checkout() {  # prints "<status> <seconds>"
  curl -s --max-time 10 -o /dev/null -w '%{http_code} %{time_total}' -X POST "$APP/api/checkout" -H 'Content-Type: application/json' -d @basket.json
}

curl -sf -X POST "$TOXI/proxies" -d '{"name":"payments","listen":"0.0.0.0:21212","upstream":"payments-stub:8080"}' > /dev/null

# 1. Payments slower than the 3 s client timeout. With this client (timeout 3 s, one retry
#    after up to 200 ms, 5 s checkout deadline) the first call times out, the retry is cut
#    off by the deadline, and checkout must answer with a clear 503 in about 5 s.
curl -sf -X DELETE "$STUB/__calls" > /dev/null
curl -sf -X POST "$TOXI/proxies/payments/toxics" -d '{"name":"slow","type":"latency","attributes":{"latency":4000,"jitter":0}}' > /dev/null
read -r status seconds <<< "$(checkout)"
[ "$status" = "503" ] || { echo "FAIL slow payments: got $status, expected 503"; exit 1; }
awk -v s="$seconds" 'BEGIN { exit !(s < 5.5) }' || { echo "FAIL slow payments: took ${seconds}s, deadline is 5 s"; exit 1; }
calls=$(curl -sf "$STUB/__calls")
[ "$calls" = "2" ] || { echo "FAIL slow payments: stub saw $calls calls, expected 2 (one retry)"; exit 1; }
echo "ok   slow payments -> $status in ${seconds}s after $calls calls"

# 2. Fault removed: the next checkout succeeds without a restart
curl -sf -X DELETE "$TOXI/proxies/payments/toxics/slow"
read -r status seconds <<< "$(checkout)"
[ "$status" = "201" ] || { echo "FAIL recovery: got $status, expected 201"; exit 1; }
echo "ok   recovered -> $status in ${seconds}s"

curl -sf -X POST "$TOXI/reset" > /dev/null

Try it: API Status Code Simulator Try it: API Health Checker

Worked example: checkout when the payment API slows down and fails

Bustan Market (fictional) wants to know what its checkout does when the payment provider has a bad day. The experiment runs in a staging environment that mirrors production, with k6 sending a steady 20 checkouts per second from test accounts. Payments go through Toxiproxy (for latency) and a stub of the provider (for errors). The relevant settings: payment client timeout 3 s; one retry after up to 200 ms (full jitter), only on timeouts and 503; a 5 s deadline on the whole checkout request; an Idempotency-Key header on every charge; a connection pool of 40; and a circuit breaker that opens when more than 50% of the last 100 calls fail.

Baseline, measured for 15 minutes before any fault: checkout p95 850 ms, success 99.6%, payment call mean 300 ms and p95 420 ms, product pages p95 300 ms.

• Run 1 — payment calls 2 s slower for 20 minutes. Expected: about 2.3 s per call on average and 2.42 s at p95, still under the 3 s timeout, so checkout p95 at most 4 s, errors at most 1%, product pages unchanged. Observed: checkout p95 4.8 s, p99 5.0 s — the deadline — and 3.4% errors; product pages 310 ms. The pool explains it: by Little's law, calls in flight = arrival rate × mean time per call ≈ 20 × 2.3 s ≈ 46, more than the 40 connections, so checkouts queued for a connection, and the 3.4% that waited longest reached the 5 s deadline and failed. Fail. The bulkhead held: product pages were unaffected.
• Fix and re-run — size the pool from the worst case the timeout allows for a single attempt: 20 calls/s × 3 s = 60, rounded up to 64. When calls time out and are retried the attempt rate can double, so no affordable pool covers a full outage; that case is left to the circuit breaker, which opens and stops the calls. Re-run: checkout p95 2.9 s, errors 0.5%. Pass.
• Run 2 — 30% of payment calls return 503 for 20 minutes. Expected: with one retry, a checkout fails only if both attempts fail, so 1 − 0.3 × 0.3 = 91% succeed; the rest get the "Card payment is unavailable - your basket is saved" message, never a 500; payment calls rise by at most 1.3 times (each call has a 30% chance of one retry); no duplicate charges; the breaker stays closed, because 30% is below its 50% threshold. Observed: 90.8% succeeded, 9.2% got the message, no 500s, 1.29 calls per checkout, zero duplicate charges when orders were reconciled with the stub's log of idempotency keys, breaker closed throughout. Pass.
• Detection — the "checkout errors" alert should fire within 5 minutes. It fired after 7 min 40 s: it averaged over 10 minutes, so with user-facing failures at about 9% against a 0.4% baseline, the average passed 5% only after a little over 5 minutes, and the alert then waited 2 more minutes plus its evaluation interval. Fail; the window was shortened to 5 minutes and the pending time to 1 minute, and on the re-run the alert fired at 3 min 50 s.
• Recovery — after the 503s stopped, success returned to baseline in 35 s with no restart. Pass.

Bustan Market (fictional) - checkout -> payment API in staging, k6 at 20 checkouts/s
Baseline (15 min): checkout p95 850 ms, success 99.6%, pages p95 300 ms,
                   payment call mean 300 ms, p95 420 ms
Settings: payment timeout 3 s; 1 retry after up to 200 ms (full jitter); checkout deadline 5 s;
          pool 40; breaker opens at >50% failures in the last 100 calls

Run  Fault (20 min each)       Expected                        Observed                        Result
1    payment calls +2 s        checkout p95 <= 4 s,            p95 4.8 s, errors 3.4%,         FAIL
                               errors <= 1%, pages unchanged   p99 5.0 s (the deadline),
                                                               pool 40/40 busy, pages 310 ms
1b   same, pool 40 -> 64       as run 1                        p95 2.9 s, errors 0.5%          PASS
2    30% of payment calls 503  >= 90% succeed, rest get the    90.8% succeed, 9.2% message,    PASS
                               message, 0 HTTP 500, 0 dupes,   0 HTTP 500, 0 duplicates,
                               calls <= x1.3, breaker closed   x1.29 calls, breaker closed
     - detection               alert within 5 min              alert at 7 min 40 s             FAIL
     - recovery after removal  baseline within 60 s            baseline after 35 s             PASS

Follow-ups: RX-14 pool size = peak calls/s x timeout, in the payment client (done, re-run passed);
                  timeouts with retries can double the attempt rate - left to the breaker
            RX-15 alert window 10 min -> 5 min, pending 2 min -> 1 min (re-run: alert at 3 min 50 s)

Try it: Response Time Analyzer Try it: API Performance Comparator Try it: HTTP Request Benchmark

Common resilience testing mistakes

• Starting in production. Production experiments come last, after the same fault has passed in staging and the abort path has been tried.
• Injecting a fault with no steady state recorded. Without a baseline, "checkout got slower" cannot be measured, and the result is an opinion.
• Changing several things at once. Two faults, a new build and a configuration change in one run leave a failure with four possible causes.
• Testing only total outages. Real dependencies are slow, or fail a fraction of requests, far more often than they are completely down — and a slow dependency is harder on pools and timeouts than a dead one.
• Measuring the servers instead of the users. CPU and pod restarts are diagnostics; the pass criteria are user-facing success rate, latency and correct data.
• No load during the experiment. Many faults only show their effect under concurrency — pool exhaustion, retry storms, growing queues. An idle staging system shrugs off what production would not.
• Not checking the data afterwards. Duplicate charges, half-written orders and lost messages do not appear on a latency graph; reconcile what the system accepted with what it stored.
• Treating the session as the deliverable. A game day without tickets, owners and a re-run date only documents the weakness.
• Running it once. Dependencies, configuration and traffic change; keep the experiments that passed as regression tests and repeat game days on a schedule.

Frequently asked questions

References

• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality modelISO/IEC · 2023The product quality model: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability and flexibility.
• NIST SP 800-34 Rev. 1: Contingency Planning Guide for Federal Information SystemsNIST · 2010Contingency planning for information systems, including the definitions of recovery time objective (RTO) and recovery point objective (RPO) used to set and test recovery targets.
• Site Reliability Engineering: How Google Runs Production SystemsGoogle / O'Reilly Media · 2016Service level objectives, error budgets, monitoring the "four golden signals" and handling overload — the operational side of resilience.
• Monitoring Distributed Systems — Site Reliability Engineering (Rob Ewaschuk)Google / O'Reilly Media · 2016Google's SRE guidance on monitoring: black-box versus white-box monitoring and the four golden signals — latency, traffic, errors and saturation.
• Addressing Cascading Failures — Site Reliability Engineering (Mike Ulrich)Google / O'Reilly Media · 2016How overload and retries turn one failure into a cascade, and the defences to test: randomised exponential backoff, retry limits and budgets, deadline propagation, load shedding and graceful degradation.
• Postmortem Culture: Learning from Failure — Site Reliability Engineering (John Lunney and Sue Lueder)Google / O'Reilly Media · 2016Blameless postmortems: which incidents call for one, what it records — impact, root causes and follow-up actions — and why blame is kept out of it.
• Release It! Second Edition: Design and Deploy Production-Ready Software (Michael T. Nygard)The Pragmatic Bookshelf · 2018Stability patterns — timeouts, circuit breakers, bulkheads, back-pressure — and the failure modes load testing is meant to find.
• Circuit Breaker (Martin Fowler)martinfowler.com · 2014The circuit breaker pattern: closed, open and half-open states, failure thresholds and reset — the behaviour a resilience test has to observe.
• Principles of Chaos Engineeringprinciplesofchaos.org · 2019The definition of chaos engineering and its experiment method: a measurable steady state, a hypothesis, real-world fault variables and a minimised blast radius.
• REL12-BP04 Test resiliency using chaos engineering — AWS Well-Architected Framework, Reliability PillarAmazon Web ServicesRunning fault-injection experiments: steady state, hypothesis, guardrails and stop conditions, pre-production before production, and keeping passed experiments as regression tests in CI/CD.
• REL12-BP05 Conduct game days regularly — AWS Well-Architected Framework, Reliability PillarAmazon Web ServicesGame days: simulating events in production-like environments to test systems, processes and team responses, with the teams who handle real events, followed by a review whose lessons are tracked to completion.
• What is AWS Fault Injection Service? — AWS FIS User GuideAmazon Web ServicesAWS's managed fault-injection service: experiment templates made of actions, targets and stop conditions, where a stop condition is a CloudWatch alarm that halts the experiment.
• What is Azure Chaos Studio?MicrosoftMicrosoft's managed chaos-engineering service for Azure: service-direct and agent-based faults, and Workspaces with preconfigured outage Scenarios (in public preview).
• Basic Features — Chaos Mesh documentationChaos Mesh (CNCF)The fault types Chaos Mesh injects on Kubernetes — pod, network, DNS, HTTP, stress, file I/O, time, kernel and JVM faults — and its chaos workflows and dashboard.
• What is Litmus — LitmusChaos documentationLitmusChaos (CNCF)LitmusChaos, a CNCF incubating chaos engineering framework that declares chaos experiments as Kubernetes custom resources.
• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• Grafana k6 documentationGrafana LabsOfficial documentation for Grafana k6, an open-source and extensible load and performance testing tool.
• RFC 9110: HTTP SemanticsIETF · 2022The meaning of HTTP methods, header fields, status codes and content negotiation, independent of the wire version.
• RFC 6585: Additional HTTP Status CodesIETF · 2012Defines 428, 429 Too Many Requests, 431 and 511, including the Retry-After header for rate limiting.
• RFC 7519: JSON Web Token (JWT)IETF · 2015Defines the JWT format — header, payload, signature — and the registered claims iss, sub, aud, exp, nbf, iat and jti.