Stress Testing

Why it matters

Expected peaks are forecasts, and forecasts are sometimes wrong: a campaign spreads further than planned, a competitor's outage sends its customers over, a batch job runs in the busy hour, or one of three servers fails and the other two carry its traffic. The question is not whether the system will ever see more than its expected peak, but what it does when that happens. A system that slows down and rejects the excess cleanly loses a few orders; one that crashes, loses data or needs an hour to come back loses the day.

For the tester, a stress test replaces "it should hold up" with measured facts: the load at which the system stopped serving, the resource that ran out first, the failure mode and the recovery time. Those facts give the product owner the headroom between the expected peak and the limit, tell operations what to watch, and give developers a reproducible failure to fix before production produces it in front of customers.

Beyond the expected peak: the stress profile

A stress test starts where the load test ended. The expected peak is the reference line; the stress profile crosses it on purpose and keeps going until the system breaks or an abort rule stops the run. Because the result is "where and how" rather than "pass or fail", the plan carries the load test's objectives as a reading aid, plus an abort rule that protects the environment, for example "stop when errors exceed 20% for one minute or an instance crashes".

• Steps, not one jump: raise the load in equal steps and hold each one until the metrics settle, for example +25% of the expected peak every five minutes. Each step is a small load test, and the step at which a metric changes behaviour is the finding. A continuous ramp is quicker but blurs where the change happened.
• Confirm the peak first: the first step repeats the expected peak, and its numbers must match the load test. A system that is already unhealthy at the peak has a load-test finding, not a stress-test one.
• Stress by subtraction: keep the expected load and remove resources instead. Take one of three instances out, halve the memory limit, throttle the network bandwidth. This is the second half of the ISTQB definition, and it is exactly what a failed server during the busy hour looks like in production.
• Naming: the Grafana k6 documentation calls a run that keeps climbing until the system fails a breakpoint test, and uses "stress test" for a run held above the average load. ISTQB covers both with stress testing; write the profile you mean into the plan.

Try it: Performance Test Data Generator Try it: Concurrent Request Test Generator

Recognising the breaking point and the failure mode

The breaking point is the load beyond which the system no longer serves what it is sent. It rarely announces itself with a crash. Read the metrics from the performance metrics lesson together, step by step (throughput, response time, error rate and resource use), and look for the moment they stop moving in parallel: the load keeps rising, but the throughput does not.

• Saturation: throughput flattens while the applied load climbs, and the difference queues somewhere. Response time, which grew slowly until then, now climbs steeply, because every request waits behind the ones before it.
• Errors replace responses: gateway timeouts (504), refused connections, 503 Service Unavailable, and finally failures that are not HTTP at all: an instance killed for using too much memory, a database refusing new connections, a disk filled with logs.
• The resource that ran out first: CPU at 100%, a connection pool fully busy, a thread pool exhausted, memory growing until the process is killed. (A pool is a fixed set of reusable database connections or worker threads; when all are busy, new requests wait.) This is the bottleneck, and the fact a developer needs most.
• Failure mode, graceful: the system slows, rejects the excess quickly and explicitly (a 503 with a Retry-After header that tells the client when to try again, a bounded queue, a reduced result instead of an error page) and keeps serving the load it can. Rejecting the excess on purpose is called load shedding. Catastrophic: it hangs, crashes, takes other services down with it (a cascading failure, often amplified by retrying clients), or accepts work that it then loses.
• Record per step: load sent, throughput served, p95 and p99, error rate by type, the saturated resource and the minute of the first symptom. The breaking point is a load level with a story, not a single number.

Try it: HTTP Status Code Lookup

Recovery: what happens when the load drops

A stress test is not finished when something breaks. Its second half drops the load back to a normal level, the expected peak or below, holds it, and watches whether the system returns to its baseline on its own. The load test already checked recovery after a ramp-down from a load the system could carry; here the system is recovering from an overload, which is a different and harder question. Recovery is a measured result: how long until the baseline is back, what was lost on the way, and whether anyone had to intervene.

• Time to recover: minutes from the load drop until p95, error rate and resource use are back to the load-test baseline. A system that stays slow at normal load has not recovered: a queue is still draining, a pool is still leaking, or an instance never came back.
• Automatic or manual: crashed instances restart, circuit breakers (switches that stop calls to a failing dependency for a while) close again and traffic resumes without a restart or a cache flush by hand. Every manual step is a finding, because at two in the morning nobody will take it.
• Backlog: queued work such as emails, notifications, payment confirmations and search indexing is processed once, neither dropped nor duplicated, and the queue reaches zero within an agreed time.
• Data integrity: compare what the system accepted with what it recorded; unique order or user ids in the test data make the two lists easy to match. Orders confirmed to the user without a payment record, half-written rows and duplicates from retried requests are the most expensive stress findings, and no latency graph shows them.

Try it: Response Time Analyzer Try it: Load Test Data Generator

Worked example: an order API stepped past its peak

The pharmacy's order API (release 5.1, three application instances, one database) passed its load test at the expected peak of 200 requests per second with p95 at 620 ms against an objective of 800 ms. The stress run below adds 50 requests per second every five minutes, aborts when errors exceed 20% for a minute or an instance crashes, and then holds 200 requests per second for fifteen minutes. The tester reads it in this order:

• Objective first crossed at 350: p95 at 1,250 ms with the database connection pool at 48 of 50 connections busy. This is the first limit, the kind of number a capacity test measures precisely (see the capacity lesson).
• Breaking point at about 385 requests per second: at the 400 and 450 steps the API serves only 380 to 385, whatever is sent; requests queue for the pool, response times jump and the first timeouts appear. The bottleneck is the database connection pool.
• Failure mode: queued requests hold memory; at 500, instance 2 is killed for using too much memory, the other two inherit its load and errors reach 71%. The abort rule fires at minute 36. Catastrophic rather than graceful: no 503, no load shedding, only gateway timeouts (504) and a crash.
• Recovery: back at 200 requests per second the instance restarts on its own and p95 is at 650 ms within four minutes; the email queue is empty by minute 49. The data check finds 12 orders confirmed without a payment record. Three defect reports follow: no load shedding when the connection pool is full, the memory growth under queueing, and the confirmation sent before the payment was recorded.

Stress test: order API, release 5.1, perf environment (3 app instances, 1 database)
Start:       expected peak 200 req/s (load test passed: p95 620 ms, 0.1 % errors)
Profile:     +50 req/s every 5 min; abort if errors > 20 % for 1 min or an instance
             crashes; then back to 200 req/s for 15 min (recovery)
Objectives:  p95 <= 800 ms, errors < 0.5 % (a reading aid here, not pass criteria)
Columns:     Sent and Served in req/s (Served = successful responses);
             Errors = % of the requests sent

Step  Sent  Served         p95 Errors  Observed
200   200   200         620 ms  0.1 %  baseline, as in the load test
250   250   250         690 ms  0.1 %
300   300   300         780 ms  0.2 %  app CPU 85 %
350   350   350       1,250 ms  0.4 %  DB pool 48/50 busy; p95 past the objective
400   400   380       2,900 ms  3.1 %  pool full, requests queue; first timeouts
450   450   385       4,800 ms   14 %  gateway 504s; throughput flat since 400
500   500    90            n/a   71 %  min 36: instance 2 killed (out of memory)
                                       -> abort, load back to 200 req/s

Recovery at 200 req/s (min 37-52)
  min 37-41   instance 2 restarting; p95 2,100 -> 650 ms; errors 8 % -> 0.1 %
  min 41-52   p95 640 ms, errors 0.1 %, DB pool 31/50; email queue empty at min 49
  data check  12 orders confirmed with no payment record; 0 duplicate orders

Frequently asked questions

References

• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• Load test types — Grafana k6 documentationGrafana LabsDefinitions of the common load test types — smoke, average-load, stress, soak, spike and breakpoint — and what each one is meant to reveal.
• Site Reliability Engineering: How Google Runs Production SystemsGoogle / O'Reilly Media · 2016Service level objectives, error budgets, monitoring the "four golden signals" and handling overload — the operational side of resilience.