Soak and Endurance Testing

Why it matters

Short tests measure a system that has just started: pools are clean, logs are small and every leak is still tiny. In production the same service may run for days or weeks between deployments. A defect that leaks a few megabytes per hour, or keeps one database connection after each timed-out payment, is invisible in a 30-minute run and becomes an outage on day four — often "fixed" by a restart that hides the cause until it happens again.

For the product, a soak test is the evidence that performance and reliability hold for the real operating period, not only for the first minutes; in ISO/IEC 25010 terms it examines resource utilisation and time behaviour over time. For the tester it changes the question from "is it fast?" to "does it stay fast, and does it stay up?" — and a leak found in a test environment, with full monitoring and no customers affected, is far easier to diagnose than the same leak in production.

What makes a soak test different

In the previous lessons each test type changed the load: a load test applies the expected workload, a stress test goes past the expected peak, and a spike test makes the load jump suddenly. A soak test keeps the load steady and makes the duration the variable. The result you look for is therefore a trend: does anything keep rising or falling while the load stays flat?

Soak testing and endurance testing name the same activity. ISTQB uses endurance testing; many tools and teams, including the k6 documentation, say soak testing. Some teams draw their own line between the two, for example by load level, but there is no standard split, so state the meaning in your test plan.

• Load: usually the expected average production load, or a realistic busy-period load, held constant. Stay clearly below the breaking point found in stress testing, or you cannot tell overload from a slow leak.
• Duration: taken from the operational context — at least one full business cycle, such as a day with its nightly batch jobs, or the longest time the service runs between restarts or deployments.
• Question: does the system behave in hour 12 as it did in hour 1, and does it give its resources back when the load stops?

What degrades over time: leaks, exhaustion and drift

A memory leak happens when a program keeps holding memory it no longer needs, so that memory is never released. Many runtimes (Java, .NET, JavaScript, Go) free memory automatically: the garbage collector (GC) regularly removes from the heap — the memory area where a program keeps its objects — every object that nothing refers to any more. In these runtimes a leak therefore means the program still holds references to objects it no longer uses: an in-memory cache without a size limit, a session list that is never cleaned, a listener added on every request and never removed. Memory rises and falls in a sawtooth as the collector runs. The signal is the level after each collection: in a healthy service it settles after warm-up; with a leak it keeps climbing until the process runs out of memory or the platform kills it. Before that, the collector works harder and response times grow.

Resource exhaustion is the same pattern with a resource that has a fixed limit. A code path that does not return its database connection to the pool when an error occurs leaks one connection each time it runs. When the pool is empty, new requests wait and then time out, although CPU and memory look normal.

• Memory: process or heap memory after garbage collection rising hour after hour.
• Pools and handles: database connections, threads, open files or sockets in use creeping towards their configured maximum.
• Disk: logs, temporary files or uploads filling the disk; without log rotation the service stops when the disk is full.
• Queues: a consumer slightly slower than its producer, so the backlog and the waiting time grow without end.
• Degradation without a leak: tables that grow during the run slow down unindexed queries, and scheduled jobs such as backups compete with traffic at a fixed hour. These are real findings too, but the fix differs, so name them precisely.

Planning and running a soak test

A soak test without server-side monitoring only tells you that the system got slower. Collect, at a fixed interval and on the same timeline as the response times, the four golden signals from Google's SRE book — latency, traffic, errors and saturation — plus every resource that can leak: memory after garbage collection, connections and threads in use, open files, free disk and queue depth.

Write the pass criteria before the run, and compare time windows instead of averaging. A 12-hour average hides a trend; compare the first steady hour with the last one, and check short windows for errors.

• Use the realistic workload mix and enough distinct test data: 20 accounts reused for 12 hours keep everything in cache and hide growth, and unique values such as order numbers must not run out mid-run.
• Keep the environment production-like — the same memory limits, log level and scheduled jobs — and agree that nobody deploys or restarts anything during the run.
• Watch the load generator too: if it runs short of memory or ports, the system under test looks slow when it is not.
• After the ramp-down, keep observing: memory, connections and threads should return close to their starting level.

Soak test plan: order API
Workload:  180 requests/s (expected weekday average), held constant
           60% browse, 25% search, 10% add to cart, 5% checkout
           5,000 distinct test users
Duration:  15 min ramp-up, 12 h steady state, 10 min ramp-down,
           then 30 min observation with no load
Monitor:   every minute: p95 and p99 per endpoint, throughput, errors,
           heap after GC, DB connections in use (pool max 50),
           threads, open files, free disk, queue depth
Pass if:   1. p95 in the last hour <= 1.2 x p95 in the first steady hour
           2. error rate < 0.5% in every 15-minute window
           3. heap after GC: no upward trend over the last 6 hours
           4. no restarts and no out-of-memory kills
           5. after ramp-down: heap and connections back near baseline

Try it: Performance Test Data Generator Try it: Load Test Data Generator Try it: Concurrent Request Test Generator

Worked example: reading a 12-hour run

The order API from the plan ran for 12 hours at a steady 180 requests per second. The table below shows selected hours that the tester exported from the monitoring dashboard.

Read it column by column. Throughput is flat, so the slowdown is not caused by more traffic. Connections stay between 12 and 14 of 50, so the pool is not the problem. Heap after garbage collection (GC) rises by about 0.24 GB per hour and never levels off: a memory leak. p95 grows with it because the collector runs longer and more often as the heap fills, and errors appear in the last hours. The run fails criteria 1 to 4; criterion 5 could not be checked because the run never reached ramp-down.

The defect report states facts and the trend, not a guess at the cause: the build, load profile, duration, the heap chart, the time of the kill, and the projection that at average load production would reach the same limit roughly every 11 to 12 hours. Ask the developers to capture two heap dumps in the test environment, in hour 1 and hour 6; comparing them shows which objects keep growing. After the fix, repeat the same profile as confirmation testing and compare the curves.

Hour   Req/s  p95 ms  Errors  Heap after GC  DB conns in use
1        180     310    0.1%         1.2 GB         12 of 50
3        180     325    0.1%         1.7 GB         13 of 50
6        180     410    0.2%         2.4 GB         12 of 50
9        180     690    0.4%         3.1 GB         14 of 50
11       180    1450    1.9%         3.6 GB         13 of 50
11:35    container killed: memory limit (4 GB) exceeded

Frequently asked questions

References

• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• Load test types — Grafana k6 documentationGrafana LabsDefinitions of the common load test types — smoke, average-load, stress, soak, spike and breakpoint — and what each one is meant to reveal.
• Monitoring Distributed Systems — Site Reliability Engineering (Rob Ewaschuk)Google / O'Reilly Media · 2016Google's SRE guidance on monitoring: black-box versus white-box monitoring and the four golden signals — latency, traffic, errors and saturation.
• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality modelISO/IEC · 2023The product quality model: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability and flexibility.