Capacity and Scalability Testing

Why it matters

Capacity decisions are made whether anyone tests or not: how many servers to run during a sale, when to buy a larger database, whether autoscaling is enough. Without measurements they are guesses. Guess low and the system fails at peak in front of real customers; guess high and the company pays for servers that add nothing, which is exactly what happens when the database is the limit.

For the tester the question becomes "how much can it take, what runs out first, and what do we gain by adding more?". Stress testing found where the system breaks and soak testing whether it lasts; capacity and scalability testing find the usable limit below the break and whether the architecture can grow past it.

Capacity: the most load that still meets the objectives

The ISTQB Performance Testing syllabus describes capacity testing as finding how many users and/or transactions a system supports while still meeting its stated performance objectives, which can also be stated as data volumes. ISO/IEC 25010 lists capacity under performance efficiency, with parameters such as concurrent users, transaction throughput and database size. So every capacity test names its dimension.

Capacity is not the breaking point. A system usually keeps answering well past its capacity, only too slowly or with too many errors. Capacity is the last load level at which every objective holds; carrying on until something breaks is stress testing.

• Fix the objectives first, for example p95 under 500 ms and errors under 0.5%. Without them there is only a breaking point.
• Use the workload model from the earlier lessons: a browse-heavy mix and a checkout-heavy mix give different capacities.
• Raise the load in equal steps and hold each step until it is steady, for example 10 minutes, so every step is a small load test.
• Record per step: throughput, p95 and p99, error rate, and CPU, memory, connection and queue usage. The first failing step shows which objective broke and which resource ran out.

Try it: Performance Test Data Generator Try it: Concurrent Request Test Generator

Scalability: scaling up and scaling out

The ISTQB Performance Testing syllabus describes scalability testing as checking whether a system can grow, for example to more users or more stored data, to meet future requirements beyond today's without violating its current performance requirements or failing. ISO/IEC 25010:2023 lists scalability as a sub-characteristic of flexibility. In practice the growth is met by adding resources in one of two ways.

Vertical scaling (scaling up) gives one server more CPU, memory or faster storage: no code change, but there is a ceiling, resizing often needs a restart, and the server stays a single point of failure. Horizontal scaling (scaling out) adds instances, identical copies of the application, behind a load balancer: no fixed ceiling, but instances must be stateless (no user session kept in one instance's memory), and shared parts such as the database or a rate-limited third-party API do not multiply with them.

A scalability test runs the same capacity test at several sizes and changes one thing at a time: 1, 2, 4 and 8 instances, or 4, 8 and 16 vCPU (virtual processor cores), with the same data, mix and objectives. Compare sizes with scaling efficiency, efficiency = C(N) / (N × C(1)), where C(N) is the capacity with N units; 100% means every unit adds its full share.

• Scaling up: check that the software uses the new resources. A fixed thread pool, the set number of worker threads that serve requests at the same time, can leave added cores idle.
• Scaling out: check that sessions survive when a request lands on another instance and that scheduled jobs do not run once per instance.
• Scaling with data: repeat the test at the data volume forecast for next year. A query that is fast on one million rows may not be on twenty million.

Try it: Load Test Data Generator Try it: Large JSON Dataset Generator

Worked example: the order API before a seasonal sale

An online store forecasts a peak of 1,800 requests per second on its order API during the yearly sale. The objective is p95 under 500 ms and errors under 0.5% for the agreed request mix. The test environment mirrors production: 4-vCPU application instances behind a load balancer, one 8-vCPU database server and production-sized data. The team ran the stepped capacity test at 1, 2, 4, 6 and 8 instances.

• Up to 4 instances the API scales out almost linearly: the application tier is the limit.
• From 4 to 8 instances capacity rises by only 80 req/s while database CPU reaches 98% and its connection pool fills: the bottleneck has moved to the database.
• The architecture tops out at about 1,500 req/s, below the forecast, and no number of application instances closes the gap. The finding is "scale or relieve the database", not "add servers".
• Next experiment: give the database more resources or less work, for example by adding a missing index or caching product reads, then repeat the 4, 6 and 8 instance runs.

Objective: p95 < 500 ms and errors < 0.5 % (agreed request mix)

Instances  Capacity (req/s)  Efficiency  Saturated first
    1             400           100 %    app CPU 92 %
    2             780            98 %    app CPU 90 %
    4           1,450            91 %    DB CPU 88 %
    6           1,520            63 %    DB CPU 97 %
    8           1,530            48 %    DB CPU 98 %, DB pool full

Efficiency = capacity / (instances x 400)
Forecast peak for the sale: 1,800 req/s

Reading the scaling curve

Plot capacity against the number of units and compare it with the straight line of perfect scaling. The shape says what to do next.

• Linear: every unit adds its full share. Capacity can be bought with more units while the shared parts keep up.
• Sub-linear: each unit adds a little less, because of coordination and shared resources. Scaling works, but each request costs more.
• Plateau: capacity stops rising, as in the worked example. A shared component is saturated; find it and scale it, because more units only add cost.
• Retrograde: capacity falls as units are added, often because instances compete for a lock or a connection pool. Report it as a defect.

From test results to a capacity plan

Capacity planning decides how much infrastructure is needed, and by when, to serve forecast demand with a safety margin. The Google SRE book names what it requires: a demand forecast covering organic growth and inorganic events such as launches and campaigns, reaching beyond the time needed to add capacity, and regular load testing that relates raw capacity (servers, disks) to service capacity.

Testers supply that second half as a capacity statement: "400 req/s per 4-vCPU instance for the agreed mix, near-linear up to 4 instances; the architecture tops out at about 1,500 req/s because database CPU saturates".

• Headroom: how far below tested capacity production should stay at peak. It is a team decision; with an 80% ceiling, a 1,800 req/s forecast needs 2,250 req/s of tested capacity.
• Spare units: one instance more than the calculation (N+1), so losing one still leaves enough capacity.
• Lead time: minutes to add an autoscaled instance, possibly weeks for a larger database. The slowest resource decides how early the plan must act.
• Production thresholds: the ISTQB syllabus notes that once scalability limits are known, thresholds can be monitored in production, for example an alert when database CPU passes 75%.

Frequently asked questions

References

• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality modelISO/IEC · 2023The product quality model: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability and flexibility.
• Site Reliability Engineering: How Google Runs Production SystemsGoogle / O'Reilly Media · 2016Service level objectives, error budgets, monitoring the "four golden signals" and handling overload — the operational side of resilience.
• Load test types — Grafana k6 documentationGrafana LabsDefinitions of the common load test types — smoke, average-load, stress, soak, spike and breakpoint — and what each one is meant to reveal.