What is Performance Testing?

Test types. Product owner, "fast" and "must not crash on day one": a load test at the expected peak, 100 requests per second with the peak journey mix, to confirm the response-time and error targets; then a stress test that keeps raising the load past the peak to find the limit and see whether the system fails gracefully (slower answers, a quick "try again later" response such as a 503) or catastrophically (crashes, lost or duplicate registrations). Registrar, "unusable for the first hour": a spike test, a jump from almost no traffic to the peak and beyond within a minute, as at 08:00, to see queueing and recovery. At 08:00 students refresh instead of thinking, so the same students send more requests: model the spike with a shorter think time or a higher request rate. Operations lead, "slower and slower": a soak (endurance) test at an ordinary day’s load for many hours, ideally days, watching memory, connections and disk for a leak. Architect: a scalability test, the same load steps with one and then two application servers; only the comparison shows whether throughput rises with the second server. Developer: a benchmark, a quick, repeatable measurement of the seat-availability endpoint on the old and the new build with the same data and environment, comparing p50 and p95. A sensible order: the single-user baseline and the benchmark first, then the load test, then stress and spike, then the scalability comparison and the soak, and the load test again after every fix.

Measurable goals. These are proposals to agree with the product owner; once agreed, they are the exit criteria of the performance tests. Peak: with the peak mix (search courses 60%, view a section 25%, register for a section 10%, log in 5%) at 100 requests per second, held for 30 minutes after a 10-minute ramp-up on production-sized staging, p95 is under 800 ms and p99 under 2 s for each journey, fewer than 0.1% of requests fail, and no registration is lost or duplicated. Headroom: the same targets hold at 150 requests per second (step 3 explains why 50%). Opening spike: from almost no traffic to 150 requests per second within one minute, held for 15 minutes, nothing crashes and no registration is lost or duplicated; requests the system cannot serve get a quick, clear answer instead of hanging; and p95 is back under 800 ms within 5 minutes after the load falls to 100 requests per second. Three weeks: at 15 requests per second for at least 48 hours, p95 at the end is no more than 10% above p95 in the first hour, memory and database connections stay flat, and nothing needs a restart. The old system degraded over days, so a shorter soak proves little unless you watch these trends. Your numbers may differ, but every goal must name the journey, the load, the percentile, the error rate, the duration and the environment; "fast" and "must not crash" name none of them.

Virtual users. Each virtual user waits for a response and then thinks before its next request, so by Little’s law virtual users ≈ requests per second × (average response time + average think time) = 100 × (0.5 + 7.5) = 800. Multiplying by the response time alone (100 × 0.5 = 50) counts only the requests in flight and would generate far too little load. With 20–50% headroom that is 960 to 1,200 virtual users, about 120 to 150 requests per second. Take the upper end here: 100 requests per second is what the overloaded old system handled, not the demand. Requests that were refused before they reached it are not in its logs, and students who gave up came back later. Treat the figure as a lower bound, and ask whether the web server logged refused or timed-out requests at 08:00. If you enter 800 users and a constant think time of 7,500 ms in the Performance Test Data Generator, it estimates about 104 requests per second, slightly more than 100, because it assumes about 200 ms per response instead of 0.5 s.

Draft plan. (1) A load generator on the server under test competes with it for CPU, memory and network, so the system looks slower than it is and the generator itself can become the limit: run k6 on separate machines and watch their CPU too. (2) One account for every user means the same student’s data comes from cache and every registration hits the same student record, so reads look faster than in production and registrations collide in ways no real day produces (a student cannot register twice for one section): generate thousands of distinct test students, ideally as many as will register, as a CSV data set, and spread the registrations across many sections. (3) 1,200 users in the same second is a spike, not a load test, and its failure would tell you nothing about the peak: ramp up over about 10 minutes, hold the steady state for 30 minutes, ramp down, and test the spike separately, with its own goal, once the load test passes. (4) The average hides the slow tail, "under 1 s" was never agreed, and errors are not checked at all: report p50, p95 and p99 per journey with the throughput and the error rate, against the agreed goals. (5) 300 students and 40 sections fit in memory and make every query fast, including the new seat-availability query the developer asked about: use production-sized data, as many students, courses and sections as production has, generated or copied from production with the students’ identities removed. Missing from the plan: a single-user baseline before the load, then the load in steps; the journey mix and the 7.5 s think time; server-side monitoring during the run (application server CPU and memory, database CPU, connection pool, slow queries); a second run to confirm that the result reproduces; and goals agreed with the product owner.

Reading the results. Check the workload first: 800 virtual users produced 102 requests per second, as the formula predicted. At the peak (step 3) the response-time and error targets are met for all requests together: p95 480 ms, p99 950 ms, errors 0.04%. But the table does not show each journey, and with a 10% share the registrations can have a much slower p95 while the overall p95 stays low; and the step lasted 10 minutes, not the 30 the goal asks for. So ask for the p95 per journey, and run the full 30-minute load test, before you call the peak goal met. The headroom goal fails. From step 3 to step 4, 200 more users added only 3 requests per second (102 to 105) where the formula expects about 125, while p50 jumped from 220 to 1,500 ms, p95 to 4,100 ms and errors to 1.2%. Latency rising while throughput flattens means a bottleneck: this build tops out at about 105 requests per second, only 5% above last semester’s logged peak and far below the 150 the headroom goal asks for.

The likely bottleneck is on the database side, not the application server’s CPU. CPU stays around 75% and follows the throughput, while the database connections peak at 47 of 50 in step 3 and are all in use from step 4. When every connection is busy, new requests wait for one, which is why even p50 jumps, and those that wait too long fail; the p99 of step 3, about twice its p95, fits the first requests waiting for a free connection. The client-side numbers cannot tell whether the pool is too small or the queries hold each connection too long, for example the new seat-availability query on production-sized data, so the developers need the database CPU, the slow-query log, lock waits and the time requests wait for a connection. Enlarging the pool is not the fix if the database itself is saturated: more connections then make it slower. The pattern repeats at step 5 and the server metrics agree with it, so it is unlikely to be noise; still, run step 4 again to confirm it reproduces before anyone tunes code. The architect’s question is still open: a second application server adds CPU, which is not the limit, and more database connections, which help only if the database has capacity to spare; the scalability test answers it once the database limit is understood.

What you report today, as a defect against the agreed goals, to the product owner and the developers: "Release candidate 3.2.0-rc1, production-sized staging, peak mix, 7.5 s think time, 10-minute steps. At 100 requests per second the overall p95 is 480 ms with 0.04% errors (per-journey figures requested). Throughput stops at about 105 requests per second: with 1,000 virtual users p95 is 4.1 s and 1.2% of requests fail; with 1,200, 7.9 s and 4.8%. Application server CPU 75%, database connections 50 of 50. The headroom goal of 150 requests per second is not met, so registration day risks a repeat of last semester." Attach the run data and the server graphs. After the fix, run the same stepped test with the same script, data and environment and compare it with this run (confirmation testing); then the 30-minute load test at 100 and at 150 requests per second, the stress test above 150 requests per second, the spike, the scalability comparison and the soak. Running them now would only hit the same limit again.

If your answers differ, check that every goal has a load, a percentile and an error rate, that your virtual users include the think time, and that you read the throughput and resource columns together with the response times. Another budget, such as p95 under 1 s, is fine once it is agreed; a conclusion that ignores the flat throughput or the full connection pool is not.