Analysing Performance Results

Why it matters

A run produces thousands of numbers, and a decision has to come out of them: launch or wait, buy a larger database or add application servers, fix a query or change a pool size. Read the wrong number and the team spends weeks scaling a tier that was never the limit, as the plateau in the previous lesson showed. Read the right one and a one-line configuration change can double capacity.

For the tester, analysis turns a performance test into a finding instead of a chart. "The system is slow at 60 requests per second" cannot be assigned to anyone. "From 60 requests per second the database connection pool saturates while database CPU stays below 50%; requests queue for a connection and 0.6% fail" can be assigned, fixed and re-tested.

Check the run, then read the curves together

First check that the run deserves interpretation. Did the tool reach the planned load with its own CPU and network well below their limits? Were ramp-up and ramp-down excluded? Were timeouts and connection errors counted as errors, as agreed? Did build, environment and data volume match the plan? A second run that does not reproduce the first is noise: fix the environment before judging the code.

Then read the curves over time, not the summary table. Load tools such as k6 and JMeter can write per-request results with timestamps, not only an end-of-run summary; use that output. Put throughput, p50, p95 and p99, error rate and the main resources on one time axis and look for four signatures.

• Response time rises while throughput flattens: the knee of the curve. A resource is saturated and requests queue for it. The knee marks the most this workload can get through; capacity, as in the previous lesson, is the last step at or before it at which every objective still holds.
• Errors rise before response time: a hard limit, not slowness. Something of fixed size (a connection pool, a thread pool, a connection or rate limit, a queue) refuses or times out the surplus with HTTP 503, 429 or timeouts while the CPUs stay well below their ceiling.
• p99 far above p95: most requests are fine and a few hit something slow, such as garbage-collection pauses, lock contention, a cache-miss path or a slow downstream call.
• Slow degradation at constant load: a leak or exhaustion of memory, connections, file handles or disk, as in the soak testing lesson.

Try it: HTTP Request Benchmark Try it: Response Time Analyzer

Find the bottleneck: CPU, memory, database, network

The bottleneck is the resource that saturates first at the moment the curve bends; everything after it is a consequence. When the database connection pool is full, application threads wait, requests queue at the load balancer and the client sees timeouts. Diagnose in a fixed order, cheapest check first, and stop at the first resource that is saturated at the knee. Then check that it is a cause, not a consequence: a CPU busy with garbage collection points to memory, and one busy with retries points to the failure behind them.

• Hard limits: errors while the CPUs stay well below their ceiling. Find the pool, thread count, connection limit, queue or rate limit at 100% in use. The fix is a setting or a design change, such as releasing connections sooner, not hardware.
• CPU: an application or database CPU sustained near its ceiling at the knee (the saturated tiers in the previous lesson ran at about 90% or more). Application CPU answers to profiling and scaling out; database CPU usually means expensive or too many queries, and adding application servers does not help and can make it worse.
• Memory: a heap that grows and never shrinks, garbage collection taking a rising share of time, swapping, or restarts after out-of-memory kills. GC pauses appear as p99 spikes, swapping slows everything at once, and restarts show as bursts of connection errors with memory falling to zero.
• Database and dependency waits: application threads idle while requests are slow, slow-query log entries, lock waits, I/O wait, or a third-party API answering in seconds. A trace of one slow request shows which call holds the time.
• Network and payload: bytes per second near the link capacity, large or uncompressed responses, many round trips per page, or retransmits. At 100 requests per second, 2 MB responses need 1.6 Gbit/s, more than a 1 Gbit/s link carries however fast the server is; check the size before blaming the server.
• Nothing saturated: suspect a single lock or queue that serialises the work, a limit you have not graphed, or an unrealistic target. Add monitoring for the missing layer and re-run instead of guessing.

Try it: Response Size Analyzer

Correlate the run with monitoring

The load tool is black-box monitoring: it sees symptoms from outside. Server monitoring is white-box: it sees causes inside. The four golden signals of the Google SRE book, latency, traffic, errors and saturation, name what has to be lined up: the tool gives the first three, the servers give saturation, and correlation works only when both share one timeline.

• One timeline: synchronise clocks and time zones, write down the clock time of the start, the end and every load step, and mark the steps on the dashboards. JMeter's CSV results, for example, give each sample's time in milliseconds since 1 January 1970 (UTC); convert it to the dashboard's time zone. Compare only steady-state windows, never one that includes ramp-up.
• Walk the layers in request order: load balancer (active connections, 5xx), application (requests and latency per endpoint, thread and connection pools, GC, heap), database (connections, slow queries, lock waits, CPU, I/O), infrastructure (CPU, memory, disk, network bytes), then logs and traces for single slow requests.
• A metric that moves when the curve bends is a hypothesis, not proof. Change one thing (raise the pool, add the index, scale out), re-run and check whether the knee moved. If it did not, the bottleneck is elsewhere; if it did, the next one is now visible.

Try it: Timestamp Converter

Worked example: a checkout run that missed its target

An online store ran a stepped load test against POST /api/checkout in staging, ten minutes per step, with the target p95 under 800 ms and errors under 0.5% at 60 requests per second. The load generator stayed below 30% CPU. The table joins the tool's results with the dashboards for the steady state of each step; Offered is the arrival rate the tool applied, Completed is the throughput.

• Step 3 misses the error target (0.6%) although p95 just passes, and p99 at 2,100 ms shows a tail. The curve bends here: step 4 offers 20 req/s more but completes only 3 req/s more.
• Neither CPU is saturated (application 64%, database 48%), but the database connection pool is at 10 of 10 from step 3 and the errors are 503s and timeouts: a hard limit. The slow-query log has nothing over 100 ms, so the database is not slow; connections are scarce. A trace of one checkout shows the transaction stays open during the payment-provider call (about 120 ms of the 180 ms a checkout takes at low load), so each request holds a connection for most of its life. Little's law confirms it: 10 connections each held for about 0.15 s serve at most about 65 requests per second (10 ÷ 0.15), close to the 61 completed at step 4.
• Finding: from 60 req/s the checkout service's database connection pool (10) saturates while database CPU stays under 50%; requests queue for a connection and 0.6% fail with 503 or a timeout. Suspected cause: the connection is held across the payment-provider call. Next experiment: release it before the external call, or raise the pool to 20 if the database's connection limit allows, then re-run steps 3 and 4.

POST /api/checkout · staging · build 4.12.0 · 10 min per step · steady state only
Target at 60 req/s: p95 < 800 ms, errors < 0.5 %

Step  Offered  Completed   p50     p95     p99   Errors  App CPU  DB CPU  DB pool   Heap
      req/s    req/s        ms      ms      ms
  1     20       20        180     310     420   0.0 %    28 %     22 %     3/10   1.1 GB
  2     40       40        190     350     520   0.0 %    51 %     41 %     6/10   1.2 GB
  3     60       58        240     790   2,100   0.6 %    64 %     48 %    10/10   1.2 GB
  4     80       61        610   3,400   5,000   7.8 %    66 %     49 %    10/10   1.2 GB

5,000 ms = client timeout. Errors: HTTP 503 and timeouts. Slow-query log: none > 100 ms.

Report a finding someone can act on

A performance report answers three questions for three audiences: were the objectives met (product owner, release manager), what limits the system and why (developers, operations), and what should change before the next run (everyone). The ISTQB Performance Testing syllabus treats analysing results and reporting as a task of its own after execution. State the verdict first and the evidence after it, and keep the raw results so the next run is compared like for like.

• Verdict per objective: target, measured value, pass or fail, at which load: "p95 790 ms (target 800) pass; errors 0.6% (target 0.5%) fail, at 60 req/s".
• Context that makes numbers comparable: build, environment and how it differs from production, data volume, workload model, load profile, dates, tool and version, and what counted as an error.
• Findings written like defect reports: observed versus expected with exact numbers, the saturated resource and its correlated evidence, reproducibility over two runs, impact at the forecast load, and a severity. One finding per bottleneck.
• Recommendations as experiments: what to change, which metric should move, and the re-run that confirms it. Add the limits of the test, so that nobody reads a staging number as a production promise.
• Two formats from one analysis: a short plain-words summary of what users would experience, and the tables and annotated graphs for engineers.

Try it: API Performance Comparator

Frequently asked questions

References

• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• Monitoring Distributed Systems — Site Reliability Engineering (Rob Ewaschuk)Google / O'Reilly Media · 2016Google's SRE guidance on monitoring: black-box versus white-box monitoring and the four golden signals — latency, traffic, errors and saturation.
• Grafana k6 documentationGrafana LabsOfficial documentation for Grafana k6, an open-source and extensible load and performance testing tool.
• Apache JMeter User's ManualApache Software FoundationHow to build, parameterise and run load tests with JMeter, and how to read its listeners and reports.