Performance Metrics

Why it matters

One number such as "the average response time was 400 ms" can hide a failing system. Errors that return quickly pull the average down, a load tool that sent less traffic than planned makes the system look healthy, and a server already at 95% CPU has no room for the next peak. Decisions such as "can we launch the campaign?" are only as good as the metrics behind them.

For the tester, metrics are the shared language with developers, operations and the product owner. A finding that names the metric, where it was measured, over which period and at what load can be acted on; "the system is slow" cannot.

Response time and the parts it is made of

Response time is the time from the moment a client starts a request until it has received the complete response. It is measured on the client side, so it includes everything on the way: finding the server, opening the connection, the server's own work and the transfer back. Server logs often record a shorter duration that leaves the network out, so label every number with where it was measured.

On a new HTTPS connection it splits into the four parts below. curl prints the moment each part ends with its -w option; the values are seconds counted from the start, so each part is the difference between two neighbours. In the sample, waiting is 0.244 − 0.061 = 0.183 s of the 0.289 s total. When waiting dominates, look at the server; when connection set-up dominates, look at the network, the distance to the server and connection reuse.

• DNS lookup: the host name is resolved to an IP address.
• TCP connect and TLS handshake: the connection is opened and secured. A reused (kept-alive) connection skips both.
• Waiting: the request travels to the server, the server processes it and the first byte of the response travels back. The end of this phase, counted from the start, is the time to first byte (TTFB); k6 uses the same name for the waiting phase alone.
• Content download: the rest of the response arrives. A large or uncompressed body makes this part grow.

fmt='dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect}'
fmt="$fmt ttfb=%{time_starttransfer} total=%{time_total}\n"
curl -s -o /dev/null -w "$fmt" https://shop.example.com/api/cart

# output in seconds from the start, rounded to milliseconds:
# dns=0.012 tcp=0.030 tls=0.061 ttfb=0.244 total=0.289

Try it: Response Time Analyzer Try it: Response Size Analyzer

Latency: say which one you mean

Latency is the most overloaded word in performance work. In networking it is the time data takes to travel, usually quoted as round-trip time (RTT). Some load tools use it for the time until the first part of the response arrives: JMeter's Latency stops there, while its Elapsed time runs until the last part. The four golden signals of the Google SRE book use latency for the whole time it takes to service a request, which this lesson calls response time.

The difference matters in practice. A new HTTPS connection needs about three round trips before the first byte arrives: one for the TCP handshake, one for a TLS 1.3 handshake and one for the request itself. With an RTT of 80 ms that is about 240 ms before the server's own work is counted, and a faster server cannot remove it; connection reuse removes the two set-up round trips, and a server closer to the users shortens all three. Tools also start and stop the clock at different points: k6's http_req_duration is sending plus waiting plus receiving and reports TCP connecting and TLS handshaking as separate metrics, while curl's time_total includes them. Write down which definition every number in a plan or report uses.

Try it: API Performance Comparator

Throughput, error rate and resource utilisation

Response time describes single requests. The next three metrics describe the system under load, and they only make sense when they are read together, over the same time window and at a known load.

• Throughput: completed work per unit of time — requests per second (RPS), transactions per second (TPS) when one user action spans several requests, or bytes per second. It counts what the system finished, not what the load tool tried to send. Load tools such as JMeter count failed requests in their throughput too, so when a target is about successful work, count the successes separately. When load keeps rising and throughput stops rising, the system has reached a limit.
• Error rate: failed requests divided by all requests in the same window, as a percentage. Agree in advance what counts as a failure: 5xx responses, timeouts and connection errors, and, as the SRE book points out, a 200 response with the wrong content; some teams also count, by policy, any response slower than an agreed limit. Report the response time of successful requests separately, because fast errors make a system look quicker than it is.
• Resource utilisation: how busy each resource is — CPU, memory, disk, network and pools such as database connections and worker threads. Utilisation says how much of a resource is used; saturation says whether work is already queueing for it. A pool with 50 of 50 connections in use and requests waiting is saturated, whatever the CPU graph shows.
• The load generator is a resource too: if its own CPU or network is at its limit, the results describe the tool, not the system under test.

Try it: HTTP Request Benchmark

Worked example: reading a ten-minute load run

A team runs ten minutes of steady load against POST /api/orders in staging. The target agreed with the product owner is at least 20 successful orders per second with an error rate below 0.5%. The summary below combines the load tool's results with the server dashboards.

From the summary: the throughput of successful orders is 11,880 / 600 = 19.8 per second, just under the target. The error rate is 120 / 12,000 = 1.0%, twice the limit. The resources explain why: the database CPU still has headroom, but the connection pool is saturated, so requests queue for a connection until they time out or the service answers 503. The finding to report is "connection pool exhausted at 20 orders per second, error rate 1.0%", with these numbers attached — not "the database is slow".

POST /api/orders · staging · 10 minutes of steady load

Duration                   600 s
Requests sent              12,000
Successful (201 Created)   11,880
Failed                     120   (84 timeouts after 5 s, 36 x HTTP 503)
Median response time       210 ms (successful requests only)
Database CPU               55% average
DB connection pool         50 of 50 in use, up to 30 requests waiting

The vocabulary you will meet

• Transaction: a named group of requests measured as one user action, such as "checkout", from its first request to its last response.
• Virtual user (VU): one simulated user in a load tool, running a script in a loop.
• Think time: the pause a real user takes between actions. Leaving it out makes 100 virtual users send far more traffic than 100 real people.
• Concurrency: how many users or requests are active at the same moment. It is not throughput: 100 concurrent users can produce 10 or 1,000 requests per second, depending on response time and think time.
• Percentile (p95): the value that 95% of the measurements are at or below; the median is p50, the middle value. Reports rely on percentiles rather than averages, and percentiles have a lesson of their own.
• SLI and SLO: a service level indicator is a measured metric, such as the share of requests answered within 300 ms; a service level objective is the target set for it.
• Bottleneck and headroom: the bottleneck is the resource that limits throughput first; headroom is the spare capacity left before a resource reaches its limit.

Frequently asked questions

References

• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• Monitoring Distributed Systems — Site Reliability Engineering (Rob Ewaschuk)Google / O'Reilly Media · 2016Google's SRE guidance on monitoring: black-box versus white-box monitoring and the four golden signals — latency, traffic, errors and saturation.
• Grafana k6 documentationGrafana LabsOfficial documentation for Grafana k6, an open-source and extensible load and performance testing tool.
• Apache JMeter User's ManualApache Software FoundationHow to build, parameterise and run load tests with JMeter, and how to read its listeners and reports.