Concurrent Users and Workload Models

Why it matters

A load test is only as meaningful as the load it applies. Run 1,350 virtual users with no pauses and they send thousands of requests per second instead of 90, so the system may fail a test unrelated to reality. Run the right number of users through the home page only, and caches answer everything while checkout, which writes orders and calls the payment provider, is never loaded.

For the tester, the model is how you challenge requirements and read results: "support 2,000 users" cannot be tested until someone says which users, doing what, at what rate.

Concurrent users, requests in flight and requests per second

Three numbers are often mixed up. Concurrent users have an active session during the same period, whether or not they are waiting for a response right now. Requests in flight, sometimes called concurrent requests, are those the server is processing at one instant. Requests per second is a rate of requests arriving or completing; the completion rate is the throughput, which equals the arrival rate while the system keeps up.

Load tools add a fourth term: a virtual user (VU), one running copy of a test script playing one user. A VU that pauses like a person behaves like one concurrent user; a VU that never pauses produces the traffic of many people. Some teams say "concurrent users" when they mean requests in flight, so ask which one a requirement means.

What a workload model contains

A workload model describes the load a test must apply to resemble production. The ISTQB Performance Testing syllabus calls a pattern of what users do an operational profile; a load profile then states how many virtual users run which operational profiles over a time period, for example a ramp-up followed by a steady state. Build the model from evidence: analytics, access logs, monitoring data or a business forecast.

Decide also how the load is generated. In a closed model a fixed number of virtual users loop through their journeys, each sending its next request only after the previous one finishes, so a slowing system receives fewer requests and the test eases off by itself. In an open model users arrive at a set rate whatever the response time, so a slowing system accumulates work. In k6, constant-vus is closed and constant-arrival-rate open. Use open models for public sites and APIs, closed ones for a fixed population such as 200 call-centre agents.

• Journeys: the user journeys that matter, such as search and checkout, each a sequence of requests.
• Mix: each journey’s share of sessions, for example 50% browsing and 15% checkout.
• Rate: sessions starting per second in the period you test, usually the peak hour.
• Data: enough distinct users, products and search terms that caches behave as in production.

Try it: Performance Test Data Generator Try it: Load Test Data Generator

Think time and pacing

Think time is the pause a real user takes between two steps of a journey: reading a page, typing an address. It sits in the script between requests. Take it from analytics (time on page) and vary it randomly, for example 15 to 24 s instead of a fixed 19.5 s, so virtual users do not move in step and hit the server in waves. JMeter adds it with timers such as the Uniform Random Timer, k6 with sleep().

Pacing is the interval from the start of one iteration (one complete journey) to the start of the next for the same virtual user. With a pacing of 300 s and a 240 s journey, the VU waits 60 s, so it starts one checkout every five minutes. If the system slows and a journey takes longer than 300 s, the planned rate is missed, which is itself a finding.

One virtual user, checkout journey: 12 requests, pacing 300 s
Each step: ~0.5 s response + think time, random 15-24 s (mean 19.5 s)

   time  what the virtual user does
    0 s  request 1: view cart, then think
   20 s  request 2: enter address, then think
    ...  requests 3 to 11, about 20 s apart
  220 s  request 12: confirm payment, then think
  240 s  journey done, wait 60 s for the next pacing slot
  300 s  iteration 2 starts: request 1: view cart

Think time = pause between requests inside one journey
Pacing     = start of one journey to start of the next

Try it: Performance Test Data Generator Try it: Concurrent Request Test Generator

Little’s law: users, rates and time

Little’s law states that in a stable system the average number of items inside equals the average arrival rate multiplied by the average time each item spends inside: L = λ × W. It holds for a shop queue, a server or a whole website, whatever the pattern of arrivals. Keep the units consistent: requests per second with seconds.

Applied at three levels it answers this lesson’s questions, and read backwards it checks a report:

• Server: requests in flight = RPS × average response time: 90 × 0.5 s = 45.
• Users: concurrent users = sessions starting per second × average session duration; for the store in the worked example below, 10 × 135 s = 1,350.
• Script: virtual users = target RPS × (response time + think time per request): 90 × 15 s = 1,350 VUs; without think time, 90 × 0.5 s = 45 VUs send the same 90 RPS. The ISTQB Performance Testing syllabus writes this as throughput = virtual users / (processing time + think time).
• Report check: 200 VUs with 0.4 s responses and 5 s of think time produce about 200 / 5.4 ≈ 37 RPS. A report showing 150 RPS means each VU spent only about 200 / 150 ≈ 1.3 s per request, so most of the planned think time was missing.
• Limits: use averages over a steady period, never p95 or a queue that is still growing.

Try it: Response Time Analyzer Try it: HTTP Request Benchmark

Worked example: from analytics to a workload model

An online store prepares for a sales evening. Last year’s busiest hour had 36,000 sessions, or 10 per second, over four journeys. For each journey, analytics give its share of sessions, the requests per session and the average time per step (response plus think time).

A journey’s RPS = sessions per second × share × requests per session; its concurrent users = its RPS × time per step, which is Little’s law. Totals: 90 RPS and 1,350 concurrent users. Checkout is 15% of sessions but 20% of requests, because a checkout session has the most requests, and 27% of users, because people spend longest on each checkout step; a test without the mix would under-load exactly the sessions and payment calls that matter most.

Peak hour: 36,000 sessions / 3,600 s = 10 new sessions per second

journey           share  requests  s/step  requests per second  concurrent users
browse catalogue   50%       8       15    10 × 0.50 × 8  = 40    40 × 15 =   600
search, compare    30%      10       12    10 × 0.30 × 10 = 30    30 × 12 =   360
checkout           15%      12       20    10 × 0.15 × 12 = 18    18 × 20 =   360
track an order      5%       4       15    10 × 0.05 × 4  =  2     2 × 15 =    30
total             100%   9 avg   15 avg                   = 90              1,350

requests = requests per session; s/step = response + think time per request
avg = weighted: 90 / 10 = 9 requests per session; 1,350 / 90 = 15 s per request
average session duration = 9 × 15 = 135 s; 10 sessions/s × 135 s = 1,350 users
In flight on the server at 0.5 s average response: 90 × 0.5 = 45 requests

Try it: Performance Test Data Generator

Frequently asked questions

References

• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• ISTQB GlossaryISTQBThe standard definitions of testing terms — test case, regression testing, severity, priority, boundary value analysis and the rest.
• Grafana k6 documentationGrafana LabsOfficial documentation for Grafana k6, an open-source and extensible load and performance testing tool.
• Apache JMeter User's ManualApache Software FoundationHow to build, parameterise and run load tests with JMeter, and how to read its listeners and reports.