Spike Testing

Why it matters

A stress test, as in the previous lesson, pushes load past the expected peak to find the limit. A spike test asks something else: the peak may be well within what the system handles once it has scaled, but it arrives faster than the system can react. New instances take minutes to start, caches are cold, connection pools are sized for normal traffic, and every request that cannot be served yet waits in a queue or fails.

This is how systems fail on their busiest day — a sale, a ticket release, a registration deadline — when lost orders and duplicate payments from retried requests cost the most. For the tester, a spike test turns "we have autoscaling" into evidence: how long the scaling lag is, what users see during it and whether the system really comes back.

The shape of a spike profile

A spike profile has five phases: a baseline at normal load, a ramp measured in seconds rather than minutes, a short hold at the spike level, a fast drop, and a recovery period at baseline. The recovery period is part of the test: it shows whether the system returns to normal. Take the spike level and its duration from the real event — a campaign's expected audience, last year's sale — not from a round number. Grafana k6's test-type guide draws a spike with no hold or a very brief one; if the real burst lasts minutes, as a flash sale does, hold it for that long.

Drive the load by arrival rate, not by a fixed number of virtual users. In that closed model each virtual user waits for its response before sending the next request, so when the system slows down the tool sends less and hides the very burst you want. The script below uses an arrival-rate executor; if k6 runs out of virtual users to keep the rate, it reports dropped_iterations: the tool fell short, not the system.

import http from 'k6/http';

export const options = {
  scenarios: {
    flash_sale_spike: {
      executor: 'ramping-arrival-rate',
      startRate: 100,          // requests per second (one request per iteration)
      timeUnit: '1s',
      preAllocatedVUs: 500,
      maxVUs: 6000,            // enough to keep the rate when responses slow down
      stages: [
        { duration: '5m', target: 100 },   // 1 baseline
        { duration: '30s', target: 600 },  // 2 spike up
        { duration: '5m', target: 600 },   // 3 hold
        { duration: '30s', target: 100 },  // 4 drop
        { duration: '10m', target: 100 },  // 5 recovery
      ],
    },
  },
};

export default function () {
  http.get('https://staging.example.com/api/checkout/summary');
}

Try it: Performance Test Data Generator Try it: Concurrent Request Test Generator

Autoscaling and the scaling lag

Autoscaling adds instances when a metric crosses a threshold, and every step takes time: the metric is averaged over a window, the rule is evaluated on a schedule, and a new instance must boot, warm up and pass its health check before the load balancer sends it traffic. The sum is the scaling lag. Until it is over, the instances you already had carry the whole burst. A spike test therefore measures the scaling chain, not only the final instance count:

• Time from the start of the spike to the scale-out decision, and from the decision to new instances serving traffic.
• The maximum instance count, and whether the spike reaches it.
• What does not scale with the application: database connections (each new instance opens its own pool), third-party rate limits, a single cache node.
• Scale-in afterwards: the cooldown, and whether the instance count flaps up and down while load hovers near a threshold.

Queues, overload and recovery

When requests arrive faster than they are served, the difference waits in a queue: in the load balancer, a web server, a connection pool or a message broker. At 600 requests per second against a capacity of 400, the backlog grows by 200 every second; after 30 seconds 6,000 requests are waiting, and a new one waits about 15 seconds behind them. When the burst ends, the backlog drains only at the spare capacity, and that drain time is part of the recovery.

A system that handles overload well keeps its queues short and refuses what it cannot serve in time — 503 Service Unavailable, which RFC 9110 defines for a temporary overload and which may carry a Retry-After header, or 429 Too Many Requests (RFC 6585) for a per-client limit — instead of letting requests wait until clients time out. Retries make a spike worse: clients that retry at once multiply the load, so check that they use exponential backoff with random jitter, and model that behaviour in your script.

• Every minute of the spike: error rate and error type (a fast 503 is a designed answer; 504 timeouts and 500 errors are not), p95 and p99 latency, and whether checkout keeps working while less important requests are refused.
• Message queues: peak depth and drain time, with nothing lost or processed twice, and no duplicate orders where a client retried a request that had in fact succeeded.
• Recovery: the time until latency, errors, queue depth and instance count are back at baseline, with nothing left stuck.

Try it: API Status Code Simulator Try it: HTTP Status Code Lookup

Worked example: a flash sale at 20:00

Staging is sized like production: 4 instances of about 100 requests per second each, behind a gateway that times out after 10 seconds. Every minute the autoscaler checks the average CPU of the last minute; above 70% it adds 4 instances, up to a maximum of 8, which need about 2 minutes to be ready, and it removes them after 5 minutes below 30%. Each instance holds 20 database connections; the database accepts 150. The tester runs the profile from the script above through the checkout journey, which places test orders, with 5,000 distinct test users, and summarises the per-minute results by phase:

• Scaling lag: the spike began at 05:00 and new instances served traffic at 08:00. Meanwhile about 600 requests per second arrived for a capacity of 400: the backlog grew until requests hit the gateway's 10-second timeout, p95 sat at 10 s and from 06:00 a third of requests failed. The sale time is known, so the team can scale out before it starts.
• Hidden limit: 8 × 20 = 160 connections, but the database accepts 150, so scaling out caused new errors. Report it as a defect with that arithmetic.
• No load shedding: requests waited for the gateway's 10-second timeout instead of being refused at once with 503 and Retry-After. With 600 arriving and 400 served, a third could not be served either way; refusing them at once would have kept the queue short and the other two thirds fast.
• Recovery: the backlog was gone about 30 seconds after the new instances arrived, and latency was at baseline as soon as the load dropped, but the order-confirmation email queue, whose workers do not scale, peaked at 9,000 messages and emptied only at 17:00, six minutes after the drop. Ask whether that delay is acceptable, and rerun the same profile after each fix.

Spike test, run 1 (staging sized like production)

time   arrival     instances  p95     errors  observation
00-05  100/s       4          220 ms  0%      baseline
05-06  up to 600   4          10 s    5%      CPU 100%, backlog builds, first 504 at 05:54
06:00                                         scale-out triggered: +4 instances
06-08  600/s       4          10 s    33%     504 from the gateway (10 s timeout)
08:00                                         4 new instances in service (maximum 8)
08-09  600/s       8          8.9 s   6%      backlog drains in about 30 s
09-11  600 to 100  8          450 ms  5%      DB: too many connections (160 > 150)
11-16  100/s       8          210 ms  0%      API at baseline; email queue peaks at 9,000
16-21  100/s       4          230 ms  0%      scale-in 16:00; email queue empty at 17:00

Try it: Load Test Data Generator

Frequently asked questions

References

• Certified Tester Performance Testing (CT-PT)ISTQB · 2018ISTQB's specialist syllabus on performance testing: basic concepts, performance measurements, test activities and tasks, and tools.
• Load test types — Grafana k6 documentationGrafana LabsDefinitions of the common load test types — smoke, average-load, stress, soak, spike and breakpoint — and what each one is meant to reveal.
• Site Reliability Engineering: How Google Runs Production SystemsGoogle / O'Reilly Media · 2016Service level objectives, error budgets, monitoring the "four golden signals" and handling overload — the operational side of resilience.
• RFC 9110: HTTP SemanticsIETF · 2022The meaning of HTTP methods, header fields, status codes and content negotiation, independent of the wire version.