Performance & Security Resilience Testing

The draft plan, item by item. 1: "fast" and "must not crash" are not targets; a test needs a number before it runs, and T1–T3 already give percentiles, a load and an error rate. 2: one endpoint in a loop is not the workload model, which here is 450 req/s with the agreed endpoint mix. "200 virtual users" says little on its own, because a virtual user waits a think time between requests: state the load in req/s and size the virtual users until the tool reaches it. 3: load tests do not run on production, at 03:00 or at any other hour: they degrade the service for real customers and can trip fraud or rate-limit controls, including the payment gateway’s. Use staging-perf; production gets only read-only synthetic checks as monitoring. 4: the average hides the slow tail; record p50, p95 and p99, and the error rate and the error type separately. 5: copying 50,000 real customers breaks the Personal Data Protection Law and the guide’s rule; generate synthetic data of the right shape (names, Saudi mobile numbers, many different accounts and seat choices, some invalid requests). 6: a 15-minute soak test sees nothing a load test does not; the sale lasts four hours, so the soak test runs at 300 req/s for at least four hours. 7: a scanner finds known patterns but not broken access control, and the night before the sale leaves no time to fix anything; it belongs in the pipeline now, next to the manual baseline checks.

Missing altogether: a spike test, although the push notification takes the platform from about 40 to 450 req/s within seconds; a stress test to find the margin; failure injection on the payment gateway and the SMS provider; a caching check; and a rule for keeping results. A one-page plan for the sale. Targets: T1–T3 with the endpoint mix at 450 req/s and at 300 req/s, for the whole sale from 20:00 to 24:00, written before testing. Workload model: the forecast mix, synthetic users and tokens, seat holds spread over the stadium with competition for the same seats, and a share of invalid requests. Runs on staging-perf: load at 450 req/s for 30 minutes; spike from 40 to 450 req/s in under a minute; stress from 450 req/s up to the breaking point; soak at 300 req/s for at least four hours. Failure injection: the payment gateway and the SMS provider each made slow, wrong (malformed body, 500, 503, 429 with Retry-After) and unavailable, with recovery observed. OWASP baseline: access control with a valid token of the wrong user and of the wrong role, injection, sign-in and sessions, headers and cookies, tokens, CORS, verbose errors, personal data in logs. Caching: Cache-Control and ETag on the heaviest responses (event pages, images, the seat layout), checked against how fresh each one must be. Records: the workload model, the environment description and the raw results of every run; SLO alerts set from T1–T3; the suite run again on every release that touches infrastructure, authentication or the sale path.

R1: pass. Seat map p95 620 ≤ 800 and p99 1,350 ≤ 1,500 (T1), pay p95 2,400 ≤ 3,000 (T2), errors 0.04 % < 0.1 % (T3): the platform keeps its promise at a steady peak. R2: fail on all three targets. Seat map p95 2,900 and p99 7,800 against 800 and 1,500, pay p95 5,100 against 3,000, errors 2.1 % against 0.1 %. T1–T3 apply from the first minute, and the notification turns the first minutes into a spike, so R2 is what the opening of the sale will look like: it blocks the launch. From minute 6 the values match R1, so the platform catches up, but too late for the buyers who come first. The draft’s criterion would have passed this run: the average of 520 ms is under 1 second while one request in twenty takes 2.9 seconds or more.

R3: pass in hour 1, fail in hour 4. Seat map p95 1,450 and p99 3,900, pay p95 4,800 and errors 0.9 % all break their targets although the load did not change, and the database connections in use climb from 60 to the pool maximum of 200. That is the pattern of connections that are taken and never given back, which only a test that lasts hours shows. It blocks the launch too, because the sale lasts four hours; the draft’s 15-minute soak and its average criterion (410 ms in hour 4) would both have missed it. R4: not judged against T1–T3, because the load is above the target; a stress test answers two other questions. Where is the limit? Errors stay below 0.1 % up to 800 req/s and jump at 850 req/s, so the limit lies between the two; record the last step that held, 800 req/s, about 1.8 times the forecast peak, as the margin. How does it fail? Badly: one step of 50 req/s takes the errors from below 0.1 % to 41 % although the load grew by only about 6 %, so it fails all at once instead of slowly and safely. The good half is that it recovers by itself within 2 minutes, without a restart. The p95 of 95 ms is not good performance either: it is low only because 41 % of the answers are fast errors, and a system that stays fast by returning errors has not passed. A generic HTML page is also the wrong answer for an API; expected: a clear 503 that the app can show, with a Retry-After header (RFC 9110 allows it on 503), and errors that grow with the excess load instead of jumping.

F1: acceptable. A malformed answer produces a specific, logged error, a clear message and an unpaid order. F2: fail. The payment service waits up to 60 s for the gateway, longer than its own caller, the API gateway, waits (20 s), so the customer sees an error while the payment goes through 11 seconds later, and may pay a second time. Expected: an outbound timeout well under 20 s; when the outcome is unknown, the order says that the payment is being confirmed and the app checks the order status instead of offering to pay again. F3: fail, and it blocks the launch. A non-idempotent request is retried blindly, three times at once with no backoff; with the real gateway, a charge that went through but whose answer was lost or late would be charged again. Expected: no automatic retry of POST /charges, or a bounded retry with backoff only with an idempotency key that the gateway supports. F4: fail. Ten retries within a second ignore Retry-After: 30; RFC 6585 defines 429 and lets the server say when to come back, and retrying at once adds load to a gateway that has just asked for less. The app then gets a generic 500. Expected: no new request before the Retry-After time, and a specific error the app can show, for example that payments are busy and the customer can try again in 30 seconds. F5: fail, because recovery is not automatic. Expected: when the gateway is back, payments resume without a restart, which is what a circuit breaker that tries the dependency again after a pause provides. Not tested yet: the same checks for the SMS provider; without the code, nobody can sign in.

S1: X-Content-Type-Options: nosniff and the cookie’s Secure attribute are correct. Missing (configuration): Strict-Transport-Security, Content-Security-Policy and frame protection (X-Frame-Options or the CSP directive frame-ancestors). The session cookie has no HttpOnly, so any script on the page can read it, including one injected through a cross-site scripting (XSS) defect, and no SameSite (sessions); set both explicitly instead of relying on browser defaults, which differ. S2 (authentication and sessions, token lifetime): exp - iat is 2,592,000 seconds, that is 30 days, from 13 October to 12 November 2026, 07:00 UTC. A leaked access token works for a month; expected: a short lifetime agreed with the team, in minutes, renewed with a refresh token. There is no aud claim, so no API can check that the token was issued for it. The token cannot show whether the server pins the algorithm: test that on staging with a token whose header says none and that has no signature; expected 401.

S3 (access control): broken object-level authorisation, API1:2023 in the OWASP API Security Top 10 and broken access control in the OWASP Top 10. A customer reads another customer’s personal data by changing an id. Severity Critical; it blocks the launch. Expected: 403 or 404 for an order that is not the caller’s. Repeat the check on every endpoint that takes an id (orders, tickets, seat holds) and with the wrong role, a customer token on staff endpoints; a scanner would not have found it. S4 (CORS): the API reflects any origin and allows credentials, so the origin policy does nothing. It is worse than a wildcard: browsers refuse * together with credentials, but they accept a reflected origin. Expected: an allow-list of the platform’s own origins, and no Access-Control-Allow-Origin for any other origin. S5 (configuration): verbose errors are on, and an exception with a file path tells anyone how the system is built. Expected: a generic error body with a correlation id, and the details only in the log. It is also a clue for R2 and R3: the database connection pool runs out under the spike and drains during the soak. Not covered yet: injection on every input that reaches the database; limits on SMS-code attempts and resends; logout that really invalidates the session and the token; the TLS configuration; personal data in logs.

The note to the release manager. No-go for the sale as tested. Must be fixed and re-tested before 22 October: R2 (the opening minutes break every target), R3 (connections run out by hour 4), F2 and F3 (customers can be charged twice), F5 (no recovery without a restart) and S3 (customers can read each other’s orders). Also to fix before the sale, each with a quick re-check: F4, S1, S2, S4, S5 and the overload response seen in R4. Still to run before the sale: failure injection on the SMS provider and the baseline checks not covered yet. After the fixes: confirmation testing of every failed item, including the full four-hour soak; R1 again as a regression test with the same workload, because the fixes touch the sale path and the comparison must be honest; and R4 again to record the new margin. Keep the workload model, the description of staging-perf and the raw results of R1–R4 with the note, set the alerts for the sale evening on T1–T3, and, if the platform is in scope of the NCA’s Essential Cybersecurity Controls (ECC), file the report as evidence for the related controls.

To check your work: R1 is the only run that passes and R4 the only one not judged against the targets; among F1–F5 only F1 is acceptable, and in S1–S5 only the nosniff header and the cookie’s Secure attribute are correct. If you passed R2 or R3 because the average stays under a second, reread the guide’s bullet on percentiles; if you read the 95 ms of R4 as good news, reread the bullet on systems that stay fast by returning errors. Another split between blockers and the other fixes is fine when each decision has its reason; a verdict without a number or a rule behind it is not.