API Testing Approaches Compared

The pattern first. Today the API gets one approach — exploratory requests — and not even real exploration: the same collection replayed by hand before each release. At best that could have caught I3, I7 and, with a large enough test store, I5. Each of the other problems needed an approach the team does not use. The model answer below names, for each problem, the approach most likely to have caught it before release and why the hand-run pass missed it; then what each approach still misses, the mix, and the reply to the team lead.

I1 — contract testing, provider-side. If every response of GET /api/v2/cart is validated in CI against the OpenAPI document, the build fails the moment deliveryFee disappears, provided the schema marks deliveryFee as required; check that it does, because a schema whose fields are all optional accepts the renamed response. The hand-run pass missed it because "delivery_fee": 9.00 looks perfectly reasonable to a person, and the web front end, changed in the same release, showed no problem. The installed apps are why this team also needs consumer-driven contracts: each app version still in use records the fields it reads, and a provider change that breaks one of them fails before it ships. A field that has to be renamed is then sent under both names, or in a new API version, until the old apps are out of use.

I2 — mocking. The gateway’s sandbox approves every payment at once, so neither a hand-run request nor a staging run can produce a gateway that answers after 40 seconds. A mock of the gateway can, together with the other error paths: no answer at all, 503 with a Retry-After header, 429, and a body that is valid JSON but the wrong shape. What the API should do when the outcome of a payment is unknown must be agreed with the team — for example, report the payment as pending and ask the gateway for its status instead of reporting a failure — and the test asserts what matters to the customer: one order, one charge. The test then runs against the mock in CI on every build. Before each release, one run against the real sandbox from staging confirms that the mock still behaves like the gateway.

I3 — exploratory testing, then a scripted test. A contract test passes the second discount, because the response has exactly the right shape: this is wrong business logic in a new endpoint. A tester exploring POST /api/v2/cart/promo with a charter such as “break the once-per-customer rule” tries the code on a second order, after cancelling the first order, from two devices at the same moment and in lower case (welcome50). The hand-run pass replays a fixed collection, and a new rule is exactly what it does not contain: sending requests by hand is not the same as exploring. Each case found becomes a scripted test, so that the rule stays fixed.

I4 — load testing with a realistic data set. Hand-run requests cannot show it: one person sends one request at a time, for test customers with about 30 orders, while the failure needs many customers with up to 2,000 orders each at the same moment. Create test data with production-like order counts, agree the targets with the product owner before the test (a response time and an error rate at the expected peak), and run it early enough to fix what it finds and run it again before 12 Nov — and again after any change to order history, which is a hot path. A load test says nothing about whether the orders listed are the right ones.

I5 — scripted tests that assert on the GraphQL body. A GraphQL server usually answers 200 even when a field fails and puts the failure in the errors array, so a check of the status code — the dashboard’s health check, or a test — stays green. The test asserts that errors is absent and that data.storeOrders is a list with the expected orders, and it runs with test stores on both sides of the point where it broke: 100 and 101 open orders (2-value boundary value analysis). A contract check against the GraphQL schema alone would pass this response: storeOrders came back as null next to the error, which GraphQL does only for a field the schema declares nullable. Fix the health check the same way, and add a store of the largest realistic size to the load test, because a single GraphQL request can be very expensive.

I6 — a webhook test in which the test plays the gateway. “Delivered at least once” means duplicates are normal, not a gateway fault, so the receiver must handle them. The test sends the same signed payment.succeeded event, with the same event id, twice and asserts one paid order and one invoice; a second test sends events out of order, such as payment.failed after payment.succeeded for the same payment, and checks the outcome the team has defined (ask for it if it is not written down). The sandbox cannot be asked to retry at the moment a test needs it, so this is mocking — producing the gateway’s behaviour on demand — run as a scripted test in CI.

I7 — scripted negative tests of the webhook receiver, and a security defect. Three requests must be rejected with the error status the team documents (for example 401) and leave the order unchanged: no X-Signature header, a signature made with the wrong secret, and a correctly signed body changed after signing. The last one is what the HMAC Generator exercise shows: one changed character gives a different signature. The collection contained only valid gateway calls, so the hand-run pass never sent such a request; the code review that found it was static testing, and a scripted test keeps it fixed. Because the API moves money and holds customers’ personal data, add tests from the OWASP API Security Top 10 2023 to the mix, object-level authorisation first: one customer must not be able to read or change another customer’s cart or orders. Test only systems you are authorised to test — here, your team’s staging.

I8 — scripted functional tests in CI. It is a regression: a filter that worked in 2.12 broke in 2.13, in a release whose hand-run pass happened not to include filters. A test that inStock=true returns no product with zero stock, in a suite that runs on every build, would have failed on the change that broke it. A contract test passes it, because a sold-out product in the list has the right shape.

What each approach still misses in this product, and what covers it. Contract testing misses wrong values inside a correct shape — I3, I8 and the null of I5 — so scripted tests assert on business rules. Scripted tests find only what they were written to find, and a new rule like I3 is not among them, so every new or changed endpoint gets an exploratory session. An exploratory session proves nothing once it ends, so each finding becomes a scripted test. The gateway mock agrees with your team and not necessarily with the gateway, so it is built from the gateway’s own API documentation and checked against the real sandbox from staging before each release. A load test says nothing about functional correctness — order history can be fast and wrong — which the CI suite covers.

The mix. On every build in CI: the scripted API suite, with I1 to I8 among its first regression tests; every response, error responses included, validated against the OpenAPI document; GraphQL tests that assert on the body; the webhook tests of I6 and I7; and the gateway mocked for its error paths. For every new or changed endpoint: a short exploratory session whose findings become scripted tests. Before each release: one run against the real gateway sandbox from staging. Before the campaign, and after any change to a hot path such as order history: a load test with production-like data. That is the guide’s default mix with two departures. The installed apps cannot be updated together with the API — versions from the last six months are still in use — so for them the API behaves like a public product and needs consumer-driven contracts and tests for every app version still supported. And it moves money and personal data, so it needs the OWASP API Security Top 10 tests.

What you do first, today: the contract. Bring the five-month-old OpenAPI document up to date with the API and validate it — an outdated or invalid document makes every contract result meaningless — then generate the first scripted cases from it (the guide’s OpenAPI → Test Cases tool helps) and add the business-rule cases a generator cannot know, such as I3. With about five weeks to 12 Nov, a workable order is: the contract and the contract checks in CI; then the scripted suite with I1 to I8, the gateway mock and the webhook tests; then the load test, early enough to fix what it finds and run it again before the campaign.

The reply to the team lead: “No: more people sending requests by hand answer only one of the five questions. Of last quarter’s eight problems, hand-run requests could at best have caught I3, I7 and perhaps I5; they could not have shown I1, because a renamed field looks fine to a person, nor I2 and I6, because the sandbox cannot produce a slow answer or a retry, nor I4, because one person cannot produce the campaign peak — and nothing they find stays checked afterwards, which is how I8 happened. Postman can run saved collections with assertions in CI, which covers scripted tests, but contract validation, the gateway mock and a load test need the specification, a mock layer and a load tool alongside. A second tester would be best spent on exploratory sessions for new endpoints and on turning their findings into scripted tests.”

Check your own work. Each problem should have an approach and the reason the current practice missed it. Other mappings are fine when the reason is stated — for example I7 under exploratory testing, as long as you add that only a scripted test keeps it fixed, or I6 under scripted tests, as long as you note that the duplicate delivery must be produced on demand. Your mix should say when each approach runs (every build, every new endpoint, every release, before the campaign) and what each one misses. In ISTQB CTFL v4.0 terms: the OpenAPI document is the test basis and, for contract tests, the test oracle; exploratory testing is an experience-based test technique; the CI suite is regression testing, and the fix of each of I1 to I8 first needs confirmation testing; the load test is non-functional testing of performance efficiency (ISO/IEC 25010); and the staging run against the real gateway is system integration testing.