API Performance Comparator

Server assisted Performance

Runs the same bounded sample against two public URLs and presents a side-by-side comparison of latency percentiles, size and status consistency.

Privacy: This tool sends the request through the Mutqan backend relay to reach the target. Only the target URL, headers and body you enter are transmitted, and nothing is stored.

What is performance testing?

Performance testing measures how a system behaves under a defined workload: response time (usually the 95th percentile, not the average), throughput (requests per second) and error rate, while watching CPU, memory and database saturation. Load testing uses the expected traffic, stress testing pushes past it to find the breaking point, soak testing holds it for hours to expose leaks, and spike testing checks recovery from a sudden surge.

Read the guide: What is Performance Testing?

How to use API Performance Comparator

  1. Enter Endpoint A and Endpoint B (for example v1 and v2, or two regions) and optional labels.
  2. Choose requests per endpoint (≤ 25), concurrency and method; add shared headers if needed.
  3. Click "Compare" — the relay measures A, then B, with identical settings.
  4. Read the verdict: which endpoint is faster at the median and whether the difference is significant.
  5. Repeat with more samples if the result is "within noise".

API Performance Comparator features

  • Runs the same bounded sample against two public endpoints, one after the other
  • Side-by-side table of median, average, p90/p95/p99, min/max and throughput
  • Percentage delta and a "better" column per metric
  • Significance hint: do the medians differ by at least 20% and by more than the larger standard deviation?
  • Error rate and status breakdown for both endpoints
  • Chart comparing p50 and p95

API Performance Comparator example

v1 vs v2 of a search endpoint

Input:

A: https://api.example.com/v1/search?q=shoes
B: https://api.example.com/v2/search?q=shoes
Requests: 10 each

Output:

Faster (median): v2 (1.6× at p50)
p50 A → B: -38% · p95 A → B: -41%
Significance: likely real
Median: 412 ms vs 256 ms · p95: 640 ms vs 377 ms

Frequently asked questions about API Performance Comparator

How is the comparison kept fair?

Both endpoints receive the same method, headers, request count and concurrency from the same relay, run one after the other so they do not compete for bandwidth.

What does "within noise" mean?

The medians differ by less than 20%, or by less than the larger of the two runs' standard deviations, so the gap could be random variation. Increase the sample size or repeat the run before drawing conclusions.

Can I compare staging against production?

Yes, as long as both URLs are public and you are allowed to send a handful of requests to them. Keep request counts low for production.

Why compare p95 as well as the median?

Two endpoints can have the same median while one has a much fatter tail. The p95 column shows the experience of the slowest users.

Are response sizes compared?

When the relay reports body sizes they appear as an extra row; otherwise use the Response Size Analyzer on each URL.

Technical notes

Sequential execution avoids the two runs competing for the same relay bandwidth, which would penalise whichever endpoint happened to run second under load. The trade-off is that network conditions can drift between runs; the significance hint calls a difference "likely real" only when the slower median is at least 1.2 times the faster one and the gap is larger than the larger of the two standard deviations; anything else is flagged as possible noise.

References