Non-Functional Testing

Why it matters

A product can pass every functional test and still fail its users: the payment page times out on the first day of a sale, a screen-reader user cannot reach the Pay button, or one customer can open another customer’s invoice. Such failures hit many users at once, and their cause is often a design decision, so late fixes are costly; the Foundation syllabus warns that late discovery of non-functional defects can seriously threaten a project.

For a tester, "does it work?" becomes "does it work well enough, for whom, under which conditions?" That needs a target agreed in advance, or "fast enough" becomes an argument at release time instead of a test result.

What it does versus how well it does it

A functional test checks behaviour against a rule: this input, this expected output. A non-functional test keeps the feature and adds a condition and a measure: how fast, how many users at once, on which device, after which failure. Many reuse a functional test and check a constraint while it runs, such as search returning the right results within 2 seconds.

It may need a special environment: a production-like system for load, real devices, assistive technology, or real users in a usability lab. Its expected result is a target, and the outcome is a measured value compared with it. Where possible it starts early, in requirement and design reviews and in component tests, because the fix for a slow or insecure design is rarely small.

The areas a tester meets first

The lesson on software quality introduced the nine ISO/IEC 25010 characteristics: functional suitability belongs to functional testing, the other eight to non-functional testing. Maintainability is judged mostly through reviews and static analysis, and safety matters where software can cause harm. In daily work testers meet these areas first:

• Performance (performance efficiency): how fast it responds, how much load it carries and what resources it uses.
• Security: only authorised people can see and change data, and attacks are resisted.
• Usability (interaction capability in ISO/IEC 25010:2023): intended users reach their goals with effectiveness, efficiency and satisfaction, the three parts of the ISO 9241-11 definition.
• Accessibility: people with disabilities can use it, usually judged against WCAG 2.2. ISO/IEC 25010 treats it as part of interaction capability.
• Compatibility: it works on the supported browsers, devices and operating systems, and with connected systems. ISO/IEC 25010 files browser and device support under flexibility (portability in 2011); teams usually call both compatibility.
• Reliability: it works correctly for as long as needed, stays available, and keeps working or recovers when something goes wrong, such as a dropped connection.
• Localization: each language and region version is complete and correct, from translation and right-to-left layout to date, number and currency formats. It is not an ISO/IEC 25010 characteristic; teams plan it as its own test type.

Try it: Test Scenario Generator Try it: Security Header Analyzer

Turning "fast" into a measurable criterion

"The page must be fast" cannot be tested, because nobody can say when it fails. For times, prefer percentiles to averages. The 95th percentile (p95) is the time within which 95% of requests finish: "p95 at most 2 s" means at least 95% of requests finish within 2 seconds, while an average of 1 second can hide requests that take 10. For a standard, name the version and level, such as WCAG 2.2 level AA.

A measurable acceptance criterion states four things:

• Measure: what is counted, such as p95 response time, error rate, task success rate or minutes to recover.
• Target: the threshold that decides pass or fail, with its unit: at most 2 s, below 1%, zero.
• Conditions: the load, data, devices, network, users and environment the target applies to.
• Method: how and where it is measured, so that two testers get comparable numbers.

Vague:       The registration page must be fast.

Measurable:
  Measure     p95 response time of "Register", and error rate
  Target      p95 <= 2 s, errors < 1%
  Conditions  2,000 concurrent students for 15 minutes,
              production-size data, staging environment
  Method      load test script; numbers from the tool report

Try it: Response Time Analyzer

Worked example: course registration day

A university opens course registration at 09:00, and most students try in the first minutes. The story is functional: "As a student, I want to register for a course section, so that I have a seat." In refinement the tester asks one question per area, and the team agrees these criteria:

• Performance: with 2,000 concurrent students for 15 minutes, p95 for Register at most 2 s and errors below 1%, in a load test on staging with production-size data.
• Security: a request with another student’s id is refused with HTTP status 403 (Forbidden) or 404 (Not Found) and returns no data, checked with two test accounts.
• Usability: at least 4 of 5 first-year students register unaided within 3 minutes in a moderated session, where a facilitator gives the task and watches without helping.
• Accessibility: WCAG 2.2 level AA; for example every step works by keyboard alone (success criterion 2.1.1) and normal text has a contrast ratio of at least 4.5:1 (1.4.3), checked with a keyboard walk-through and a contrast checker.
• Compatibility: the registration tests pass on every browser and device in the support matrix, run on each row.
• Reliability: if the connection drops after Register is pressed, the student has one registration or none, never two, checked by cutting the network mid-request and counting registrations.
• Localization: the Arabic interface has no untranslated text, reads right to left and shows class times in the university’s time zone, checked with Arabic test data.

Try it: Arabic Test Data Generator

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality modelISO/IEC · 2023The product quality model: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability and flexibility.
• ISO 9241-11:2018 — Ergonomics of human-system interaction — Part 11: Usability: Definitions and conceptsISO · 2018The framework for the concept of usability: effectiveness, efficiency and satisfaction for specified users, goals and context of use, with usability treated as an outcome of use.
• Web Content Accessibility Guidelines (WCAG) 2.2W3C · 2024The W3C Recommendation that defines the testable success criteria for accessible web content, organised under four principles and three conformance levels (A, AA, AAA).