Usability Testing

Why it matters

A product can pass every functional and accessibility test and still lose users, because a working feature that people cannot find, understand or finish is a failure from their point of view. Those failures rarely appear in the defect tracker; they appear as abandoned carts, support calls and one-star reviews, and by then the design decisions that caused them are expensive to undo.

For the tester, usability testing adds a second kind of evidence. Functional tests show that the system follows its rules; usability tests show whether real people can reach their goals with it. It also sharpens your own eye: once you have watched someone struggle with an unlabelled icon, you report such problems earlier and with evidence ("4 of 5 participants opened Transfers first") instead of opinion ("the menu is confusing").

Usability, user experience and three kinds of evaluation

Usability is about specified users, specified goals and a specified context: a payment flow can be usable for a daily customer on a phone and unusable for a first-time user on a small screen. ISO 9241-11 names three components, and each gives usability testing one of its measures. Effectiveness is the accuracy and completeness with which users achieve their goals, measured as task success and the errors made on the way. Efficiency is the resources used in relation to the results achieved, most often time, measured as time on task. Satisfaction is the extent to which the user's physical, cognitive and emotional responses to using the product meet their needs and expectations, measured with ratings and questionnaires.

User experience (UX) is wider than usability. The ISO 9241 series describes it as a person's perceptions and responses that result from using, or expecting to use, a product: expectations before the first screen, emotions during use, trust and memory afterwards. Usability is a necessary part of a good UX, not the whole of it. ISO/IEC 25010:2023 covers the same ground under interaction capability, with sub-characteristics such as learnability, operability and user error protection. The ISTQB usability testing syllabus distinguishes three ways to evaluate usability, and a good team uses more than one:

• Usability reviews: one or more reviewers, preferably usability or domain experts, examine the interface and judge where it breaks usability requirements, guidelines or principles, instead of watching users attempt tasks. Heuristic evaluation, a review against a short list of heuristics, is the best known.
• Usability testing with representative users: participants attempt realistic tasks while the team observes and measures. Most of this lesson is about this method.
• User surveys: participants answer a questionnaire about their experience, after a test session or after real use.

Usability (ISO 9241-11)       User experience (UX)
------------------------      ------------------------------------
specified users, goals        the whole person, before, during
and context of use            and after use
effectiveness  -> success,    expectations, emotions, trust,
                  errors      memory of the product
efficiency     -> time        includes usability, and more
satisfaction   -> ratings

This lesson: measure usability in a session.
Next lesson:  UX testing of flows, navigation, messages and forms.

Planning: goals, tasks and participants

Start with the questions the team needs answered, such as "Can a new customer pay a bill without help?" Then write tasks as realistic scenarios in the participant's own words, with the data they need and a definition of success. Never use the labels on the screen: "Open Payments and add a biller" tests whether people can follow instructions, not whether they can find payments.

Recruit participants who match the real user profile: age, experience with similar products, language, device. Colleagues who know the product are not representative. Many teams run about five participants per user group in a round, fix what they found, and run another round, because the second round often shows a different set of problems. Decide the roles: a moderator who gives the tasks and asks neutral questions, and a note-taker who records what happens; other stakeholders, such as developers, may watch as observers. Run one pilot session to fix unclear tasks and timing, and keep consent and privacy in order: participants agree to being observed or recorded, and no identifying data goes into the report.

Task 3 of 4  (a scenario, in the participant's words)
  "Your electricity bill for this month has arrived. Pay it from
   this app. The account number is on the paper bill in front of you."
Success:   bill paid from the current account, confirmation shown
Target:    unaided success for at least 4 of 5 participants,
           within 2 minutes, at most 1 wrong path
Do not say: "Payments", "biller", "Add" - the words on the screen

Try it: Test Scenario Generator

Running the session: think-aloud and the four measures

Begin with a short briefing: the product is being tested, not the participant; there are no right or wrong answers; please think aloud. In the think-aloud protocol the participant says what they are looking for, what they expect and what confuses them while they work. The moderator watches, prompts only with neutral questions such as "What are you thinking now?" or "What did you expect to happen?", and never explains or helps. If a participant is completely stuck, the moderator notes the moment and either moves on to the next task or gives the smallest possible hint; a task finished after a hint is recorded as assisted, not as a success.

The note-taker records four measures for every task, plus the quotes and wrong turns that explain them:

• Success: completed unaided, completed with assistance, or failed (gave up, ran out of time, or finished a different task believing it was done).
• Time on task: from reading the task to the success criterion being met; report the median (the middle value when the times are sorted), because one lost participant distorts an average.
• Errors: wrong paths, wrong controls, incorrect input, and whether the person noticed and recovered; count them per task.
• Satisfaction: a rating after each task (for example 1 to 5) and a questionnaire after the session, such as the System Usability Scale, ten statements that produce a score from 0 to 100.

Try it: Percentage Calculator

Heuristic evaluation: an expert review without users

A heuristic evaluation is a review in which a small group of evaluators independently walk through the main flows and judge every screen against a short list of recognised usability principles, then merge their notes and rate each problem's severity. It needs no participants, so it fits early prototypes, wireframes and the week before a usability test.

A heuristic evaluation predicts problems; a usability test observes them. Evaluators find many issues quickly and cheaply, but they also flag things users never notice, miss things only users reveal, and cannot tell you success rates or times. Use it first, fix the obvious, then test with users. The best-known list of principles is Jakob Nielsen's ten usability heuristics:

• Visibility of system status: the interface says what is happening, such as a saved or sending state.
• Match between system and the real world: words and concepts the user knows, not internal ones.
• User control and freedom: an easy way to undo, cancel or go back.
• Consistency and standards: the same words and patterns throughout, and the platform's conventions.
• Error prevention: constraints and confirmations that stop mistakes before they happen.
• Recognition rather than recall: options and actions visible, not memorised.
• Flexibility and efficiency of use: shortcuts for experienced users without burdening new ones.
• Aesthetic and minimalist design: nothing that competes with the information users need.
• Help users recognize, diagnose and recover from errors: plain-language messages that say what went wrong and how to fix it.
• Help and documentation: available when needed, focused on the user's task.

Try it: AI QA Checklist Generator

Worked example: paying a bill in a banking app

A bank adds bill payments to its app. The target users are existing customers who use the app only for transfers. The team recruits five of them, and the moderator gives the task from the card above with a paper bill as the data. Targets: at least 4 of 5 succeed unaided within 2 minutes with at most one wrong path, and a post-task rating of 4 or more.

After the five sessions, the note-taker consolidates across participants: success per task, median time, errors per task and the ratings, compared with the targets agreed in planning. Each observed problem becomes a finding with its evidence: where it happened, how many participants hit it, and what they said.

• Results: 2 of 5 completed the task unaided, 2 needed a hint and 1 failed; the median time was 4:10 against a target of 2:00, the median number of wrong paths 3 against 1, and the median rating 2 against 4. Every target is missed, and the note-taker's notes say why: four participants opened Transfers first, because Payments was hidden under More, and two then typed the bill's account number into the transfer amount field, because the account and amount fields on that screen were labelled only with icons.
• The participant recorded as failed pressed Pay twice: no confirmation appeared at once and the button stayed active, so the bill was paid twice and they gave up unsure whether it was paid at all.
• The findings go to the tracker with their evidence and the heuristic each violates: recognition rather than recall for the hidden menu, match with the real world for the icon-only fields, visibility of system status and error prevention for the missing confirmation. The double payment is also a functional defect, reported separately with Critical severity. The consolidated sheet below is what the report is built from.

Task: pay this month's electricity bill (5 participants)
Measure                                Target           Result
Success unaided / assisted / failed    4 of 5 unaided   2 / 2 / 1
Time on task (median)                  <= 2:00          4:10
Wrong paths per participant (median)   <= 1             3
Post-task rating 1 to 5 (median)       >= 4             2
Notes: 4 opened Transfers first; 2 typed the account number into
       the amount field; 1 pressed Pay twice, no confirmation shown

Frequently asked questions

References

• ISO 9241-11:2018 — Ergonomics of human-system interaction — Part 11: Usability: Definitions and conceptsISO · 2018The framework for the concept of usability: effectiveness, efficiency and satisfaction for specified users, goals and context of use, with usability treated as an outcome of use.
• Certified Tester Usability Testing (CT-UT)ISTQB · 2018ISTQB's specialist syllabus on usability testing: usability, user experience and accessibility, evaluation methods and approaches, and the relevant standards.
• 10 Usability Heuristics for User Interface Design (Jakob Nielsen)Nielsen Norman Group · 1994Jakob Nielsen's ten broad rules of thumb for interface design — such as visibility of system status and error prevention — originally written for heuristic evaluation.
• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality modelISO/IEC · 2023The product quality model: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability and flexibility.