Module 25 · Professional QA & test management

Continuous Improvement

By the Mutqan Soft QA team · · 18 min · Advanced

Continuous improvement means changing how a team works in small, regular steps and checking each step against evidence, instead of waiting for a big reorganisation. In testing, it covers the whole test process: how work reaches testing, how tests are designed and run, and how the team learns from defects that escaped.

For example, a retrospective shows that many stories are reopened after testing. Instead of adding an approval step for everyone, the team tries one change for two sprints, counts reopened stories before and after, and then decides whether to keep it. That loop of plan, do, check and act is the core of this lesson.

Why it matters

Process problems repeat quietly: the same late testing, the same unstable environment, the same kind of escaped defect, sprint after sprint. Each repetition costs rework and delay, and teams get used to it. The ISTQB Foundation syllabus v4.0 calls retrospectives critical for continuous improvement and stresses that recommended improvements are followed up. Without that follow-up, the same complaints return and people stop raising them.

Testers see where work waits, where defects come from and which checks are missing, and they hold much of the data that shows it. Turning that view into small, measured changes makes testing faster and more effective, and shows the team that raising a problem leads somewhere.

Retrospectives that lead to change

The lessons on Scrum and quality culture introduced the retrospective. The ISTQB Foundation syllabus v4.0 adds the detail: it is often held at the end of an iteration or project, at a release milestone or when needed; developers, the product owner and others take part, not only testers; and it asks what was successful and should be retained, what could be improved, and how to put the improvements into practice. Its results are recorded, normally as part of the test completion report.

Retrospectives often fail at the evidence and at the follow-through. A tester helps with both:

  • Bring facts, not impressions: reopened stories, days waiting for testing, test environment downtime, escaped defects.
  • Ask why until you reach a cause the team can change; "testing is always late" is a symptom.
  • Leave with one or two actions, each with an owner and a date, and start the next retrospective by checking them.
  • Keep one improvement backlog for retrospective actions, postmortem actions and lessons learned at test completion.

Process measures: evidence before and after

A process measure describes how the work flows, such as waiting, rework or speed of feedback, rather than the quality of the product. Start from the goal and the question that would show progress, then pick the measure; this is the idea of the Goal-Question-Metric approach.

Record a baseline over several sprints before changing anything, because small numbers vary, and keep the definition the same afterwards. Prefer a leading measure that moves within weeks over a lagging one such as incidents per quarter, and add a balancing measure that would reveal a side effect. Measure the team’s process, never individuals: as the lesson on QA metrics and KPIs showed, a measure that becomes a personal target stops measuring.

  • Waiting: days from "ready for test" to the start of testing; hours the test environment is unavailable.
  • Rework: share of stories reopened after testing.
  • Feedback: time from a commit to a test result; pipeline runs that failed because of flaky tests.
  • Follow-through: retrospective actions completed by their agreed date.

Try it: Percentage Calculator Try it: Date Difference Calculator

Small experiments: plan, do, check, act

Most process changes are guesses, so run each one as a small experiment with the plan-do-check-act (PDCA) cycle associated with Walter Shewhart and W. Edwards Deming. Write the experiment down before you start, so the result cannot be reinterpreted later, and change one thing at a time.

  • Plan: the problem and its evidence, a hypothesis ("if we…, then…, because…"), one change, the measure and its baseline, a balancing measure, a timebox and a decision rule.
  • Do: run the smallest version of the change, for a few stories or one team, for the timebox.
  • Check: compare the result with the baseline and the decision rule, and look for side effects.
  • Act: adopt the change and make it standard, for example in the Definition of Done or a checklist; adapt it and run another cycle; or abandon it and record what you learned.

Worked example: fewer stories reopened after testing

A team building a clinic booking web app reviews sprints 10 to 12: 9 of 24 finished stories (37.5%) were reopened after testing. The tester sorted the reasons, and 6 of the 9 came from business rules nobody had discussed, such as two appointments booked for the same time.

Plan: if the product owner, a developer and the tester agree examples for each story before development, fewer stories will be reopened, because the rules are settled before coding. The change: a 20-minute example session per story for two sprints. Measure: reopened stories (baseline 37.5%); balancing measure: stories finished per sprint (baseline 8). Rule: adopt if no more than 3 stories are reopened in the two sprints and at least 7 are finished in each.

  • Do: the team runs the sessions for 16 stories, and the tester records reopen reasons with the same categories as before.
  • Check: 3 of 16 stories (18.75%) are reopened, only 1 because of an undiscussed rule, and 8 stories are finished in each sprint. The rule is met. Sixteen stories is a small sample, but the reasons changed as the hypothesis predicted.
  • Act: the team adopts the session, adds "examples agreed" to its Definition of Ready, keeps counting, and takes the next problem from the backlog.
One plan-do-check-act cycle from the worked example: with examples agreed before development, reopened stories fell from 9 of 24 to 3 of 16, so the team adopted the change.
  1. 1PlanHypothesis, measure, baseline, decision ruleAgree examples before development; baseline 9 of 24 reopened (37.5%)
  2. 2DoSmallest version, for a timeboxA 20-minute example session for each of 16 stories over two sprints
  3. 3CheckCompare with the baseline and the rule3 of 16 reopened (18.75%); 8 stories finished in each sprintRule met
  4. 4ActAdopt, adapt or abandonAdopt: "examples agreed" joins the Definition of Ready

Back to step 1 Next cycle: the next problem from the improvement backlog

Sixteen stories is a small sample: the team keeps counting after adopting the change.

Try it: Acceptance Criteria Generator Try it: Percentage Calculator

Test process improvement models

You do not need a model to start improving. Some organisations, however, assess their test process against a published model to find gaps systematically across many teams. The models fall into two families, and analytical approaches complement both:

  • Process reference models measure maturity. TMMi (Test Maturity Model integration) is a staged model with five maturity levels, from Initial to Optimization; TPI NEXT assesses each key area of the test process separately.
  • Content reference models, such as Critical Testing Processes (CTP) and the Systematic Test and Evaluation Process (STEP), describe good practice for the main test activities without a maturity scale.
  • Analytical approaches start from your own data, for example causal analysis of defects or Goal-Question-Metric.

Common mistakes

  • Ending a retrospective with ten actions and no owners or dates, then never checking them.
  • Changing several things at once, so nobody can tell which change helped.
  • Having no baseline, or changing how a measure is counted halfway through.
  • Declaring success after one sprint, although small numbers vary.
  • Turning a process measure into a target for individuals, such as bugs found per tester.
  • Pursuing a maturity level instead of fixing the team’s actual problems.

The QA perspective

On a real team, the tester often holds the evidence: the defect tracker, the test management tool and the pipeline history show where work waits and where defects come from. Before each retrospective, prepare two or three numbers and one concrete example. Work with the Scrum Master or test manager on the improvement backlog, with developers and the product owner on experiments, and with operations so that postmortem actions join the same list.

Keep each experiment card with its result and decision, so the team remembers why a practice exists and can drop it when it stops helping. Useful questions: "What would we see if this worked?", "What could get worse?" and "Which earlier action is still open?"

Practice exercise

Turn retrospective notes into one experiment

Your task

A team building an insurance claims portal held its sprint 14 retrospective. Under "did not go well" it wrote: testing always happens in the last three days of the sprint; staging was down again on Tuesday; two defects escaped to production, both in claim status emails; too many meetings. The data from the last three sprints is below.

Choose one problem to work on first, justify the choice with the data, and write an experiment card for it. Then say what should happen to the other problems.

Material
Insurance claims portal - data for the sprint 14 retrospective (2-week sprints)

                                            Sprint 12   Sprint 13   Sprint 14
Stories finished                                  9           8           9
Stories reopened after testing                    3           4           3
Average days a story waited for testing         2.5         3.0         2.8
Staging unavailable (hours)                       6          11           9
Defects escaped to production                     1           0           2
Open actions from earlier retrospectives          2           4           6
Steps
  1. Mark which numbers are stable, rising or falling.
  2. Write the hypothesis as "if we…, then…, because…", with one change.
  3. Calculate the baseline over all three sprints, choose a balancing measure and set the decision rule.
  4. Decide where each remaining problem goes.
Show the model answer
Model answer

Reading the data: reopened stories and waiting time are stable, staging downtime and escaped defects vary, and open actions from earlier retrospectives rise from 2 to 6.

A good first choice is late testing: the data confirm it in every sprint, and it plausibly feeds the reopened stories. Staging downtime is a defensible alternative if the team can influence the environment.

Card: stories wait about 2.8 days on average for testing (the mean of 2.5, 3.0 and 2.8). Hypothesis: if the team limits work in progress (WIP) to three stories in development, stories will wait less for testing, because developers finish one story before starting the next, so stories reach the tester one by one instead of in a batch at the end of the sprint. Change: that limit for two sprints. Measure: average days waiting, baseline 2.8. Balancing: stories finished per sprint, baseline about 9 (26 in three sprints), and reopened stories, baseline 10 of 26 (38%). Rule: adopt if waiting falls below 1.5 days, at least 8 stories are finished per sprint and no more than 38% are reopened; adapt and run another cycle if waiting falls but stays at 1.5 days or more, or if a balancing measure gets worse; abandon if waiting does not fall.

Rising open actions (2, 4, 6) show that the follow-through is failing: agree at most two actions per retrospective, each with an owner and a date, check them first next time, and close or drop the six open ones explicitly.

The escaped claim-email defects go through incident feedback: a postmortem, the test gaps and new regression tests. Staging downtime stays on the improvement backlog as the next candidate and goes into the project risk log with an owner, because it threatens test execution. "Too many meetings" has no data yet; at the next retrospective, ask which meetings and how long they take.

Frequently asked questions

How is a retrospective different from a postmortem?

A retrospective is held regularly, at the end of a sprint, iteration or release, about how the team worked, whether or not anything went wrong. A postmortem is triggered by one incident and analyses that event. Both should be blameless, and the actions from both belong in the same improvement backlog. Some organisations also call a retrospective at the end of a project a post-mortem or post-project meeting, so check which one is meant.

Does a team need TMMi or TPI NEXT to improve?

No. Retrospectives, a few process measures and small experiments are enough for most teams. A model helps when an organisation wants a systematic assessment across many teams or has to show a defined process to a customer or auditor. Even then, each change the model suggests is best introduced as a small, measured step.

What if an experiment does not work?

Then you have learned something cheaply. Record the result and why you think it failed, drop or adapt the change, and keep the problem on the improvement backlog. An experiment really fails only when it teaches nothing, for example because there was no baseline or the decision rule was set after the result was known.

How many experiments should run at once, and how often should we review them?

One or two at a time, matched to the team’s rhythm: in Scrum, review them at every Sprint Retrospective; in Kanban or continuous flow, at a fixed interval such as every two weeks. The more experiments run in parallel, the harder it is to tell which change caused which effect.

References

Tools

Practise with Mutqan tools

Percentage Calculator

Calculate percentages, percentage change, increase/decrease and what percent X is of Y.

Utilities

Date Difference Calculator

Calculate the exact difference between two dates in years, months, days, hours and total units.

Utilities

Go deeper

Your progress on this topic

Part of the QA Roadmap