DevOps for Testers

Why it matters

When the people who build, test and run a product work in silos, changes pile up into large, risky releases, feedback on a defect arrives days after the code was written, and production problems reach the team through unhappy users.

For a tester, DevOps changes the job rather than removing it. Releases come too often for a full manual regression pass each time, so repeated checks must be automated and trustworthy, while exploratory testing and risk analysis matter more, because automated checks only find what someone thought to check. A tester who understands the pipeline, production signals and delivery metrics can show where quality is lost and argue for a fix with data.

A culture first, then a set of practices

The word joins development and operations. The point is not a job title or a tool but removing the wall between the people who change a system and the people who keep it running. The whole team shares responsibility for the product in production, treats failures as something to learn from rather than a reason to blame, and improves its way of working in small steps, as you saw in the Kanban lesson. Some teams summarise the idea as CALMS: culture, automation, lean thinking, measurement and sharing.

• Version control for everything: code, tests, pipeline definitions, configuration and infrastructure.
• Small batches: many small changes instead of a few big releases, each easy to test and to roll back.
• Continuous integration and delivery: an automated pipeline builds, tests and prepares every change for release (the subject of the next lesson).
• Infrastructure as code: environments are built from scripts, so a test environment can match production.
• Monitoring and blameless post-incident reviews: every incident becomes an improvement.

The DevOps loop: testing in every stage

Teams often draw DevOps as an endless loop: plan, code, build, test, release, deploy, operate, monitor. It is a common picture, not a standard, but it shows that "test" is one stage while testing activities happen in all of them.

In plan, you question acceptance criteria and agree how each change will be tested and monitored. In code and build, the unit tests developers write and static analysis give feedback within minutes. In test, automated regression and API tests run while you explore what is new. In release, test results and known risks feed the quality gate (a checkpoint where a change must meet agreed criteria before it moves on). In deploy, smoke tests (quick checks that the main functions work after a deployment) and health checks confirm the new version against agreed rollback criteria. In operate and monitor, incidents and production signals show what testing missed, and each finding returns to the next plan as a new test.

How QA fits: automation, fast feedback, monitoring

The ISTQB syllabus lists what DevOps gives testing, including fast feedback on code quality and regressions, stable test environments, more attention to performance and reliability, and less repetitive manual testing. It also names risks and challenges: the delivery pipeline must be defined and set up, CI/CD tools must be introduced and maintained, and test automation needs extra resources and can be hard to establish and maintain. Manual testing, especially from the user’s perspective, is still needed. The tester’s contribution falls into three areas.

• Automation: agree with the team which checks run on every change and at which level (mostly unit and API tests, few end-to-end journeys), and keep them reliable. A flaky test, one that passes or fails at random on the same code, teaches the team to ignore red builds.
• Fast feedback: a check that fails five minutes after a commit is fixed by the developer who just wrote the code; the same defect found two weeks later costs a context switch, a bug report and a retest. Martin Fowler’s description of continuous integration stresses keeping the build fast.
• Monitoring: only production has real users, data and load. Learn what the team watches (Google’s SRE book recommends four golden signals: latency, traffic, errors and saturation) and ask for checks that would have caught recent incidents, such as a synthetic login every few minutes.

Try it: Regression Scenario Generator Try it: API Health Checker Try it: Response Time Analyzer

DORA metrics: signals of delivery quality

DORA (DevOps Research and Assessment), a research programme run by Google Cloud, describes delivery performance with five metrics: three for throughput, how quickly changes reach users, and two for instability, how often they hurt them. A slow or flaky test suite lengthens lead time; gaps in testing show up as failed deployments. Read the metrics as signals about the whole delivery system, never as scores for testers, and measure one application or service at a time against its own trend.

• Change lead time (throughput): the time from commit to running in production. It starts at the commit, so it is shorter than the request-to-delivery lead time many Kanban teams track.
• Deployment frequency (throughput): how often changes reach production.
• Failed deployment recovery time (throughput): how long recovery takes after a deployment that fails and needs immediate intervention.
• Change fail rate (instability): the share of deployments that need immediate intervention afterwards, such as a rollback or a hotfix; the closest delivery-level signal of escaped defects.
• Deployment rework rate (instability): the share of deployments that are unplanned and caused by an incident in production.

Worked example: a month of delivery data

A team running an online shop brings its delivery data to the monthly retrospective, comparing weeks 5–8 with the four weeks before.

The team deploys more often, but instability rose and recovery slowed, so the extra speed is no improvement. A metric shows where to look, not why, so the tester examines the four failed deployments and the longer lead time.

• Two payment failures: the pipeline tested only against a mock of the payment provider, and the provider had changed a field in its API. Action: a nightly API test against the provider’s sandbox.
• A database migration timed out on production-sized data. Action: rehearse migrations in staging on an anonymised copy of that size.
• A wrong discount, reported by customers hours later; failures found this way pushed recovery time up. Action: a post-deployment smoke test that places a test order and checks its total.
• Lead time grew because changes queued for a manual regression pass held twice a week. Action: automate the stable checks and keep exploratory sessions for changed areas, the wait-time thinking from Kanban.

Online shop, production deployments   Weeks 1-4       Weeks 5-8

Deployment frequency                   12              16
Change lead time (median)              2 days          3 days
Failed deployment recovery (median)    45 min          3 h
Change fail rate                       1 of 12 (8%)    4 of 16 (25%)
Deployment rework rate                 1 of 12 (8%)    3 of 16 (19%)

Try it: REST API Tester Try it: API Endpoint Tester

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• DORA's software delivery performance metricsDORA (Google Cloud)The DORA delivery metrics — change lead time, deployment frequency, failed deployment recovery time, change fail rate and related measures — grouped by throughput and instability.
• Monitoring Distributed Systems — Site Reliability Engineering (Rob Ewaschuk)Google / O'Reilly Media · 2016Google's SRE guidance on monitoring: black-box versus white-box monitoring and the four golden signals — latency, traffic, errors and saturation.
• Continuous Integration (Martin Fowler)martinfowler.com · 2024The widely referenced description of continuous integration: integrate often, keep the build self-testing and fast, and fix a broken build immediately.