Production Quality and Monitoring

Why it matters

Testing before release samples the product. As one of the seven testing principles says, exhaustive testing is impossible, and no test environment reproduces every device, data combination, third-party response and traffic peak of real use, so some defects reach production. Without monitoring, the first sign is a customer complaint hours later, and each hour adds failed orders, wrong data or support work.

For a tester, production shows which risks were real and which tests were missing. A tester who can read its signals helps judge whether a release is healthy, writes precise reports for production failures and turns each escaped defect into a new test or check.

Production quality as agreed targets

The lesson on shift-left and shift-right introduced the four golden signals. To judge production quality, turn them into targets. Google’s Site Reliability Engineering (SRE) book uses three terms. A service level indicator (SLI) is a carefully defined measure of the service users get, such as the share of order requests that succeed within 2 seconds. A service level objective (SLO) is the target for it, such as 99.5% over 30 days. A service level agreement (SLA) is a contract with customers that states the consequences of missing the objectives, such as a partial refund.

An SLO also sets an error budget: the failure it allows, 100% minus the objective. At 99.5% and about 200,000 order requests a month, that is about 1,000 failed or slow requests. In the approach the SRE book describes, a team that has spent its budget halts releases for a while and invests in reliability and testing, so "is production OK?" becomes a number everyone agreed on.

• Measure journeys users feel, such as sign-in, search and checkout, rather than server CPU.
• Add business correctness: orders per hour, payments matched to orders, confirmations actually sent. Some defects raise no error at all.

Try it: Response Time Analyzer

Five kinds of monitoring signal

The SRE book says monitoring should answer two questions: what is broken, and why? White-box monitoring uses what the system records about itself; black-box monitoring tests behaviour from outside, as a user would see it. Each kind of signal answers a different question, and together they work as a chain: a symptom raises an alert, and the details lead to the cause.

• Metrics: numbers over time, such as requests per minute, error rate, p95 latency (the time within which 95% of requests finish) or orders per hour. They are cheap to keep for months and show trends and sudden changes, but not which user was affected or why.
• Logs: timestamped records of single events. Structured logs, one JSON object per line with fields such as time, level, service, route and request id, can be filtered. Write times in UTC, so that events from servers in different time zones can be compared. As the OWASP Logging Cheat Sheet advises, do not record passwords, session ids, access tokens, card data or sensitive personal data in logs.
• Traces: the path of one request through every service it touches, as nested timed steps called spans. A trace shows where the time went and which call failed. Services pass the trace id on in a request header (the W3C Trace Context standard names it traceparent); writing the same id into the logs links the two.
• Alerts: rules that watch signals and notify a person. An urgent alert sent to the person on call, the one who answers alerts during that shift, is called a page. Page on symptoms users feel, such as an SLI dropping, rather than on every possible cause; every page should need urgent, thoughtful action, and many teams link it to a runbook, the written steps for handling it. Less urgent problems become tickets. Too many pages teach people to ignore them.
• Synthetic checks: scripted journeys run from outside every few minutes, such as a monitoring account logging in and searching. They catch outages on quiet paths and at night, but cover only what someone scripted.

Try it: JSON Lines Validator Try it: Timestamp Converter Try it: Cron Expression Explainer

Test the monitoring too

Monitoring is code and configuration, so it has defects: an alert that can never fire, a log line without the request id, a synthetic check that has failed quietly for a week. For each risky change, ask which signal would show it broken in production and who would be told, then check that in a test environment.

• Make a dependency fail, for example with a stub that returns 503, and confirm that the error shows in metrics, logs and traces under one request id and that the alert reaches a person.
• Read sample log lines: valid structure, UTC times, enough context to act on, and no personal or secret data.

Try it: API Status Code Simulator Try it: JSON Lines Validator

Testing in production safely

Some checks only make sense in production: the post-deployment smoke test, synthetic checks, trying a feature flag with staff before customers see it, or comparing a canary with its control group. They add to testing before release and never replace it, and they are safe only with guardrails agreed in advance.

• Permission: test only systems your team owns or has written permission to test, agree the activity, time and owner, and tell the engineer on call.
• Test identities: use dedicated accounts, marked so that analytics, billing, reports and customer messages exclude them; never use real customer accounts or data.
• No real side effects: prefer read-only checks, cancel and clean up anything you create, and never trigger real payments, deliveries or messages to customers.
• Small blast radius: target flags at staff or test accounts, keep canary shares small, and have a stop switch you have tried before.
• No load tests on production without an explicit, scheduled agreement: real users share the same servers.

Try it: API Health Checker

Worked example: an alert on the order service

An online store in Riyadh (UTC+3) has an SLO for POST /orders: 99.5% of requests succeed within 2 seconds over 30 days, an error budget of about 1,000 failures a month. Release 4.2 goes out at 10:20. At 10:31 the alert below pages the developer on call, and the tester who tested the release joins.

The synthetic checkout check is still green because it pays by card. Splitting the success rate by payment method shows that only wallet payments fail, about 30% of orders. Filtering the logs by route and level gives the line below, and its trace id opens a trace in which the payment service waits 10 seconds for the wallet provider and gives up.

• Cause: the release copied a staging setting, so production called a wallet host it cannot reach. The provider_host field showed this in one line.
• Action: a rollback criterion in the release plan is met, so the team rolls back to 4.1 at 10:52 and closes the alert only when the SLI is normal again. About 90 orders failed, close to a tenth of the month’s error budget.
• Gaps for the follow-up review, the subject of the next lesson: the synthetic check pays only by card, nothing compared the production configuration with expected values, and no alert watched success per payment method.

ALERT 10:31 (UTC+3)  orders-api  page: developer on call
  POST /orders succeeded within 2 s: 72% over the last 10 min (SLO 99.5%)

Log line, pretty-printed (times in UTC):
{ "ts": "2026-10-04T07:29:12.418Z", "level": "error",
  "service": "payment-service", "route": "POST /orders",
  "trace_id": "7d2a91c4e0b54f3a9b18c6e2f4a05d71",
  "method": "wallet", "error": "provider_timeout",
  "elapsed_ms": 10001, "provider_host": "wallet.staging.internal" }

Trace 7d2a91c4...  total 10.05 s, failed
  api-gateway           POST /orders        10.05 s
    orders-service      create order        10.04 s
      payment-service   charge wallet       10.01 s   timeout
        wallet provider (no response)       10.00 s

Try it: Timestamp Converter

Frequently asked questions

References

• Monitoring Distributed Systems — Site Reliability Engineering (Rob Ewaschuk)Google / O'Reilly Media · 2016Google's SRE guidance on monitoring: black-box versus white-box monitoring and the four golden signals — latency, traffic, errors and saturation.
• Site Reliability Engineering: How Google Runs Production SystemsGoogle / O'Reilly Media · 2016Service level objectives, error budgets, monitoring the "four golden signals" and handling overload — the operational side of resilience.
• OWASP Cheat Sheet Series: LoggingOWASP FoundationWhat to log and what never to log (passwords, tokens, personal data), and how to keep logs tamper-evident.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.