Deployment Verification Testing

What deployment verification covers

A deployment changes more than code. It changes container images, configuration, secrets, database schema, infrastructure and routing, often in one step. Deployment verification checks the result of all of these together, in the target environment, with the question: is the new version running, correctly configured, reachable, and behaving at least as well as the previous one?

It helps to separate deploying from releasing. Deploying puts new code on servers; releasing exposes new behaviour to users. Techniques such as canary releases, blue-green switching and feature flags exist to separate the two, so a team can verify a deployment with little or no user exposure before releasing. The smoke suite is the same one described in the guide on smoke, sanity and regression testing, now pointed at the freshly deployed environment.

• Version check — the environment serves the version that was meant to be deployed, on every instance.
• Health and dependencies — services are up and can reach their databases, queues, caches and third parties.
• Critical paths — login, the main business transaction and its API respond correctly.
• Configuration and security — correct URLs, certificates, redirects and security headers; no debug mode.
• Behaviour under real traffic — errors, latency and business metrics comparable with the previous version.

Decide success and rollback criteria before you deploy

The most important part of deployment verification happens before the deployment: writing down what “good” looks like and what will trigger a rollback. Criteria decided during an incident are decided under pressure, by whoever is awake, with a strong bias towards “give it five more minutes”. Criteria decided in advance turn the rollback into a routine step instead of an argument.

A plan fits on one page. It names the rollout steps, the people on call, the signals and thresholds that define success, the conditions under which the team rolls back without discussion, how the rollback is done and how it will be verified. The example below is for a service deployed as a canary; the thresholds are illustrations, and your own should come from the service’s normal baseline.

Release verification plan: orders-service 5.9.0 (example)
Deployment:   canary 5% -> 25% -> 100%, 15 min per step; flag new_checkout OFF at deploy
People:       release engineer + tester on call (named in the release ticket)
Before:       backup confirmed; migration 0042 is additive only (expand step)

Post-deploy smoke:  smoke.sh against the canary, then against the production URL

Success signals, canary compared with baseline, at every step:
  - HTTP 5xx rate           <= baseline + 0.5 percentage points
  - p95 latency             <= baseline x 1.2
  - checkout conversion     within 10% of baseline (same window)
  - no new error signatures in the logs

Rollback criteria (any one => roll back, no debate):
  - smoke failure on canary or production
  - 5xx rate above 2% for 5 consecutive minutes
  - payment failures above baseline x 2
  - data integrity alert (duplicate or missing orders)

Rollback method:    redeploy image 5.8.3; new_checkout stays OFF; migration 0042 is kept (additive)
Rollback verified:  smoke.sh passes and /version reports 5.8.3 on every instance
Done when:          100% of traffic for 1 hour with all signals green

Post-deployment smoke tests

Run the smoke suite as the last step of the deployment job, against the environment just deployed, and make the pipeline fail if it fails. Keep it short (a few minutes), independent of test data that may not exist in production, and safe to run against production: read-only calls, or writes made with a dedicated test account whose data is excluded from reports and billing. Check negative behaviour too — a protected endpoint that suddenly returns 200 without a token is a deployment defect.

The script below is an example of the simplest useful form: HTTP checks with explicit expected status codes and a version check. Most teams later move the same checks into their API or UI test framework so that results are published in the same report format as other tests.

• Hit every instance or region if you can, not just the load balancer — a single stale instance is a classic partial-deployment defect.
• Bypass caches for at least one check; a CDN can serve a healthy-looking cached page while the origin returns errors.
• Make the health endpoint meaningful: it should report dependency status, not just return 200 because the process is alive.
• Check security headers, TLS and redirects after infrastructure changes; they are configured outside the code and are easy to lose.

#!/usr/bin/env bash
# Example post-deployment smoke script. Usage: ./smoke.sh https://staging.example.com
set -euo pipefail
BASE_URL="$1"
EXPECTED_VERSION="${EXPECTED_VERSION:-}"

check() {  # name, path, expected HTTP status
  local name="$1" path="$2" expected="$3" status
  status=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$BASE_URL$path" || true)
  if [ "$status" != "$expected" ]; then
    echo "FAIL $name: $path returned $status, expected $expected"
    exit 1
  fi
  echo "ok   $name ($status)"
}

check "health"            "/health"                200
check "home page"         "/"                      200
check "product API"       "/api/v1/products/1001"  200
check "auth still needed" "/api/v1/orders"         401

# Only if the service exposes its version
if [ -n "$EXPECTED_VERSION" ]; then
  deployed=$(curl -s --max-time 10 "$BASE_URL/version" || true)
  if [ "$deployed" != "$EXPECTED_VERSION" ]; then
    echo "FAIL version: got $deployed, expected $EXPECTED_VERSION"
    exit 1
  fi
  echo "ok   version ($deployed)"
fi

Try it: API Health Checker Try it: API Endpoint Tester Try it: Security Header Analyzer Try it: Redirect Checker

Synthetic checks after the deployment

A smoke test runs once; synthetic monitoring keeps running. Synthetic checks are scripted transactions — load the home page, log in with a monitoring account, search, add to cart, call a key API — executed on a schedule from outside the system, every few minutes, with alerts on failure or slowness. They catch problems that appear some time after a deployment: a memory leak, an expiring certificate, a scheduled job that breaks a cache, a third party that changed its behaviour.

Treat synthetic checks as tests: keep them in version control, review them with each feature change, give them dedicated test accounts and data, and make sure a failure reaches a person. A synthetic check that has been failing quietly for a week is worse than none, because it creates the impression of coverage. When you write the schedule as a cron expression, check it and its next run times before relying on it.

Try it: Cron Expression Generator Try it: Cron Expression Explainer Try it: Response Time Analyzer

Canary releases, blue-green and feature flags

Progressive delivery exposes a change to a growing share of traffic, with a check at each step. In a canary release, the new version first receives a small share of traffic — a few percent, or internal users — and its behaviour is compared with the baseline running the old version before the share is increased. Google’s SRE workbook describes canarying as comparing the canary population against a control, which is the key point for testers: compare like with like, over the same time window, not the canary against yesterday’s whole fleet.

Blue-green deployment keeps two identical environments; the new version is deployed to the idle one, verified there with the smoke suite, and traffic is switched over, with the old environment kept ready for an immediate switch back. Feature flags separate release from deployment inside the code: new behaviour ships switched off, is enabled for internal users, then for a percentage of customers, and can be switched off without a redeploy.

• Test both flag states, and the transition between them for users in the middle of a flow (a cart created with the flag off, checked out with it on).
• Verify targeting: the flag is on exactly for the intended users, regions or percentage, and off for everyone else.
• Run the smoke suite against the canary or the idle environment directly, before any real traffic reaches it.
• Keep both versions compatible during the rollout: APIs, message formats and database schema must work with old and new code at the same time.
• Remove flags after full release; long-lived flags multiply the combinations to test.

Try it: API Response Comparator

Monitoring signals to watch

During and after a rollout, verification relies on production signals rather than test results. The Google SRE book’s four golden signals are a sensible minimum: latency, traffic, errors and saturation. Add business signals, because many deployment defects produce no errors at all — orders simply stop, or totals are wrong. Compare each signal with the baseline for the same period and look for step changes that line up with the deployment time.

• Errors — HTTP 5xx rate, failed jobs, client-side JavaScript errors, new error messages that did not exist before the deployment.
• Latency — p95 and p99 of key endpoints, not only the average; slow successful requests are still a regression.
• Traffic — sudden drops can mean users cannot reach a feature; sudden rises can mean retry storms.
• Saturation — CPU, memory, connection pools, queue depth; a slow leak shows here hours after a clean smoke test.
• Business — sign-ups, logins, checkouts, payments, messages sent, compared with the same hour on previous days.
• Delivery health over time — change fail rate and failed deployment recovery time, two of the DORA metrics, show whether deployment verification and rollback are working across many releases.

Try it: Response Time Analyzer

Rollback and release validation

A rollback is itself a deployment and must be verified the same way: run the smoke suite again and confirm the version endpoint reports the previous version on every instance. Rehearse the rollback on staging for any release that changes infrastructure or data, and time it — a rollback that takes an hour is not a safety net for a defect that costs money every minute. Sometimes a forward fix is faster and safer than a rollback; decide which applies in the plan, not during the incident.

Database migrations are what most often make rollback hard. Once a migration drops a column, renames a table or rewrites data, the previous version of the code may no longer run against the new schema, and reversing the migration may lose data written since the deployment. The common mitigation is the expand-and-contract approach: first add new structures in a backward-compatible way, deploy code that works with both, migrate data, and only in a later release remove the old structures. Testing migrations — on a production-sized copy, with old code against the new schema, and with measurement of locks and duration — is covered in the guide on database transactions and migrations testing.

• Rollback criteria written before the deployment, with numeric thresholds and a named decision owner.
• Rollback procedure rehearsed and timed; the artefact of the previous version still available.
• Old code tested against the new schema whenever a migration ships with the release.
• Destructive migration steps (drop, rename, type change) shipped in a later release than the code that stops using them.
• Rollback verified by the same smoke suite and version check as the deployment.

Typical deployment defects

• Production returns 5.8.3 on two of six instances after deploying 5.9.0 (partial rollout).
• Health endpoint returns 200 while the database connection pool is exhausted.
• Smoke test passes against a cached CDN page while the origin returns 502.
• GET /api/v1/orders returns 200 without a token after gateway configuration change.
• Strict-Transport-Security header missing on the new load balancer.
• Canary shows three times the baseline 5xx rate on POST /orders for users with saved cards.
• Flag new_checkout enabled for 5% also shows the new flow to logged-out visitors.
• Rollback to 5.8.3 fails: old code cannot read orders with the new status PENDING_REVIEW.
• Synthetic login check failing for six days because its test account password expired.

Deployment verification checklist

• Verification plan written: rollout steps, success signals, rollback criteria, owners.
• Migrations reviewed for backward compatibility; old code tested against the new schema.
• Post-deployment smoke suite runs automatically and fails the pipeline on failure.
• Version confirmed on every instance or region.
• Security headers, TLS and redirects checked after infrastructure changes.
• Canary or idle environment verified before real traffic; compared with baseline at each step.
• Feature flags tested in both states and for correct targeting.
• Golden signals and business signals watched for the agreed period.
• Rollback rehearsed, timed and verified with the same smoke suite.
• Synthetic checks updated for the new behaviour and alerting a real person.

Frequently asked questions

References

• Canarying Releases — The Site Reliability WorkbookGoogle / O'Reilly MediaReleasing a change to a small share of traffic first and comparing it with the baseline before rolling out — deployment verification as practised in SRE.
• Site Reliability Engineering: How Google Runs Production SystemsGoogle / O'Reilly Media · 2016Service level objectives, error budgets, monitoring the "four golden signals" and handling overload — the operational side of resilience.
• DORA's software delivery performance metricsDORA (Google Cloud)The DORA delivery metrics — change lead time, deployment frequency, failed deployment recovery time, change fail rate and related measures — grouped by throughput and instability.
• Release It! Second Edition: Design and Deploy Production-Ready Software (Michael T. Nygard)The Pragmatic Bookshelf · 2018Stability patterns — timeouts, circuit breakers, bulkheads, back-pressure — and the failure modes load testing is meant to find.
• Continuous Integration (Martin Fowler)martinfowler.com · 2024The widely referenced description of continuous integration: integrate often, keep the build self-testing and fast, and fix a broken build immediately.
• GitHub Actions documentationGitHubHow workflows, jobs, runners and artifacts are defined in GitHub Actions — where automated tests run on each push or pull request.