CI/CD Testing

Continuous integration, continuous delivery and continuous deployment

Continuous integration (CI) is the practice of merging every developer’s work into a shared mainline frequently — at least daily — with each merge verified by an automated, self-testing build. Martin Fowler’s widely referenced description adds the rules that make it work: keep the build fast, fix a broken build immediately, and treat a red mainline as everyone’s problem. CI is a team habit supported by a tool, not the tool itself; a CI server that runs once a week on a long-lived branch is not continuous integration.

Continuous delivery extends CI so that every change that passes the pipeline is releasable: it is built once, tested in production-like environments and could be deployed to production at the push of a button. The decision to deploy remains a human or business decision. Continuous deployment removes that decision: every change that passes all automated stages goes to production automatically. The difference matters to testers because under continuous deployment there is no manual checkpoint left — whatever the pipeline does not check, nobody checks before users see it.

• Continuous integration — frequent merges to mainline, each verified by an automated build and tests.
• Continuous delivery — every passing change is deployable; a person chooses when to release.
• Continuous deployment — every passing change is deployed automatically; post-deployment verification becomes the last safety net.
• In all three, the automated tests are the evidence. A pipeline is only as trustworthy as the tests it runs and the gates it enforces.

Test stages in a pipeline and why the order matters

A pipeline arranges checks so that the cheapest, fastest and most likely to fail run first, and the slow, expensive and environment-dependent ones run later, only on changes that survived the earlier stages. This is the test pyramid applied to time: a developer should learn about a typo in minutes, not after a forty-minute browser suite. Fast feedback is not a convenience; when feedback is slow, people batch changes, and a failure in a large batch is harder to attribute to one cause.

The exact stages differ by team, but most pipelines converge on a similar order. Each stage should have a time budget; when a stage grows past it, the team either parallelises it, moves slow tests to a later stage, or removes duplicated coverage.

• Commit stage (minutes) — static checks (lint, type check, formatting, dependency and secret scanning), unit tests and component tests with dependencies replaced by test doubles. Runs on every push and pull request.
• Integration and API stage — the application against real databases, queues and service containers; API tests checked against the contract (OpenAPI). This is where most business rules are verified at reasonable cost.
• End-to-end / UI stage — a small set of critical user journeys through a real browser or device; the slowest and the most prone to flakiness, so kept deliberately small.
• Non-functional stage — a short performance smoke (a few minutes of load against key endpoints with thresholds on latency and error rate), security scans (static analysis, dependency vulnerabilities, a baseline dynamic scan), accessibility checks on key pages.
• Post-deployment stage — smoke tests and health checks against the environment just deployed, then synthetic monitoring. Covered in depth in the deployment verification guide.
• Scheduled stages — the full regression suite, long load tests and full security scans run nightly or weekly instead of on every change, and report to an owner who acts on failures.

Try it: OpenAPI → Test Cases Try it: Concurrent Request Test Generator Try it: Response Time Analyzer

Example: a pipeline with commit, API, E2E and deploy stages

The workflow below is an example written in GitHub Actions syntax. It runs static checks and unit tests first, then API tests against a throwaway PostgreSQL service container with migrations and synthetic seed data, then the browser suite split into four parallel shards, and on the main branch deploys to staging and runs a smoke script. Every test job uploads its reports and artifacts even when it fails (`if: always()`), because a failure without its evidence is the most expensive kind to investigate. GitLab CI and Jenkins express the same structure with stages and jobs in `.gitlab-ci.yml` or a Jenkinsfile.

• `needs` makes the order explicit: the E2E shards never start if unit or API tests fail.
• `concurrency` cancels a pull request’s running workflow when a newer push to the same pull request starts another, saving runner time. On main and the nightly schedule, which share the group of refs/heads/main, a running workflow is never cancelled, so a deployment is not interrupted halfway; but, as in every concurrency group by default, a run still pending in the group is cancelled when a newer one is queued.
• `timeout-minutes` turns a hung test into a visible failure instead of a job that waits for hours.
• The deploy token comes from the platform’s secret store; the database password is a literal only because the container lives for one job.
• The schedule runs on the default branch; cron times in GitHub Actions are in UTC unless the schedule sets a time zone.

# Example pipeline (GitHub Actions syntax). Adapt job names, scripts and versions to your project.
name: ci

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "0 1 * * *"   # nightly full regression, 01:00 UTC

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  commit-stage:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test:unit          # writes reports/junit-unit.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: unit-reports
          path: reports/

  api-tests:
    needs: commit-stage
    runs-on: ubuntu-latest
    timeout-minutes: 20
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: ci-only-password   # throwaway container, not a real secret
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    env:
      DATABASE_URL: postgres://postgres:ci-only-password@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm run db:migrate
      - run: npm run db:seed            # synthetic test data only
      - run: npm run test:api           # writes reports/junit-api.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: api-reports
          path: reports/

  e2e:
    needs: api-tests
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shard }}/4   # config starts the app
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: e2e-results-${{ matrix.shard }}
          path: |
            playwright-report/
            test-results/

  deploy-staging:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    needs: e2e
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v7
      - run: ./scripts/deploy.sh staging
        env:
          DEPLOY_TOKEN: ${{ secrets.STAGING_DEPLOY_TOKEN }}
      - run: ./scripts/smoke.sh https://staging.example.com

Try it: YAML Formatter Try it: Cron Expression Generator Try it: Cron Expression Explainer

Quality gates: criteria that can stop a change

A quality gate is a check with a pass/fail rule that decides whether a change may proceed to the next stage — merge, deploy to staging, promote to production. It is the automated form of exit criteria. A gate is useful when it is objective, fast to evaluate, tied to a real risk and enforced without exceptions; a gate that is routinely overridden teaches everyone to ignore it.

Be careful with vanity gates. A rule such as “line coverage must be at least 80%” is easy to measure and easy to satisfy with tests that execute code without asserting anything useful. Coverage is informative as a trend and as a way to find untested code; as a raw threshold without context it rewards the wrong behaviour. A more useful variant is coverage of the changed lines, reviewed together with the tests themselves. The same applies to “number of tests” or “pass rate 100% after retries”: they describe activity, not risk.

• All required checks green: build, lint, type check, unit, API and E2E tests — with no test skipped outside the quarantine list.
• No new critical or high vulnerabilities in dependencies or static analysis, or a recorded, time-limited exception.
• API contract unchanged, or the change is backward compatible (no removed fields, no new required inputs) unless versioned.
• Performance smoke within thresholds, for example p95 latency under 400 ms and error rate under 1% at the smoke load, compared with the last main-branch baseline.
• Database migrations applied cleanly to a copy of the current schema and, where the process requires it, a tested rollback or forward-fix plan.
• For production promotion: post-deployment smoke on staging passed and no open blocker defects on the release.

Test reports and artifacts

A pipeline’s result is only as useful as what it leaves behind. JUnit XML is the common interchange format for test results: most test runners can write it and most CI systems can read it. It is a de facto format rather than a formal standard, so attributes vary slightly between tools, but the core structure — test suites containing test cases, each with an optional failure, error or skipped element and a duration — is widely supported. GitLab, for example, reads it through `artifacts:reports:junit` and shows new and fixed failures in the merge request; Jenkins publishes it as test results with history per test.

Beyond the pass/fail summary, keep the artifacts a person needs to diagnose a failure without re-running it: logs of the application and the test runner, screenshots and videos for UI failures, traces (a Playwright trace records actions, network and DOM snapshots and can be opened after the run), HTTP request/response captures for API failures, and coverage reports. Set a retention period so artifacts are kept long enough to compare with previous runs but do not accumulate forever, and make sure they contain no real personal data or secrets.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="3" failures="1" skipped="1" time="4.21">
  <testsuite name="orders-api" tests="3" failures="1" skipped="1" time="4.21">
    <testcase classname="orders.create" name="returns 201 for a valid order" time="0.84"/>
    <testcase classname="orders.create" name="rejects a negative quantity with 400" time="0.31">
      <failure message="expected status 400, received 500">AssertionError: expected 400, received 500
    at tests/orders.spec.ts:42</failure>
    </testcase>
    <testcase classname="orders.refund" name="refunds a captured payment" time="0">
      <skipped message="quarantined: FLAKY-112"/>
    </testcase>
  </testsuite>
</testsuites>

Environments, test data and secrets

Pipeline tests need somewhere to run. Ephemeral environments — created for one pipeline run or one pull request and destroyed afterwards — give each change a clean, isolated stack: service containers for databases and queues, or a short-lived preview deployment per branch. They remove a major cause of false failures: tests interfering with each other through shared data. Long-lived shared environments (a staging that mirrors production) are still needed for integration with real third parties and for release rehearsal, but they drift, so check their versions and configuration as part of the pipeline.

Test data should be created by the pipeline: migrations build the schema, seed scripts load a known synthetic baseline, and each test creates the specific records it needs, with unique identifiers so parallel runs do not collide. Never copy production personal data into CI. Secrets — API keys, deploy tokens, database passwords for shared environments — belong in the CI platform’s secret store or an external vault, scoped to the environments that need them, masked in logs and rotated. On GitHub Actions, workflows triggered by pull requests from forks do not receive repository secrets by default, which is a safety feature to keep, not a problem to work around.

• One environment per run where possible; shared environments only where they add realism you cannot get otherwise.
• Schema from migrations, baseline from seed scripts, per-test data with unique ids.
• Mock or stub third parties at their boundary in early stages; test the real integration in a later, smaller stage.
• Secrets never in the repository, never printed, never in artifacts.

Try it: Test Data Factory

Parallel execution and sharding

When a suite outgrows its time budget, run it in parallel. Within one machine, runners use several workers; across machines, the suite is split into shards, each running a subset on its own runner, as in the example above. Sharding cuts wall-clock time roughly in proportion to the number of shards if the shards are balanced; splitting by recorded test duration balances better than splitting by file count.

Parallelism exposes hidden dependencies between tests: two tests that edit the same user, a test that relies on another having run first, a fixed port or file path. These are test design defects, and they show up as intermittent failures. Isolate data per worker, avoid order dependencies and merge the per-shard reports into one result so the gate evaluates the whole suite.

Handling flaky tests in the pipeline

A flaky test passes and fails on the same code. In a pipeline it does double damage: it blocks good changes, and it teaches people to press “re-run” until green, after which real failures are ignored too. Martin Fowler’s article on non-determinism describes quarantine as a response: move the flaky test out of the gating suite into a separate, still-running set, so the pipeline stays trustworthy while the test is fixed. The causes and fixes are covered in the flaky tests guide; the pipeline needs a policy.

• Retries: allow at most one automatic retry, report a test that passed only on retry as flaky rather than passed, and never retry a whole job silently.
• Quarantine: a quarantined test still runs and reports, does not gate, has an owner and a ticket, and has a deadline to be fixed or deleted.
• Tracking: record every flaky occurrence (test id, commit, error, link to the run) so the team can rank the worst offenders; flake rate per test is the number to watch.
• Limits: cap the quarantine list size; a growing quarantine is a signal that test design needs attention, not a place to park problems.

Pipeline failure analysis: a triage flow

A red pipeline has four common explanations: a product defect introduced by the change, a test that is wrong or outdated, an environment or infrastructure problem, or a flaky test. Treating every red build as a product bug wastes developer time; treating every red build as flakiness lets defects through. A short, shared triage flow gets the classification right in minutes. The first question is always the same: what exactly failed, and what did the artifacts record?

• Read the first error, not the last; later failures are often consequences.
• Compare with the last green run of the same job: changed dependencies, runner image, data or configuration.
• Keep a failure log with the classification of each red build; over a month it shows whether the pipeline suffers mainly from defects, flakiness or environments.
• Product defects found by the pipeline get a normal bug report with the failing test, the commit and the artifacts attached.

Pipeline failed
|
|-- 1. Did the job fail before any test ran? (dependency install, build, image pull, runner lost)
|      yes -> PIPELINE / INFRASTRUCTURE problem. Owner: platform. Fix, then re-run.
|
|-- 2. Did it fail during setup? (DB not ready, migration error, service unhealthy, expired secret)
|      yes -> ENVIRONMENT problem. Fix the environment; do not touch the tests.
|
|-- 3. Re-run the failed tests once on the SAME commit. Do they pass now?
|      yes -> FLAKY candidate. Log it (test id, commit, error, run link), open or update its ticket,
|             quarantine if it repeats. The change is not proven safe by the green re-run.
|      no  -> deterministic failure, continue.
|
|-- 4. Does the same test fail on main without this change?
|      yes -> PRE-EXISTING break or environment drift. Notify whoever owns main.
|      no  -> caused by this change, continue.
|
`-- 5. Is the product behaviour wrong, or is the test's expectation out of date?
       product wrong       -> PRODUCT DEFECT. Bug report, merge blocked.
       intended new change -> TEST UPDATE in the same change, reviewed like code.

Try it: AI Error Message Analyzer

What a tester does in CI/CD

In teams with a pipeline, testers spend less time executing scripted checks by hand and more time deciding what the pipeline checks, keeping it trustworthy and testing what it cannot. The work below is a practical summary; which tests to automate at which level is covered in the manual versus automated testing guide, and the metrics that show whether delivery is healthy — change fail rate and failed deployment recovery time, among the DORA metrics — are worth reviewing with the team every release.

• Map risks to stages: which check, at which stage, protects against which failure.
• Design and review the regression suite, including its selection rules for pull requests versus nightly runs.
• Define quality gates with the team and challenge vanity gates.
• Own the flaky-test policy and the failure log; triage red builds with developers.
• Write post-deployment smoke checks and agree rollback criteria before release.
• Keep exploratory testing for new features — no pipeline replaces a person looking for what nobody thought to automate.

Try it: Regression Scenario Generator Try it: API Health Checker

Frequently asked questions

References

• Continuous Integration (Martin Fowler)martinfowler.com · 2024The widely referenced description of continuous integration: integrate often, keep the build self-testing and fast, and fix a broken build immediately.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• GitHub Actions documentationGitHubHow workflows, jobs, runners and artifacts are defined in GitHub Actions — where automated tests run on each push or pull request.
• GitLab CI/CDGitLabHow pipelines, jobs and artifacts are defined — where an automated regression suite runs on every change.
• Unit test reports — GitLab CI/CDGitLabPublishing JUnit XML results as pipeline artifacts so failed tests appear in the merge request and can be compared between runs.
• Pipeline — Jenkins User HandbookJenkins ProjectJenkins pipelines defined as code in a Jenkinsfile: stages, steps, parallel execution and test result publishing.
• Eradicating Non-Determinism in Tests (Martin Fowler)martinfowler.com · 2011Why flaky tests destroy a suite's value, their common causes (isolation, asynchrony, remote services, time, resource leaks) and quarantine as a response.
• DORA's software delivery performance metricsDORA (Google Cloud)The DORA delivery metrics — change lead time, deployment frequency, failed deployment recovery time, change fail rate and related measures — grouped by throughput and instability.