Contract Testing

Why it matters

Services and apps are released independently. A provider change can pass all its own tests and still break a consumer, because those tests do not know what the consumer reads. Without contract tests the break appears only when both run together: days later in an end-to-end environment, or in production, where old mobile app versions keep calling the API.

For the tester, contract tests answer a precise release question: will this provider version work with every consumer version it will meet? When it will not, they name the consumer, the interaction and the field.

Consumers, providers and the contract

Martin Fowler describes a contract test as a check that a test double, the stub or mock that other tests use in place of a service, still behaves like the real service. Contract testing extends this to both sides: each is tested against the same written contract.

For an HTTP API, each interaction in a contract states:

• The request: method, path, query parameters, the headers that matter, and the body.
• The expected response: status code, headers, and the body fields with their types or patterns.
• The provider state: the data that must exist for the interaction to make sense, such as "order ORD-1042 exists".

Consumer-driven contract testing

In consumer-driven contract testing the consumers write the contract through their own tests, so the provider knows exactly which fields each consumer uses and can change the rest freely. The cost: every consumer team writes and maintains these tests and the provider runs them, which suits services within one organisation, not a public API with unknown consumers.

Pact, an open-source tool for this approach, works in five steps (its contract file is shown below):

• Consumer test: the consumer's client code calls a mock provider that answers as the test declares, proving the consumer handles that response.
• Contract file: the run writes each interaction into a pact: the requests the consumer makes and the response fields its test expects, which should be only the fields it reads.
• Publish: a broker stores the contract with the consumer's version.
• Provider verification: the provider's pipeline sets up each provider state, replays each request against the real provider and compares the response with the expected one, by type or pattern where exact values do not matter; extra fields are allowed by default.
• Deployment check: before either side deploys, the broker confirms that this version was verified against the versions in the target environment (can-i-deploy in Pact).

{
  "consumer": { "name": "mobile-app" },
  "provider": { "name": "orders-api" },
  "interactions": [
    {
      "description": "a request for an existing order",
      "providerStates": [{ "name": "order ORD-1042 exists and is shipped" }],
      "request": { "method": "GET", "path": "/orders/ORD-1042" },
      "response": {
        "status": 200,
        "headers": { "Content-Type": "application/json" },
        "body": { "id": "ORD-1042", "status": "shipped", "total": 149.5 },
        "matchingRules": {
          "body": {
            "$.id": { "matchers": [{ "match": "regex", "regex": "^ORD-[0-9]+$" }] },
            "$.total": { "matchers": [{ "match": "type" }] }
          }
        }
      }
    }
  ],
  "metadata": { "pactSpecification": { "version": "3.0.0" } }
}

Try it: Response Contract Comparator

Specification-based contract testing with OpenAPI

Here the provider's OpenAPI document is the contract; some teams call this provider-driven contract testing. The check from the API schema validation lesson runs on every response in the provider's tests, errors included, and consumers build their mocks from the same document.

Its blind spot: if code and document change together, the provider still matches its own document. So each new version of the document is also compared with the production version, and breaking changes are flagged:

• Breaking: removing or renaming an operation or a response field, changing a type, making an optional request field required, rejecting request values that were accepted before.
• Usually safe: adding an operation, an optional request field or a response field, if consumers ignore unknown fields.
• Needs a decision: a new enum value or null in a response; a client that handles only the known values can fail.
• The limit: the document says what the provider offers, not which consumer uses what, so a "breaking" change may affect nobody.

Try it: API Contract Validator Try it: OpenAPI Validator

Worked example: a field rename in the Orders API

The Orders API has two consumers with contracts: the mobile app reads id, status and total; the delivery service reads id, status and deliveryCity. A release renames total to grandTotal, turns createdAt from a text date into a number and adds an optional giftMessage. The OpenAPI document was updated too, so validation against it passes.

• Comparing the two documents flags two breaking changes: total removed and createdAt retyped.
• Provider verification fails for the mobile app at $.total; the delivery service passes, because it uses neither field.
• No contract reads createdAt, so the type change is safe for both known consumers, though not for consumers outside the broker.
• The deployment check blocks the release. The team returns both total and grandTotal, marks total as deprecated, and removes it when no supported app version needs it.
• An end-to-end run would have shown the rename only after both services were deployed, as an order screen without an amount.

Try it: JSON Diff Try it: OpenAPI Viewer

What contract tests catch compared with end-to-end tests

An end-to-end test drives a journey through every real service, so it finds an incompatible interface only after all services are deployed together, it is slow, and its failure must be traced through many interfaces. A contract test checks one interface in seconds, in each side's own pipeline, and names the consumer, the interaction and the field.

• Caught before deployment: removed, renamed or retyped fields, changed paths, methods, status codes or headers, and requests the provider no longer accepts.
• Missed: wrong values, business rules and meaning. A wrongly calculated total, or riyals sent where halalas are expected (the integration testing lesson), still has the right type.
• Missed: configuration and deployment, such as a wrong base URL, missing credentials or an expired certificate.
• Missed: anything in no contract, such as an enum value or an error that no provider state produces.

Frequently asked questions

References

• Contract Test (Martin Fowler)martinfowler.com · 2011Contract tests: periodic checks that the calls made against a test double return the same results as the same calls to the real external service.
• Introduction — Pact DocsPact FoundationConsumer-driven contract testing with Pact: each application is tested in isolation against a contract generated while the consumer's tests run.
• OpenAPI SpecificationOpenAPI InitiativeThe standard for describing REST APIs: paths, operations, parameters, schemas, responses and security schemes.
• The Practical Test Pyramid (Ham Vocke)martinfowler.com · 2018A widely cited explanation of how to balance unit, integration, contract and UI tests in an automated test portfolio.