Compatibility Testing

Why it matters

Compatibility defects live between systems, so no single team owns them. Each side passes its own tests: the app sends valid JSON, the partner accepts it, and only together does the order arrive with the wrong date. They also cluster on release day, when a new server meets thousands of devices still running last month’s app, accounts full of data in the old shape and partners who changed nothing.

For the tester the danger is false coverage: a regression suite that runs on a fresh installation, against mocks (simulated partner systems), on the newest platform can be green while real users fail. Naming the product’s compatibility relationships, choosing which to test from usage data and recording what was left out turns "it works here" into a statement the team can rely on.

Compatibility in ISO/IEC 25010

ISO/IEC 25010:2023 describes compatibility as the degree to which a product can exchange information with other products, perform its functions while sharing a common environment and resources with them, or both. The ISTQB Foundation syllabus lists compatibility among the characteristics that non-functional testing evaluates. Its two sub-characteristics, the characteristic that holds platform support, and the relationship between versions each lead to a different kind of test:

• Interoperability: systems exchange information and can use what was exchanged. A delivered file is not enough; the receiver must read the amounts, dates and codes with the meaning the sender intended.
• Co-existence: the product performs its functions while sharing an environment and resources with other products, without harming them. It must survive its neighbours and must not break them.
• Flexibility: adaptability covers running on other operating systems, browsers and devices; installability and replaceability cover installing the product and a new version replacing an old one, which is where upgrade testing belongs.
• Backward and forward compatibility: the standard has no separate term for compatibility between versions of one product. In practice it is interoperability between versions, such as an old app talking to a new server, plus replaceability.

Interoperability: test the exchange, not just the sender

Interoperability tests follow data across a boundary and check both sides: what was sent, what arrived and what the receiver did with it. First list every connected system, from the architecture diagram, the integration settings and the keys the product holds; forgotten integrations are the untested ones. Then cover four kinds of exchange:

• Connected systems: create a record in your product and confirm it in the other system field by field: amounts with currency and decimals, dates with their time zone, status codes mapped to the receiver’s codes, Arabic names intact. Then the unhappy paths: the partner is slow, rejects the record or receives it twice.
• Import and export formats: open your exports in the programs customers really use, and import files those programs produce. Vary the encoding (UTF-8 with and without a byte order mark), the separator (RFC 4180 describes comma-separated CSV, but spreadsheets in regions with a decimal comma often write semicolons), date and number formats, line breaks inside quoted fields, and an empty file.
• API and schema versions: compare the old and new contract, such as the OpenAPI description (a machine-readable list of the API’s endpoints and fields) or the JSON Schema of a response (the rules its fields must follow). Adding an optional field is usually safe; removing or renaming a field, changing its type, making an optional request field required or adding an enumeration value can break clients already released. Replay the previous client’s requests unchanged.
• Third-party integrations: payment, maps, sign-in and SMS providers change on their own schedule. Test in the provider’s sandbox (its test environment) at the API version your product pins, read its change log and deprecation notices, and check what your product does while the provider is down.

Pharmacy order response, v1 (read by every released app)
{ "id": "A-1042", "total": 149.5, "deliveryDate": "2026-10-04" }

Pharmacy order response, v2 (proposed)
{ "id": "A-1042", "total": "149.50", "currency": "SAR",
  "delivery": { "date": "2026-10-04", "slot": "AM" } }

Change                        Effect on released clients
"currency" added              usually safe if clients ignore unknown fields
"total" number -> string      breaking: arithmetic and validation fail
"deliveryDate" moved          breaking: the field old clients read is gone

Try it: Response Contract Comparator Try it: JSON Schema Validator Try it: CSV Dataset Generator

Co-existence: sharing the environment

Co-existence tests run the product next to what really shares its device, browser or server, and check both directions: your product keeps working, and so do its neighbours. These failures are rarely in the requirements, so take the neighbours from support tickets, error logs and the company’s standard device setup:

• Shared resources: a camera, microphone or audio output another app is using; a file, port or printer another program holds; low storage or memory; a database or queue shared with other services. Your product should wait, explain or recover, and release what it took.
• Other software: antivirus scanning your files, a VPN changing the network, a screen reader, a phone call interrupting a payment.
• Browser extensions: ad blockers can hide elements that match their filter lists, such as a banner with a class like ad-banner; password managers and translators insert icons or rewrite text; dark-mode extensions recolour pages. Test the critical journeys with the extensions users report, in a separate browser profile.
• Side-by-side versions: two versions installed on one computer, a test build next to the store build on one phone, or two tabs where one still runs the old web app after a deployment. Check shared settings, files and local storage, and which version opens a link.

Support policy, backward and forward compatibility

Users never update together. On release day a new server meets every installed app version, devices on the oldest supported operating system and data written by earlier versions. A written support policy says which of these must work: the minimum operating system versions, the browsers, how many app versions the server still serves (often the current and the previous, N and N-1) and for how long, and what an unsupported user sees, such as a clear update prompt rather than a silent error.

Backward compatibility means the new version works with what older versions produced and with the older clients still in use. Forward compatibility is the mirror image: an older version copes with data from a newer one, usually by ignoring fields it does not know; rollbacks depend on it. Test upgrade paths from the old world, never only from a clean installation:

• Old data, new code: upgrade with realistic accounts (orders, saved addresses, a half-finished form) and open old exports and bookmarks. Everything must still be there, correct and usable.
• Old clients, new server: run each supported client version, unchanged, against the new server, and check the update prompt in the first unsupported one.
• Settings migration: language, notification and accessibility settings, saved sign-in and preferences must survive the upgrade, including a jump over several versions, such as 3.8 straight to 4.2.
• New data, old code: save data with the new version, then roll back or open it in the previous client; the old version should ignore what it does not know, not crash or overwrite it.

Try it: JSON Diff

Worked example: one coverage matrix from usage data

A pharmacy chain sells through a website and Android and iOS apps. Release 4.2 changes the order API and adds a delivery slot; after the contract comparison in the interoperability section, the team kept total as a number and kept deliveryDate, so released apps can still read the response. Instead of separate lists, the tester builds one compatibility coverage matrix whose rows come from evidence: 90 days of analytics, the client-version header in the API log, the app stores’ version statistics, the integration list and support tickets. Each row gets a depth, what is left out is written down, and inside a row the remaining parameters, such as language and payment method, are combined by pairwise selection, so that every pair of values appears in at least one test, as in the lesson on pairwise testing. The numbers are illustrative.

The platform rows reuse earlier work instead of repeating it: the lesson on responsive testing supplies the viewport widths and breakpoints, the lesson on cross-browser testing supplies the engines and the support tiers (Tier 1 runs the critical journeys and the regression set on every release, Tier 2 a smoke set of critical journeys, Tier 3 only best-effort checks), and the lesson on device fragmentation, later in the mobile module, shows how to choose phones. This matrix adds what those lessons do not cover: client versions, connected systems, formats, neighbours and upgrades, so that one page answers what was tested, against what, and what was skipped. The plan found three defects:

• Old client, new server: app 4.1, behind 31% of orders, omits the new deliverySlot field from its order request (the response returns the slot as delivery.slot). API 4.2 treats the field as required and answers 400 Bad Request instead of applying the default, the request-side break the interoperability section warns about. This blocks the release.
• Interoperability: the new export to the stock system writes dates as 04/10/2026, day first, and the stock system reads 10 April. Both systems passed their own tests; only the field-by-field check found it.
• Co-existence: a popular password-manager extension places its icon over the show-password button on the sign-in page. Neither product is wrong alone; the team moves the button.

Release 4.2 compatibility coverage matrix (illustrative numbers, last 90 days)

Area          Row                                     Evidence          Depth
Platforms     Tiers 1 and 2, cross-browser plan       analytics         per tier
              Android: oldest supported, current      store statistics  critical journeys
              iOS: previous and current               store statistics  critical journeys
Clients       app 4.2 (new)                           -                 full regression
              app 4.1 (31% of orders)                 API client log    critical journeys
              app 4.0 and older (2%, unsupported)     API client log    update prompt
Interop       payment provider, pinned API version    integration list  pay, fail, refund
              stock system export (CSV)               integration list  field by field
              SMS provider                            integration list  send, outage
Co-existence  password-manager extension              support tickets   sign-in journey
Upgrade       4.0 and 4.1 to 4.2, real accounts       -                 data, settings
Rollback      4.2 data opened in app 4.1              -                 forward check
Left out (recorded in the test plan): Firefox on Android (0.3%), tablets in landscape

Try it: Pairwise Testing Generator Try it: User-Agent Parser

Frequently asked questions

References

• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality modelISO/IEC · 2023The product quality model: functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability and flexibility.
• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• OpenAPI SpecificationOpenAPI InitiativeThe standard for describing REST APIs: paths, operations, parameters, schemas, responses and security schemes.
• RFC 4180: Common Format and MIME Type for Comma-Separated Values (CSV) FilesIETF · 2005The commonly implemented CSV rules: comma separators, CRLF line ends, quoting of fields that contain commas, quotes or line breaks.