Manual vs Automated Testing

Overview. Six checks are automated: K1 (UI), K2, K3 and K4 (API), K8 (UI, on a browser grid) and K9 (a load script). The first five pass all three questions; K9 runs only before a date and after risky changes, but a load test cannot be done by hand at all, so it is automated as soon as it has an agreed expected result. Four stay with a person: K5 and K10 are judgement, K6 is exploratory and K7 runs once. "Not yet" is the answer for automating group booking: it waits until the design is final (see K6). The pyramid decides the level: a check lives at the lowest level where it can be expressed, so K2, K3 and K4 are API tests and need no UI test, and of the automated checks only K1 and K8 drive a browser. If your classification differs, check it against the reasons below; a different choice with a written reason is acceptable, a check automated "because it can be" is not.

K1 — automate, UI level, the one end-to-end journey in the pipeline. It runs on every change in GitLab CI and before every release (frequency), ends in a definite result (a ticket with a QR code for the two chosen seats), and apart from the payment step the booking screens do not change in this release (stability). The payment step will change once, on 15 Oct: plan that maintenance for the switch day (the new provider’s test card and screens) rather than calling the flow unstable. Keep the script to the journey; prices, seat status and availability are checked below it.

K2 — automate, API level, on every commit. An exact request, an exact response and an OpenAPI specification to compare against is the ideal automation target, and the mobile apps depend on this contract, so a renamed or missing field breaks them on the day it ships. The check takes seconds and belongs next to the unit tests in GitLab CI.

K3 — automate, API level, data-driven, on every commit; not a UI test, because the API decides the price and the web app only shows it. A developer unit test of the price calculation is an equally valid level, since it is the lowest level that can express the rule. PR-7 has been unchanged for two years (stability), the expected price is exact (determinism) and it is checked on every release (frequency). The rows are the boundaries between the age bands and the two kinds of day, prices in SAR: on a day other than Tuesday, age 2 → free and no seat, 3 → 25.00, 12 → 25.00, 13 → 45.00, 64 → 45.00, 65 → 30.00; on a Tuesday the same ages → free, then 20.00 for the other five. Add rows at the edges of the day in the cinema’s time zone: Monday 23:59, Tuesday 00:00, Tuesday 23:59 and Wednesday 00:00. Before writing those expected results, ask the product owner what "Tuesday" means — the day of the screening or the day of the booking — and record the answer: a check without a definite expected result is not ready to automate.

K4 — automate, API level, in two cases: a seat held while one customer pays is refused to a second customer (two requests one after the other), and two bookings for the same seat sent at the same moment end with exactly one success and one refusal (for example 201 Created and 409 Conflict). The outcome is definite even though the timing is not, and the check matters on every release because a double-booked seat costs refunds and trust. By hand, two testers with two browsers cannot make their clicks land in the same millisecond, so the second case would pass without the race ever being tested; generating concurrent requests is automation by definition. Run it nightly or on every commit of the booking service.

K8 — automate, UI level, nightly and before each release, not on every commit. It is the K1 journey run again on the four browsers, on a desktop and on a phone, through a browser grid with real or emulated devices: one script, eight configurations. By hand it would be the same journey repeated eight times every two weeks — repetitive, deterministic work, exactly what the rule sends to automation — and because K1 is already scripted, the extra cost is configuration rather than new code. It is slower than the rest of the suite, which is why it does not block every commit.

K9 — automate, as a load script that sends the HTTP requests of the seat map and the checkout without a browser; a load test is automation by definition, since 2,000 customers at 10:00 cannot be produced by hand. It runs before 24 Oct and again after any change to the seat-holding or checkout code, against a staging environment sized like production — not against production, and never against a system the team does not own, such as the payment provider, without written permission. Give it a definite expected result before it runs, for example the response time and error rate the team agrees to accept for the seat map and for the checkout; otherwise it produces numbers but no verdict.

K5 — manual, scripted, each release, on a real phone with the Arabic locale. "Usable" and "reads correctly right-to-left" are judgements with no expected value to compare against, so the check fails determinism. Write a short checklist (seat labels readable, seat numbers in the same order as in the hall, the selected seat visible, the price in the right place) so that the check is repeatable. A screenshot comparison can later guard the layout against unintended changes, but it only shows that something changed; whether the result is usable stays a judgement for a person.

K6 — exploratory session, time-boxed and charter-driven, for example: "Explore group booking of 6 to 20 seats, including changing seats and paying, to discover confusing or broken behaviour", in a 60–90 minute session with notes. There is no definite expected result yet — the expected behaviour is still being discovered — and the screens are unstable, so this is the first pass over a new feature that the guide gives to a person. When the design is final, write the cases worth keeping and hand the stable ones to automation, usually one group-booking journey next to K1.

K7 — one-off, by hand or with a throwaway query, not in the suite. It runs once, on 15 Oct, so it fails frequency, and a script would cost more than the check. Do it carefully all the same: record the ids of the 1,214 orders before the switch, compare the list — not only the count — after it, and keep the result with the release notes.

K10 — manual scripted review, with the deterministic part split off. Whether a message is understandable and tells the customer what to do cannot be compared with an expected value; a tester reads each message in context, in English and in Arabic, once per release or whenever a text changes. The deterministic part goes to automation: the API returns the agreed error code (an API check next to K2), and every error state shows a message at all (a step in the UI journey next to K1).

Q1. Automation replaces scripted manual regression — repeating known checks by hand — and it should: K1, K2, K3 and K8 are exactly that, and K9 and the race case of K4 cannot be done by hand at all. It cannot replace exploratory testing (K6), usability and localisation judgement (K5, K10) or the review of requirements, because a script only checks what someone anticipated when it was written. A team that automates everything ends up with a strong regression suite and a growing number of production defects nobody wrote a script for. The realistic promise is different: once the suite guards what already works on every change, the testers’ time moves ahead of it — requirements, new features, exploratory sessions.

Q2. First, the time from a commit to the regression result, which should be minutes and should fall: today the unit tests answer in 4 minutes but the UI suite answers the next morning, so a developer who breaks the checkout learns it a day later. Second, the share of production defects that an existing automated check could have caught, which should fall over time; if it stays high, the suite does not cover what matters, however green it is.

Q3. Fix or remove the flaky seat-map check this week: a suite that is red once a week when nothing is broken trains the team to ignore red, and the fact that nobody re-runs it shows that this has already happened. Go through the 38 checks and move every check of a business rule — prices, seat status, availability — out of the browser and down to the API level, where it runs in seconds; a rule tested at the API level does not need a UI test. Keep in the UI suite only the journeys that can fail in the UI, starting with K1 and later group booking, so that it runs in minutes on every change in GitLab CI instead of once a night; only the cross-browser run (K8) stays nightly. Review the suite for duplicates and dead checks each quarter, and give the automation engineer the rest of this list, in this order, as the first month’s work.