Bug Hunting and Risk-Based Exploration

Why it matters

Testing time is always limited, and two of the seven testing principles explain why aiming matters: exhaustive testing is impossible, and defects cluster together. Exploration that wanders across the product spends most of its time where nothing is wrong; exploration aimed at risk spends it where a failure would be expensive — money, permissions, data that cannot be recovered.

A bug bash adds many different eyes in a short time: a support agent tries what customers complain about, a designer notices layout problems, a developer probes failure modes they know from the code. For a tester, organising one is a visible way to lead quality work — but the value comes from the preparation that aims it and the report that turns raw findings into decisions, not from the event itself.

Risk-based exploration: aim before you explore

Risk-based exploration applies risk-based testing to exploratory work: the risk level decides where you explore first, how deeply and with how many people. Start from the product risks of the release, rated by likelihood and impact as in risk-based testing. For one feature you do not need a full risk register; a short ranked list agreed with the product owner and a developer is enough.

Then turn each high risk into charters with the template from session-based test management, and let the level set the depth. A very high risk may get several sessions, a second tester or a bug bash; a high risk at least one full session; a medium risk a short session if time allows; a low risk a short tour or nothing at all, as long as that decision is recorded. Likelihood tends to rise with these signals:

• New or rewritten code, and code that many people changed in this release.
• Complex rules: money, dates and time zones, permissions, flows that can be done in several orders.
• Integrations: payment providers, SMS and email, other teams’ APIs, and what happens when they are slow or fail.
• Defect clusters: components with many defects, reopened defects or production incidents in past releases.
• Customer signals: support tickets, app store reviews and analytics showing where users struggle.
• Gaps: weak automated coverage or vague acceptance criteria.

Try it: Risk-Based Test Generator

Heuristics and tours as hunting tactics

The exploratory testing guide explains heuristics, rules of thumb that suggest what to try next, and tours, themed routes through the product. In a hunt you choose them by the risk you are targeting, so that the tactic attacks the feared failure instead of repeating the happy path, and you keep the list of typical errors from error guessing at hand. A quick recap, matched to common risks:

• Wrong money — vary the inputs: discounts, rounding, zero and maximum amounts, totals that do not divide evenly. Despite its name, the money tour is not the tactic here: it visits the features that sell the product, the ones shown in demos.
• Lost or inconsistent data — the FedEx tour: follow one record through every screen, email, export and API that touches it.
• State conflicts — vary states and sequences: two tabs, two devices, back and forward, submit twice; the obsessive-compulsive tour repeats one action until something gives.
• Failing dependencies — the saboteur tour and interruptions: a slow or lost network, timeouts, a payment declined in the sandbox.
• Wrong access — take a persona: a removed member, a guest, a user whose session expired.
• Display and localisation — the supermodel tour: Arabic and English, right-to-left layout, long names, zoom.
• Neglected corners — the back-alley tour: settings, error pages and limits nobody demonstrates.

Try it: Edge Case Generator Try it: Special Character Dataset Generator

Organising a bug bash

A bug bash brings people with different knowledge to the same build for a short, intense hunt. Its value depends on preparation: without it, a dozen people spend the first half hour logging in, test the same happy path and report bugs that are already known. A tester usually acts as the facilitator; the product owner agrees the goal, and the development lead agrees to freeze the build and keep time free for fixes.

Hold it early enough to fix what it finds. A bug bash the day before release usually leaves time to list the problems, not to fix them.

• Goal and scope — one sentence on what the bash is for, the areas in scope ranked by risk, what is out of scope, and the list of known issues so that nobody reports them again.
• Environment and data — a stable build in an agreed test environment, test accounts with the roles each charter needs, synthetic data and sandbox payment cards. Never real customer data.
• Charters and pairs — one charter per pair, highest risks first; where you can, pair a tester with a non-tester so that different knowledge meets.
• Reporting — one shared sheet, or a tracker label, with fixed fields: reporter, charter, title, steps, expected and actual result, evidence, device and build. Raw findings go there first; triage decides what becomes a defect report.
• On the day — a short kick-off on goal, scope, rules and reporting; a time box such as 60 to 90 minutes; a facilitator who answers questions and watches for duplicates and blockers; a wrap-up round on the most important finding and what nobody reached.
• Afterwards — triage the same day or the next morning: remove duplicates, reproduce, set severity and priority, file the defects linked to the bash, then report the outcome.

Try it: Fake User Generator Try it: Test Scenario Generator

Worked example: a bug bash before a family-account release

A fictional clinic booking app is adding family accounts: the account holder adds up to five members, books appointments for them and pays from a shared wallet; cancellations more than 24 hours ahead are refunded to the wallet. Two weeks before release, the QA lead ranks the risks with the product owner and a developer and runs a 90-minute bug bash for 12 people in six pairs.

Two pairs take the wallet, because a double charge or a wrong refund is the highest risk. The other pairs take the last free slot booked by two members at once, privacy after a member is removed, the 24-hour cancellation rule, and the Arabic and English screens. The next morning the QA lead and the development lead triage the 31 raw findings, and the QA lead sends this summary (VH very high, H high, M medium):

• Critical: the wallet is charged twice when Pay is tapped again after a timeout.
• Critical: a removed member can still open the family’s upcoming appointments from a saved link.
• Question: a cancellation exactly 24 hours ahead is refunded, but the rule says “more than 24 hours”. Which is intended?

Bug bash: family accounts    build 4.6.0 (staging)    90 min, 12 people, 6 pairs

Risk (level)                    Pairs  Raw  New defects  Highest severity
Wallet charge and refund (VH)     2     9        4       Critical
Last slot, two members (VH)       1     5        2       Major
Removed member privacy (VH)       1     4        1       Critical
24 h cancellation rule (H)        1     6        2       Major
Arabic and English screens (M)    1     7        3       Minor

Triage:      31 raw -> 12 new defects, 9 duplicates, 4 known issues,
             3 questions, 3 not a defect
Not reached: refund to an expired card (no sandbox card for it)
Risk now:    wallet and privacy stay VH until the Critical fixes pass confirmation
             testing; refunds to expired cards are still untested;
             slot, cancellation and screens lower once their fixes are confirmed
Follow-up:   6 regression cases (2 automated at API level),
             2 charters for the next sprint

Reporting the outcomes

The outcome of a bug hunt or a bug bash is information for a decision, not a score. A useful report answers four questions: what did we cover, what did we find, what did we not reach, and what is the risk now? Send it to the product owner and the development lead and add it to the test progress reporting for the release; the release decision stays with them.

Be careful with numbers. Twelve defects mean little on their own, and few defects in an area may mean that it is solid or that nobody looked hard. As the first testing principle says, testing shows the presence of defects, not their absence.

• Coverage — which risks and charters were explored, by how many people and for how long.
• Findings — new defects by severity with their IDs; duplicates, known issues and non-defects counted separately, which shows that triage happened.
• Gaps — what was not reached and why: a blocked environment, missing data, too little time.
• Questions — open questions with an owner, such as unclear rules found during the hunt.
• Risk now — for each risk, whether it is reduced, unchanged or still unknown, and why.
• Follow-up — fixes and confirmation testing, new regression cases and which to automate, new charters, and what to change at the next bash.

Frequently asked questions

References

• ISTQB Certified Tester Foundation Level (CTFL) v4.0ISTQB · 2023The foundation syllabus of software testing: test levels and types, test design techniques, test management and tools.
• Explore It! Reduce Risk and Increase Confidence with Exploratory Testing (Elisabeth Hendrickson)The Pragmatic Bookshelf · 2013The discipline of exploratory testing: charters, heuristics for varying inputs and states, and how it complements scripted tests.
• Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design (James A. Whittaker)Addison-Wesley (Pearson) · 2009The book that organised exploratory testing around "tours" — structured ways to explore features, data and user journeys.
• Session-Based Test Management (Jonathan Bach)Satisfice, Inc. · 2000The original description of session-based test management: chartered, time-boxed exploratory sessions, session sheets and debriefs.