bug-reproduction

---

name: bug-reproduction description: >- Turn a vague bug report into a VERIFIED minimal reproduction and then a failing regression test, agent-driven end to end. Covers extracting the implicit repro from a thin report (env, build, steps, data), the reproduce-minimize-isolate-capture loop, git bisect to find the introducing commit, building a deterministic minimal repro (fixed seeds, frozen time, stubbed network), writing the failing regression test BEFORE the fix (red) and confirming the fix flips it green, and writing repro evidence back into the ticket. Distinguishes flaky-not-reproducible from environment-specific. Use when: "reproduce this bug," "minimal reproduction," "repro steps," "find the commit that broke it," "git bisect," "make the repro deterministic," "write a failing test for this bug," "regression test for a defect," "can't reproduce this bug." Not for: Classifying/deduplicating/severity-routing existing failures without reproducing them — that is ai-bug-triage. Generating tests from specs rather than from a defect — that is ai-test-generation. Related: ai-bug-triage, ai-test-generation, test-reliability, systematic-debugging, qa-project-context. license: MIT metadata: author: kindlmann version: "1.0" category: ai-qa

Quick Route

You have…	Go to
A thin report and no idea how to trigger it	Step 1: Extract the implicit repro
A messy 14-step repro to clean up	Step 2: The reproduce-minimize-isolate-capture loop
"Worked last month, broken now"	Step 3: Bisect to the introducing commit
A repro that passes/fails inconsistently	Step 4: Make the repro deterministic
A clean repro, no fix yet	Step 5: Write the failing regression test (red) first
"The dev says it's fixed, test is green"	Step 6: Verify the fix actually fixes it
"It won't reproduce for me but does for the user"	Step 7: Flaky vs environment vs not-reproducible
Repro + test done	Step 8: Write the evidence back into the ticket

Discovery Questions

First, check .agents/qa-project-context.md in the project root — it carries the tech stack, test runner, known-flaky areas, and environment matrix. Pass over any question it already answers. If it is missing, suggest creating one with the qa-project-context skill.

• What is the report, verbatim? The thinner it is, the more you must extract before touching code — a one-line report sets the whole intake agenda (Step 1).
• Does it reproduce at all yet, and how reliably? "Every time" vs "sometimes" decides whether you go straight to minimizing or into determinism work first.
• Did it ever work? A known-good past release unlocks git bisect to the introducing commit; no known-good point means you debug forward instead.
• What is the test runner and stack? Vitest/Jest vs Playwright changes the determinism API (vi.setSystemTime vs page.clock) and where the regression test lands.
• What are the non-determinism sources? Time-of-day rules, randomness, third-party APIs, locale — each must be pinned for the repro to be trustworthy.

---

Core Principles

1. Reproduce before you theorize. The strongest wrong instinct is to read a symptom and jump to a root cause ("sounds like a float rounding bug, let me patch the total calc"). Don't. Extract the repro, make it fail on demand, and only then form a hypothesis. A fix without a reproduction is a guess you can't falsify.

2. A repro is a deterministic artifact, not a story. "It happens around midnight with a random code" is a story. Freeze the clock, seed the RNG, and stub the network so the same inputs produce the same failure on every run and every machine. If it isn't deterministic, you can't bisect it, test it, or prove it fixed.

3. Minimize one variable at a time, and re-confirm after every cut. Shrinking the repro is a search, not a rewrite. Remove one step, data field, or dependency, then re-run and confirm it still reproduces. Removing several at once tells you nothing about which one mattered.

4. The regression test is written RED, before the fix. Assert the real expected value, watch it fail first (proving it catches this bug), then watch the fix flip it green. A test added after the fix, or one disabled / marked pending, proves nothing.

5. Green isn't done — revert-to-verify is. A passing test can pass for the wrong reason. Temporarily remove the fix and confirm the test goes red again. Only a test that fails without the fix actually guards against the bug.

---

Step 1: Extract the implicit repro

A thin report ("Checkout is broken, order total is wrong sometimes") names a symptom, not a path. Before any code, extract or ask for every reproducibility dimension. Never invent the repro steps from imagination, and never theorize a root cause yet — both come after the bug reproduces.

For a "wrong total" / data-correctness bug, the load-bearing dimensions a thin report most often omits are:

• Exact steps to reproduce — the click-by-click path, not "checkout is broken."
• Build / version / commit (git SHA) — they may be on a build where it's already fixed.
• Environment — browser + version, OS, device.
• Input data — cart contents, quantities, the account/user, coupon, the exact fixture. A total is a pure function of its inputs; without them you are guessing.
• Expected vs actual — the number they expected and the number they saw.
• Frequency — every time, or intermittent? "Sometimes" points at non-determinism.
• Locale / timezone / currency — rounding, tax, and formatting are locale-specific; a total "wrong" in de-DE may be correct in en-US.
• Timestamp of occurrence, plus any logs/screenshots/network trace.

Write these into a single repro spec before touching code. If a row is blank, that is your next question to the reporter — not a license to start writing the fix or theorizing.

See references/intake.md for the full extraction checklist, why each load-bearing dimension matters for "wrong total," and the repro-spec template.

---

Step 2: The reproduce→minimize→isolate→capture loop

Given a confirmed-but-messy repro (e.g. 14 manual UI steps across 3 pages), do not hand the 14-step version to the developer and do not rewrite it wholesale. Run this loop:

1. REPRODUCE / confirm. First establish a baseline: run the full repro and confirm it actually fails. You can only minimize something that currently reproduces.
2. MINIMIZE. Remove one step, field, or dependency. Re-run. If it still reproduces, keep the cut; if it no longer fails, that element was load-bearing — restore it. Repeat, one variable at a time, until every remaining piece is necessary. This is the core ordering rule: never minimize before confirming it reproduces, and never omit verifying it still fails after each cut.
3. ISOLATE. Narrow the failure to the smallest layer that still shows it — drop from a 3-page UI flow to a single page, then to a unit/API call against the offending function if the bug lives below the UI.
4. CAPTURE. Record the now-minimal repro as evidence: the smallest steps or the single command, plus logs/trace/screenshot. This is what the developer and the regression test consume.

The output is the smallest sequence that still reproduces — not the original walkthrough.

---

Step 3: Bisect to the introducing commit

The bug is on HEAD but a past release was clean, and you have a command that exits non-zero when the bug is present. Use git bisect run to binary-search history automatically — do not manually check out each commit, and do not use git revert to hunt for it.

git bisect start
git bisect bad HEAD          # current commit has the bug   (alias: git bisect new)
git bisect good v2.4.0       # last clean release            (alias: git bisect old)
git bisect run npm test -- checkout-total.spec.ts   # ONE targeted test, never the full suite
# bisect prints "<sha> is the first bad commit"
git bisect reset             # ALWAYS clean up — restores the original HEAD

The exit-code contract git bisect run uses: exit code 0 = good (bug absent), non-zero (1–124) = bad (bug present), exit 125 = skip (untestable). So your command must return 0 when the feature is fine and non-zero when the bug reproduces — most runners already do this. Run one targeted test, not npm test:all / the whole suite: an unrelated failure at an old commit would mark it bad and send the search down the wrong half.

The good/bad pair assumes a regression (good in the past, bad now). The old/new aliases mean the same search and read better when hunting any state transition.

See references/bisect.md for the full happy path and the skip/untestable wrapper.

Bisect skip and determinism (untestable or flaky commits)

Two things corrupt a naive bisect, and the default bad answer — "exit 1 on any failure" — walks into both:

• Old commits won't build. A compile error exits 1, which bisect reads as "bug present" and marks a clean commit bad. Wrong: an unbuildable commit is untestable — your wrapper script must exit 125 (skip) on build failure, distinguishing it from a real bad commit.
• Flaky network/timing failures. A transient un-stubbed third-party call exits 1 and gets blamed. Force determinism during bisect — stub the network, pin TZ, seed the RNG — so only the real bug can fail the step. If a commit's result flips between runs, treat it as untestable (exit 125), not bad.

Wrap the step in a script that returns 0 = good, 1 = bad, 125 = skip (the valid bad range is 1–127 excluding 125), guards the build, stubs the network, and retries once to catch flakiness. Then git bisect run ./bisect-step.sh. Full wrapper in references/bisect.md.

---

Step 4: Make the repro deterministic

The bug "only around midnight, with a random discount code, via a third-party pricing API" has three non-determinism sources. Pin all three so it fails the same way every single run — do not wait for midnight, do not let it hit the real pricing API, and never use a sleep/setTimeout/waitForTimeout to paper over timing.

Source	Vitest	Playwright
Time	vi.useFakeTimers() + vi.setSystemTime(new Date('…'))	page.clock.install({ time }) + page.clock.setFixedTime(…)
Randomness	faker.seed(1337) (or stub Math.random)	seed the app's RNG via an init hook
Network	MSW setupServer + http.get → HttpResponse.json	page.route(...) → route.fulfill(...)

Key points:

• Vitest: vi.setSystemTime only works after vi.useFakeTimers(). Seed faker in beforeEach. Set MSW onUnhandledRequest: 'error' so a missed stub fails loudly.
• Playwright: page.clock.install/setFixedTime must run before page.goto. page.clock is the supported API — it exists; do not fall back to a hand-rolled Date override, and do not bump the timeout to "make it pass."
• Pin locale/timezone/currency (TZ=UTC, LANG) when the bug is locale-sensitive.

Avoid: jest.useFakeTimers('legacy') (and timers: 'legacy') — legacy fake timers are deprecated and don't mock Date/Date.now, so the clock stays live and your "frozen" repro still drifts. Modern timers are the default since Jest 27 — just call jest.useFakeTimers() + jest.setSystemTime(), or vi.useFakeTimers() in Vitest. (Jest 30, 2025)

See references/determinism.md for the full Vitest (vi.useFakeTimers + vi.setSystemTime

• faker.seed + MSW setupServer) and Playwright (page.clock + page.route) recipes, plus a 10-run determinism check.

---

Step 5: Write the failing regression test (red) first

You have a clean deterministic repro and the dev hasn't fixed it yet. Write the regression test now, before the fix, as TDD-for-bugs:

1. Encode the minimal repro as a test that asserts the real expected value — expect(total).toBe(2754). A tautological assertion that always passes proves nothing.
2. Run it and confirm it fails before the fix — it must be red first. A test that doesn't go red isn't exercising the bug.
3. Commit the test (or stage it on the fix branch) so the test guards the fix in CI.
4. After the dev's fix lands, re-run: it should flip to green. Same test, no edits.

Expected state: red before the fix, green after the fix.

Wrong moves that defeat the point — a test that genuinely fails until this bug is fixed: writing the test only after the fix already landed; disabling the failing test (pending markers, an always-true tautology, or removing the assertion) just to keep CI green; or fixing first and bolting a test on afterward. Keep the assertion live and let it go red.

See references/determinism.md for the deterministic test bodies the assertion sits in.

---

Step 6: Verify the fix actually fixes it

The dev pushed a fix and the regression test now passes. Green is necessary but not sufficient — a test can pass for an unrelated reason. Do not close on green alone or trust the dev's word. Run the validity check:

1. Revert the fix temporarily (stash it, or git stash/comment the fix line) and re-run the test. Confirm it still fails without the fix. That proves the test actually exercises this bug (the test catches the bug) — that it passes because of the fix, and is not passing for another reason unrelated to the defect.
2. Restore the fix and confirm green returns.
3. Re-run deterministically several times (--repeat-each / a loop) to confirm the green is stable, not a lucky pass.

Only when the test is red-without-fix and green-with-fix, repeatably, is the fix verified. This revert-to-verify step is the whole point of the regression test and is the one most often left out.

---

Step 7: Flaky vs environment vs not-reproducible

You spent two hours and it won't reproduce for you, but it clearly happens for the user. Do not close as "cannot reproduce" immediately, do not assume flaky and quarantine, and do not conclude "doesn't repro on my machine so it isn't real." These are three distinct diagnoses with distinct evidence:

Diagnosis	Discriminating evidence	What you do
Flaky	Same code, same env, passes and fails on the same commit — run --repeat-each 50 (or rerun the same command many times) in one environment and watch it flip	Find the non-determinism (time/RNG/network/race), make it deterministic (Step 4)
Environment-specific	Reproduces only under a different config — timezone, locale, viewport, OS, browser version, CI vs local — and is stable within that config	Match the user's environment: reproduce their timezone/locale/OS/browser, then minimize
Data-dependent	Reproduces only with the user's specific account/input	Get and replicate their data/fixture; the bug rides on the input, not the platform
Genuinely not reproducible	None of the above reproduces after matching env + data + repeat runs	Document what you tried (envs, run counts, data) and the negative result — don't silently close

The step bare attempts miss: match the reported user environment and data before judging — replicate their config (timezone, locale, OS, browser version) and reproduce their env, then re-run. --repeat-each in the same env isolates true flakiness; cross-env divergence points to environment-specific; input-dependence points to data-specific.

---

Step 8: Write the evidence back into the ticket

Reproduction and the committed regression test are done. Replace the vague original report with a structured block — do not paste the raw 14-step UI walkthrough, and do not just write "reproduced, closing." Include all seven elements:

1. Minimal steps / repro command — the smallest path, not the walkthrough.
2. Environment + build/commit — exact SHA and platform.
3. Expected vs actual — the concrete numbers.
4. Introducing commit — the offending commit from git bisect.
5. Regression test — link to the committed test file and path.
6. Evidence — logs, screenshot, trace, or artifact.
7. Determinism notes — seed, frozen time, and stubs so anyone can re-run identically.

See references/ticket-writeback.md for the copy-paste Markdown structure.

---

Anti-Patterns

1. Jumping to the fix before reproducing

Reading "total is wrong" and patching the total calc, or guessing "looks like a rounding bug," before you can make it fail on demand. You can't prove an unreproduced fix works. Extract the repro first (Step 1).

2. Minimizing before confirming it reproduces

Stripping steps from a repro you never confirmed actually fails. You end up "minimizing" something that was never broken. Confirm the baseline fails, then cut.

3. Removing several variables at once

Cutting three steps in one pass, so when it stops reproducing you don't know which one mattered. Remove one variable at a time and re-run after each.

4. Bisecting the whole suite, or by hand

git bisect run npm test:all lets unrelated failures mark commits bad; manual checkout-and-test is slow and error-prone. Run one targeted test under git bisect run.

5. Exit 1 on build failure during bisect

Treating an unbuildable commit as "bug present." It marks clean commits bad and corrupts the search. Return exit 125 (skip) for untestable commits; reserve non-zero for the genuine bug.

6. Live time, RNG, or network in the repro

Leaving new Date() and Math.random unmocked, hitting the real pricing API, or "waiting for midnight" with a sleep. The repro becomes a coin flip. Freeze time, seed the RNG, stub the network (Step 4).

7. Writing the test after the fix, or disabling it

Adding the regression test once the bug is already gone, or neutering a failing test (pending markers, a tautological assertion, a removed assertion) to keep CI green. The test never proves it catches the bug. Write it red, before the fix.

8. Closing on green without revert-to-verify

"The test passes, close it." A test can pass for the wrong reason. Revert the fix, confirm it goes red again, restore, confirm green.

9. Closing as "cannot reproduce" on first failure to repro

Collapsing flaky / environment-specific / not-reproducible into one dismissal. Match the user's environment and data and use --repeat-each before judging (Step 7).

---

Failure Modes

Symptom	Likely cause	Fix / check
Bisect lands on an obviously unrelated commit	Full suite or flaky failures marking commits bad	Switch to one targeted test; wrap with exit-125 skip + network stub
Repro passes locally, fails in CI (or vice versa)	Environment-specific (TZ, locale, OS, browser)	Pin TZ/LANG; match the failing environment (Step 7)
Test is green but you're not sure it catches the bug	Never ran it red	Revert the fix and confirm it fails (Step 6)
vi.setSystemTime has no effect	Called before vi.useFakeTimers()	Call useFakeTimers() first
page.clock time not applied to app startup	install/setFixedTime ran after page.goto	Move clock setup before navigation
Repro flips pass/fail run to run	Live time/RNG/network not pinned	Apply Step 4; confirm with --repeat-each 10

---

Verification

• The repro command/test fails on demand: run it 10× (--repeat-each 10 or a loop) and confirm it fails every time before the fix.
• git bisect run … terminates with "<sha> is the first bad commit" and git bisect reset leaves you on the original HEAD.
• With the fix reverted the regression test exits non-zero; with the fix applied it exits 0.
• The ticket block contains all seven write-back elements (Step 8) — grep it for the commit SHA, the test path, and the determinism notes.

---

Done When

• A documented minimal reproduction exists — smallest steps or a single command — that fails on demand, verified failing across repeated runs.
• If the bug is a regression, git bisect has named the introducing commit SHA and it is recorded in the ticket.
• The repro is deterministic: time frozen, RNG seeded, network stubbed — proven by 10 identical consecutive runs.
• A regression test is committed that was red before the fix and green after, and was confirmed to fail when the fix is reverted.
• The ticket carries the structured evidence block with all seven elements (minimal steps, environment/build, expected vs actual, introducing commit, regression-test link, evidence artifact, determinism notes); the original vague report is replaced, not left.
• If it did not reproduce, it is classified (flaky / environment-specific / data-dependent / not-reproducible) with the evidence that led there — never silently closed.

---

Related Skills

• ai-bug-triage — Classify, deduplicate, and severity-route existing failures. Triage decides whether and where a failure matters; come here to actually reproduce one and write the failing test. Triage hands off; bug-reproduction picks up.
• ai-test-generation — Generate tests from specs/PRDs/stories. Use it when the source is a requirement; use this skill when the source is a defect and the test must first go red against the bug.
• test-reliability — Runtime self-healing and quarantine for a flaky test. When Step 7 diagnoses true flakiness, go there to stabilize or quarantine; here you only diagnose.
• qa-project-context — Stack, test runner, environment matrix, and known-flaky areas that shape every step above. Check it first.
• systematic-debugging (superpowers:systematic-debugging) — The general root-cause debugging loop once you have a deterministic repro; this skill produces that repro and the failing test that guards the eventual fix.