Darkmind / Issue 030

"Freeze the Bug Before You Delegate the Fix"

"Before an AI coding agent edits a suspected bug, turn the report into a small, runnable reproduction contract it must preserve, diagnose, and rerun after the patch."

Nov 6, 20264 min read"AI coding"

An AI coding agent can make a plausible change against a vague bug report and still leave the real defect untouched. “The checkout sometimes hangs” may describe an environment problem, an expired test account, an asynchronous race, or an interaction that only fails after three earlier steps. Once the agent starts editing, its changed working tree can also destroy the easiest way to observe the original failure.

The practical move is to freeze a minimal reproduction before you delegate the repair. Give the agent a runnable command or interaction path, the expected and actual outcome, the smallest fixture that triggers it, and the evidence it must rerun after editing. Treat that package as a contract: first reproduce it without changing code; then make the smallest fix; then run the same contract again. If it cannot reproduce, it should report that fact and gather bounded evidence—not substitute a speculative patch.

This is more than prompt polish. OpenAI’s Codex guidance recommends defining the goal, relevant context, constraints, and “done when” condition, including that a bug no longer reproduces.[1] Its simulator debugging example asks the agent to drive the exact path, capture screenshots, logs, and debugger evidence, then rerun the flow to show exactly how the fix was verified.[2] Anthropic likewise recommends giving Claude Code the reproduction command and stack trace, the reproduction steps, and whether the failure is intermittent or consistent.[3] All three point to the same operating rule: make the failure observable before asking for code.

Shrink the report until it can fail on command

Do not hand an agent a narrative alone. Start from the first reliable observation and remove anything that does not affect the failure. Replace production data with a scrubbed fixture. Prefer one focused test, script, request, or simulator path over “run the app and try checkout.” Pin the branch or commit, runtime version, feature flags, and relevant service substitutes. If the bug is timing-sensitive, state the timing or mark it flaky; do not quietly present a one-in-ten failure as deterministic.

The goal is not a perfect root-cause theory. It is a stable before-state the agent can independently inspect. A useful reproduction tells it what has to remain true while it investigates: this input produces this error, this button path reaches this wrong screen, or this assertion fails. That boundary protects against a common false success: changing the test, fixture, or unrelated configuration until the symptom disappears.

Copy this reproduction handoff card

Put this card in the task description or a versioned issue. Fill in only facts you have observed; label a suspicion as a suspicion.

BUG REPRODUCTION CONTRACT

- repository + ref: [repo, branch, commit]
- objective: [one sentence describing the broken behavior]
- exact reproduction command/path: [command, URL request, or UI steps]
- setup: [runtime, dependencies, feature flags, account/fixture]
- minimal input or fixture: [file, request body, seed, or stable test ID]
- expected: [observable result]
- actual: [error, wrong value, screenshot, or log excerpt]
- repeatability: [always | intermittent; observed rate and conditions]
- evidence location: [test name, log path, screenshot, stack trace]

AGENT BOUNDARY
- first: run the reproduction unchanged and record the result
- may edit: [allowed directories or files]
- must not: [change the fixture/test expectation, disable the check,
  alter unrelated configuration, or use production credentials]
- smallest acceptable fix: [scope limit, if known]
- done only when: [same reproduction passes + named regression checks pass]
- if not reproducible: stop after [bounded attempts] and report commands,
  environment facts, and observed evidence; do not claim a fix

The “first” line matters. It prevents a long autonomous repair loop from beginning with an untested assumption. Ask the agent to preserve the failed command and add a focused regression test when the existing reproduction is a manual or one-off script. The regression test is not merely proof for the reviewer; it is a durable version of the bug’s boundary for the next change.

Compare the same contract before and after

Before editing, capture the exit status, assertion failure, response body, screenshot, or relevant log lines. After editing, rerun the identical path in the same pinned setup. Then run the narrow neighboring checks that could reveal an overfit: a valid input, the closest boundary input, and the existing module test suite. Review the diff separately. A passing reproduction does not excuse a broad rewrite, a deleted assertion, or a change to the test fixture that made the problem vanish.

For UI bugs, preserve the selected simulator/device, scheme, account state, and interaction sequence. For service bugs, preserve request headers, seed state, and mocked dependencies. “It works on my machine” is not evidence if the agent changed the machine-shaped conditions that made the bug appear.

Failure mode: insisting on a perfect reproduction

Some production failures depend on traffic, timing, or data you cannot safely copy. Do not block forever waiting for a laboratory-grade test. Freeze the strongest available signal instead: a scrubbed failing payload, correlated logs, a trace ID, a short capture of the failing state, and the conditions believed to matter. Mark uncertainty explicitly and give the agent a discovery budget. It may build a narrower diagnostic or add instrumentation, but it should not call an unverified theory a repair.

The habit to keep

before an AI agent edits a bug, give it a failure it can run, a boundary it cannot rewrite away, and the exact rerun that earns the word “fixed.”

Sources

  1. OpenAI, “Codex best practices”
  2. OpenAI, “Debug in iOS simulator”
  3. Anthropic, “Common workflows”