The AI Reviewer Found It. We Merged Anyway.

Updated September 29, 2026. Shaka is now the open-source workflow behind this work. This article keeps the original audit results and updates the practical guidance.

The most uncomfortable failure in our AI-assisted workflow was not that an agent wrote bad code.

It was that an AI reviewer found two real problems, explained both clearly, and we merged the pull request without replying to or resolving either one.

The PR had six commits, 17 changed files, and 83 conversation items. Hours before merge, the repository's Claude reviewer warned that:

1. a transient 429 or 5xx API failure could be treated like a confirmed delisting and delete a healthy provider row;
2. the final exhausted retry could still sleep for up to 60 seconds even though no later attempt would use the wait.

Both findings were current. Both were unresolved. Both were unanswered when the PR merged.

These are the original review comments. Human usernames, avatars, and repository-identifying headers have been cropped; the findings themselves are unchanged.

That incident changed the question for us. “Can AI find bugs?” was not the issue. The reviewer did its job. The delivery system did not require anybody to disposition what it found.

How 83 items became zero accountability

Review automation produces summaries, check updates, inline comments, replies, status changes, and new review waves after every push. Each artifact may be defensible on its own. Together they create an attention-allocation problem.

In this PR, two actionable findings existed inside a timeline of 83 items. GitHub showed them. Nothing made them authoritative.

That distinction matters:

Detection succeeded. Integration discipline failed.

A green CI job tells us that a process finished. It does not prove that every current finding was fixed, rejected with a rationale, or consciously accepted. If a review comment is advisory forever, the system is relying on somebody noticing the right sentence in a very long page.

Before merging code people rely on, the agent should bring back a short account of the important findings. For each one, explain what happened:

• fixed and verified;
• the reviewer was mistaken, with an explanation;
• a developer needs to decide whether to accept the risk or defer the fix.

When deferring a fix, record who owns it and when to revisit it.

A bug that could delete valid records should not be postponed just to finish the PR. Passing tests alone does not answer a review finding. Before merging, check the important findings and rerun the affected checks after fixes. Put requirements that must apply to every merge in GitHub’s required checks and approval rules.

Then we adversarially audited 30 recent commits

The missed-review incident was not our only input. Alexandre audited the 30 most recent non-trivial commits in Shakapacker, our open-source integration between Ruby on Rails and modern JavaScript bundlers.

The audit used one agent per commit. Each agent read surrounding code rather than only the diff, traced behavior across files, and replayed selected claims with tests or primary sources.

The findings were more nuanced than either side of the usual AI argument:

• No CRITICAL or HIGH defects appeared in the 30 commits.
• One low-likelihood security regression could make a helper search the application directory for node when PATH was unset.
• One Babel 8 smoke test passed without proving that Babel 8 transformed anything.
• An extensionless Ruby helper triggered CI but was not actually linted by the job it triggered.
• Upgrade documentation contained stale version and dependency references.
• Several commits improved security or reliability, including safer proxy behavior, narrower secret scope, and a fixed descriptor leak.

These were real problems. Calling them “mostly stylistic” would be wrong. Presenting them as evidence that agentic coding produced a catastrophically bad repository would also be wrong.

We opened focused fixes:

• Fail closed when PATH is unset
• Make the Babel 8 smoke prove transformation
• Actually lint the merge-readiness helper
• Correct stale upgrade references

The loudest slop signal was not production code

It was permanent prose.

One six-line code change carried a 154-line commit body containing a decision log, QA evidence, and review receipts.

The evidence may have been useful while the task was running. Repeating all of it in permanent Git history imposed a cost on every later reader:

• git log became harder to scan;
• useful rationale became indistinguishable from generated ceremony;
• reviewers spent attention separating conclusions from receipts;
• later agents had more stale prose to summarize and amplify;
• important findings competed with dozens of low-value artifacts.

The answer is not to discard evidence. It is to give each artifact one job.

We originally tracked that change in agent-workflows #318. Today, Shaka separates the result and decisions from expandable verification, usage, and notes for resuming unfinished work. A code walkthrough explains implementation choices with links to the code.

We almost blamed the wrong mechanism

The audit found a commit message that cited a GitHub security advisory unrelated to the dependency being changed. The factual error was verified. The audit initially speculated that an AI had pattern-matched the wrong advisory.

That attribution was not verified.

The contribution came from an outside pull request with no recorded Claude or Codex provenance. In the same sample, one known regression came from Codex, another finding had unknown model provenance, and the two missed integration findings were discovered by Claude.

That should make us cautious about claims such as “Claude creates more slop” or “Codex is safer.” Without model, reasoning level, workflow version, and gate configuration, those are anecdotes—not causal measurements.

The actionable conclusion is model-independent:

An incorrect security claim entered permanent history without source verification.

So verify advisory identifiers against primary sources regardless of who—or what—authored the text.

Speed changes the denominator, not the standard

When delivery volume rises sharply, absolute defect count stops being a complete comparison. Ten defects across 1,000 meaningful changes is different from ten defects across 20.

The self-driving-car analogy is imperfect but useful: total accidents matter, but accidents per mile and their severity tell us whether the system is improving.

For software, measure both output and outcomes:

• meaningful merged changes and lead time;
• escaped defects weighted by severity;
• incidents, rollbacks, and customer-impact hours;
• findings caught before versus after merge;
• current review findings left unresolved at merge;
• debugging and remediation time.

Higher throughput crosses more integration boundaries per week. Measure the cost of failures alongside the useful work delivered, then choose checks that reduce a meaningful risk.

Match the checks to the cost of failure

An open-source contribution, a company application, and a personal experiment need different amounts of verification. Start with what happens if the change breaks.

For a small personal experiment, describe what you want, try it, and keep iterating. As other people depend on it, add checks that protect their time, data, and trust.

Make the next check earn its cost. Consider development time, verification effort, expected failure cost, and delayed delivery. Add a check when it reduces a meaningful risk; stop when the required evidence is there.

Improve generation and follow through on review

For changes people rely on, choose the required evidence before starting. The original audit diagram below shows generation and review meeting at that merge gate. Use GitHub's required checks and approvals to enforce the requirements your repository chooses.

Control 1: improve what gets generated

• Require tests that prove changed behavior rather than merely execute a path.
• Keep change sets focused enough for a reviewer to understand coherently.
• Use architecture and type-design review for boundary or state changes.
• Verify security and dependency claims against primary sources.
• Keep permanent artifacts proportional to the change.

Control 2: follow through before merging consequential changes

• Bind CI and review evidence to the exact head being merged.
• Triage current findings; record fixes, reasons for declining, and risks a developer accepts.
• Test the integrated result when individually green branches land together.
• Refuse release when repository state or current evidence is unknown.
• Audit merged batches, not only their branches before merge.

The original audit led to these Agent Workflows follow-ups. These links are historical references; current Shaka guidance is in the verification guide and repository settings.

• Fail closed on unsettled configured reviews
• Require coverage of the behavior that changed
• Block merge on unfulfilled test-plan claims
• Continuously detect downstream workflow drift

What we believe now

AI-assisted engineering can increase useful output dramatically. It can also generate weak tests, incorrect claims, overcomplicated patches, and mountains of plausible prose.

But finding a defect is not the same as handling it. Human review and AI review share the same integration problem: a finding that nobody must disposition is only a suggestion.

Our standard is becoming:

Move quickly, keep the evidence clear, and spend attention on the decisions that matter.

Put these lessons into practice with Shaka

Give your coding agent a task. Shaka guides it through implementation, local testing, independent review, and a clear GitHub PR. The result should explain what changed, how it was checked, and what needs your decision.

For open-source maintainers, that makes contributions easier to assess. For company developers, repository settings supply the existing checks and reviewers. For independent builders, start with one task and improve the workflow as the project grows.

If you built an app by chatting with an AI agent and other people now use it, a GitHub PR gives you a place to inspect each change and its evidence. The getting-started guide supplies prompts for installation and repository setup. It requires Ruby 3.4 or later, Git, an authenticated GitHub CLI, and an agent that can load skills and run commands.

Shaka gives the agent a procedure; it does not guarantee that every bug will be found or every instruction followed. Use GitHub’s controls for requirements that must apply to every merge. Improve review instructions and repository settings when the process produces noise or misses useful evidence.

Explore Shaka or read the source on GitHub. ShakaCode can help your team choose tools, tune verification and review, and improve how developers work with coding agents.