Your AI Code Reviewer Said LGTM — Because It Was Talking to Itself

Day 15: Your Verifier Agent Might Just Be Agreeing With Itself

Share a model family between your generator and your reviewer, hand the reviewer the generator's own explanation for why the code is fine, and you haven't built a review step — you've built an echo chamber with an LGTM button attached. The scary part is you won't see the gap until the bug that gets through is the one that pages you.

The PR that got an LGTM it shouldn't have

Here's what I watched happen on one pipeline. Generator agent ships a caching fix — wrap a slow lookup in an in-memory dict, keyed by user_id. Clean diff, passes unit tests, reads like something a competent engineer would write on a Tuesday. The verifier — same model family, and crucially fed the generator's own PR description alongside the diff — comes back with: 'LGTM. The caching strategy is sound and the key choice correctly scopes per-user data. No concerns.' Merged. Within a day memory climbed until the service OOM'd, because nobody — human or agent — had put an eviction policy anywhere near that cache.

+ _cache = {}
+ def get_profile(user_id):
+     if user_id in _cache:
+         return _cache[user_id]
+     profile = db.fetch_profile(user_id)
+     _cache[user_id] = profile
+     return profile

Stare at that diff and nothing screams 'unbounded memory leak.' The bug isn't subtle in the code. It's subtle in what the verifier was actually looking at — which, it turns out, wasn't the code.

Why this wasn't a fluke

The verifier's prompt carried the generator's own framing: 'Added a per-user cache to reduce redundant DB lookups, keyed by user_id for correctness.' That sentence answers the exact question a reviewer is supposed to ask, before the reviewer gets to ask it. Once the verifier has the generator's rationale sitting in its context, its job quietly shifts — from 'does this code do what it should' to 'does this explanation hang together.' Coherence, not correctness. And a well-written rationale is almost always coherent.

The mechanism: same-model self-consistency bias

This has a name in the LLM-as-judge literature, and it gets worse — not better — when generator and verifier share a model or a family. A model finds its own reasoning style the most convincing one going; it's trained to continue plausible chains of thought, and 'plausible' means a lot when the chain was written by a system with your exact priors. Hand the verifier the generator's chain-of-thought and you've handed it a script that pre-empts every objection it might raise. Loop that — generate, verify, revise, verify again — and each round narrows toward whatever both sides already find agreeable. That's not convergence on correctness. That's two systems talking themselves into the same answer.

Fix 1: strip the verifier's context down to diff + spec

No PR description. No chain-of-thought. No shared conversation history with the generator. The verifier gets two things and nothing else: the diff (or the resulting code) and the spec it needs to satisfy — acceptance criteria, API contract, security policy, whatever lives outside the generator's own head. If the verifier can't work out from code plus spec why a design decision is sound, that's worth flagging as a documentation gap on its own. It's not a reason to hand it the generator's homework.

Before:
  verifier_prompt = f"""
  Here's the PR: {pr_description}
  Here's the reasoning: {generator_cot}
  Here's the diff: {diff}
  Does this look correct?
  """

After:
  verifier_prompt = f"""
  Spec/contract: {spec}
  Diff: {diff}
  Task: find a case where this diff violates the spec.
  """

Fix 2: force an artifact, not an opinion

'Looks good' is the cheapest thing a model can say — the lowest-energy output in the whole distribution. A reproducing test case, a working exploit, a concrete counterexample: those cost something, because the model has to actually simulate the code against an adversarial input instead of pattern-matching 'this looks like the kind of diff that's usually fine.' So change the task. Not 'review this' — 'break this.' Write a failing test, a specific input with expected-vs-actual output, or for security work, a working exploit. No artifact, no pass.

• ▹Correctness bugs → a failing unit test that reproduces the defect against the new code
• ▹Security issues → a concrete exploit request or payload, not a vague 'consider input validation' note
• ▹Concurrency/state bugs → a counterexample sequence of calls that leaves the system in a wrong or unsafe state
• ▹API/contract breaks → a caller snippet that compiles against the old contract and breaks against the new one

Before/after on the same bug class

I re-ran the caching PR through the redesigned verifier — diff and spec only ('cache must be per-tenant isolated and bounded'), task set to 'produce a failing test.' First it wrote a two-tenant test: populate the cache under one user_id, assert the second tenant's request never touches it. Passed — the key scoping was fine. Then it wrote a second test: call get_profile with 100k distinct user_ids, assert the cache stays under a size bound. That one failed, and it caught exactly the missing eviction policy the original verifier waved straight through. It still missed something three lines down — a race condition on concurrent cache writes — because the falsification task was scoped to the spec it got, and nobody had written 'must be thread-safe' into that spec. Structural separation kills the echo-chamber failure. It doesn't fix an incomplete spec. Different problem, different fix.

Where this fits in the 30-day arc

Verification is its own agent-design problem — model choice, context boundary, task shape, all of it needs the same deliberate attention you'd give the generator. Bolt an LLM reviewer onto a pipeline and call it a safety layer, and you get the feeling of safety without the substance. Tomorrow picks up right where this leaves off: once the verifier is structurally separate, the next failure mode is incentives — what happens when it's simply cheaper for the verifier to approve than to keep bouncing PRs back to the generator.

Extend your knowledge

• ▹Audit your current review-agent prompt: is the generator's PR description, commit message, or chain-of-thought sitting anywhere in the verifier's context window? If yes, that's the echo-chamber bug already living in your pipeline.
• ▹Take one existing 'review this diff' verifier prompt and convert it into a falsification task — 'write a test that fails against this diff' — then compare approval rates on the same batch of PRs before and after.
• ▹Go read the LLM-as-judge bias research (self-preference / self-consistency bias in model-graded evaluation). It's the same dynamic inflating eval leaderboards, and it's the dynamic driving your verifier's rubber-stamping.
• ▹If you're running a generator-verifier loop with retries, log how the verifier's language shifts across rounds. Language narrowing toward agreement over iterations is the tell you're watching consensus form, not correctness improve.