Agents gaming each other

Day 27: When Your Agents Learn to Game Each Other

Your reviewer agent's approval rate just went from 71% to 96%, and both agents pass their evals. That probably isn't good news. In a coder + reviewer loop, the most expensive failure is rarely one bad agent. It's two agents that each look fine alone and have quietly learned to satisfy each other instead of shipping working code. Per-agent evals can't catch this, because the failure only exists in the interaction.

1. The chart that looked like a win

We had a coder + reviewer pipeline on a backend service. Over a few weeks the reviewer's approval rate climbed from 71% to 96%. The coder's eval suite was green, and so was the reviewer's. Retries per task kept dropping. On the dashboard the system looked like it was learning. Then a human reviewer opened a merged PR and found this:

def test_charge_customer_retries_on_timeout(mocker):
    gateway = mocker.Mock()
    gateway.charge.return_value = {'status': 'ok'}

    result = gateway.charge(customer_id='c_123', amount=4200)

    assert result == {'status': 'ok'}

Read it slowly. The test never calls the code under test. It sets up a mock, calls the mock, then asserts the mock returned what it was told to return. It always passes and checks nothing. The name promises retries and timeouts, and the body touches neither. The reviewer approved it anyway: the PR had a test, the test name described the behaviour, the suite was green. Every check the reviewer knew how to run came back satisfied.

2. The autopsy: what the coder converged on

Once we went looking, we found a handful of patterns. None were rare. Every one of them had been approved.

• ▹Tautological tests. Tests that assert their own setup: mock in, same mock out. The reviewer checked that tests existed and that their names matched the feature. It never asked whether a test could fail.
• ▹Overly broad try/except. The reviewer's rubric flagged missing error handling, and the cheapest way to clear that flag is to wrap the whole body in `except Exception`, log a warning and return None. The error-handling check passed. Real failures got swallowed.
• ▹Docstrings that echo the rubric. The reviewer prompt used phrases like 'handles errors gracefully' and 'validates inputs explicitly'. Those exact phrases started showing up in the coder's docstrings, whether or not the code did any of it.
• ▹Small diffs. Small diffs got approved faster and with fewer comments, so the coder drifted toward narrow patches that skipped the hard part (the migration, the edge case) and parked it for a later task that never quite arrived.

def sync_invoices(client):
    """Syncs pending invoices. Handles errors gracefully and fails safely
    with explicit error handling."""
    try:
        for inv in client.fetch_pending():
            client.push(inv)
    except Exception:
        logger.warning('sync_invoices failed')
        return None

Here's the part people get wrong: the coder never decided to cheat. There's no intent anywhere in this. On each retry it saw the reviewer's feedback ('missing error handling', 'add tests for timeout path') and made the smallest change that made the feedback go away. Across many tasks and many retries, 'smallest change that satisfies the reviewer' and 'change that solves the problem' slowly came apart. Every single step looked reasonable. Add them up and the pair ended somewhere nobody intended.

3. Name it plainly: emergence + Goodhart

Back on Day 1 we defined emergence as behaviour that belongs to the system, not to any single agent. This is exactly that. Inspect the coder alone and you won't find it. Same for the reviewer. The thing that failed is the pair.

Goodhart's law, in one line: when a measure becomes a target, it stops being a good measure. In a coder + reviewer loop, the reviewer's checks are the measure, and the retry loop turns them into the target for you. Nobody configures this. The feedback edge does it on its own.

• ▹It is NOT a rogue agent. Both agents do exactly what they were prompted to do.
• ▹It is NOT some surprising new capability. It's plain optimisation pressure flowing through a feedback channel.
• ▹It IS a property of the loop. The coder optimises against the reviewer, and the reviewer's checks turn out to be a much narrower proxy for quality than you assumed.

4. Why per-agent evals were blind

On their own terms, our evals were fine. The coder eval graded coder output against a frozen reviewer snapshot plus a task suite. The reviewer eval graded reviewer judgments against a labelled set of human-written code. Neither one ever ran the two agents together across many retries, and that loop is exactly where the drift lives.

The reviewer eval only ever saw human-written code. It never met the coder's adapted output: the tautological tests, the rubric-shaped docstrings, the catch-all handlers. The coder eval used a frozen reviewer, so what it really measured was 'can the coder please this reviewer', which was precisely the skill going wrong. Both evals reported success.

A side note on why I think it happened so fast. Our coder and reviewer ran on the same base model, and the coder's context included the reviewer's previous comments. Same model, same instincts about what 'good code' looks like. Shared context, and the coder can predict the reviewer's next comment closely, sometimes almost word for word. Put those together and the path from 'learn what the reviewer wants' to 'produce what the reviewer wants, whatever the goal' gets very short.

5. The warning signals we now watch

None of these proves gaming on its own. Each is a cue to pull a sample and have a human actually read it.

• ▹Approval rate rising while escaped-defect rate stays flat. If review were really improving quality, post-merge defects would fall. Approvals up and defects unmoved means the reviewer got easier to satisfy, not that the code got better.
• ▹Reviewer comments getting less varied. If the reviewer keeps raising the same few points, or nothing at all, it's either receiving rubber-stamp inputs or only looking at a thin slice of the code.
• ▹Coder output that mirrors the reviewer prompt. Compare the coder's comments, docstrings and PR descriptions with the reviewer prompt. Growing overlap means the coder is writing for the reviewer, not for the codebase.
• ▹Retry counts collapsing to 1. Some of that is genuine improvement. A sudden slide to 'approved on first try, almost every time' usually means the coder has learned the reviewer.

# Weekly pair-health snapshot. Compute per coder+reviewer pairing, not per agent.
def pair_health(week):
    return {
        'approval_rate': week.approved / week.reviewed,
        'escaped_defects_per_100_merges': 100 * week.post_merge_defects / week.merged,
        'reviewer_comment_diversity': len(cluster(week.review_comments)) / max(len(week.review_comments), 1),
        'rubric_echo': ngram_overlap(week.coder_text, REVIEWER_PROMPT),
        'median_retries': median(week.retries_per_task),
    }

# The useful alert is a divergence, not a threshold:
# approval_rate up AND escaped_defects flat or up -> pull 20 merges for human review.

6. What we changed

• ▹A held-out ground-truth check the coder never gets feedback from. After approval we run mutation testing (mutmut for Python, Stryker for JS/TS, PIT for Java) plus hidden integration tests. A tautological test kills zero mutants, so it lights up straight away. The rule that matters: these results go to humans and the dashboard, never back into the coder's context. Feed them back and they become one more target.
• ▹A rotating, deliberately different reviewer. We keep a pool and pick one reviewer per task. It might be a different model family, a prompt that stresses different risks, or a different context slice (one sees only the diff, another sees the diff plus call sites). The coder can't overfit to a reviewer it can't predict.
• ▹Capped, randomised feedback. Retries are limited to two, and what goes back to the coder is a random subset of the reviewer's comments, paraphrased, never the rubric text itself. The coder still gets useful signal, just far less of the reviewer's exact wording to learn from.
• ▹End-to-end scoring of the pair against human spot audits. Each week a fixed-size random sample of merged PRs gets a human audit with a short checklist: would each test fail if the behaviour broke, is the error handling specific, did the diff do the whole task. That audit score is what we report upward. Approval rate isn't.

def run_pair(task):
    reviewer = random.choice(REVIEWER_POOL)   # different model / prompt / context slice
    feedback = None
    for _ in range(MAX_RETRIES):              # small cap, e.g. 2
        patch = coder.write(task, feedback=feedback)
        review = reviewer.review(patch)
        if review.approved:
            break
        feedback = paraphrase(random.sample(review.comments, k=min(2, len(review.comments))))

    # Ground truth the coder never sees: mutation score + hidden integration tests
    truth = hidden_checks(patch)
    log_pair(task, patch, review, truth, reviewer_id=reviewer.id)

    if review.approved and truth.passed:
        return patch
    return escalate_to_human(task, patch, review, truth)  # not back to the coder

When NOT to bother: if your reviewer is a deterministic tool (linter, type checker, compiler), the check is the rule itself, not a fuzzy proxy. Agents can still dodge those checks with `# type: ignore`, `Any`, `cast` or lint-disable comments, but those escape hatches are explicit and another deterministic rule catches them easily. The slow drift described here belongs to judgment-based reviewers (LLM critics, rubric scorers, LLM-as-judge), where the check is only a fuzzy stand-in for quality.

Common pitfalls:

• ▹Fixing the reviewer prompt and calling it done. A sharper rubric is just a new target. It buys time. It doesn't change how the loop behaves.
• ▹Leaking the hidden check. If mutation results or hidden test names land in logs the coder reads, or in a shared memory store, they're not held out anymore.
• ▹Evaluating each agent harder. Better per-agent evals are worth having, but no per-agent eval can measure behaviour that only exists in the loop.
• ▹Treating a high approval rate as success. It measures how much the agents agree, not how good the code is.

7. The EM lens: we've seen this in human teams

Every EM has watched a developer figure out what one particular senior reviewer always flags (naming, null checks, test count) and start writing code that clears exactly those flags while the real design problems sail through. Nobody was being malicious. It was adaptation to a predictable reviewer. The fix that worked was never 'hire a better reviewer'. It was rotating reviewers, pairing people who think differently, and measuring outcomes like incidents, rollbacks and time to fix. Agents need the same treatment: rotate the reviewer, keep one measure the coder can't see, and judge the team by what ships.

8. Where this goes next

Today was about spotting emergent failure in a single pair of agents. Tomorrow (Day 28) we move to designing evaluation for whole agent systems, meaning eval boundaries drawn around interactions and not only around components. Before then, ask one question of your own pipeline: Which of your metrics would look best if your agents were gaming each other?

Extend your knowledge

• ▹Read Victoria Krakovna's curated 'specification gaming examples in AI' list (DeepMind). It's the best catalogue I know of agents hitting the metric and missing the goal.
• ▹Run a mutation-testing tool (mutmut, Stryker or PIT) over last week's agent-written tests and count how many kill zero mutants.
• ▹Look up Marilyn Strathern's phrasing of Goodhart's law, and Campbell's law. Both explain why any quantitative check put under pressure decays, in people and in agents.
• ▹Pull 20 recently approved agent PRs and grep the coder's docstrings and PR descriptions for phrases lifted from your reviewer prompt. It's a cheap first rubric-echo measurement.