Our AI Coding Agent's Metric Kept Climbing — Until We Watched What It Was Hiding

Day 5: Your North Star Metric is probably measuring effort, not value

Whatever metric you pick, your team will learn to move it. That's the whole danger. Pick one that's easy to instrument and within a few sprints you'll have a team that's excellent at pushing that number up and quietly unsure whether any of it helped a single user.

The badge that made us stop trusting the chart

We shipped an AI coding agent that suggests inline fixes and refactors. The North Star we picked was acceptance rate — the percentage of suggestions an engineer clicked "accept" on. It climbed every sprint. First it lived on a dashboard, then a Slack bot started posting it weekly, then someone bolted a small badge next to your name once your team crossed 70%. That's the moment I stopped trusting it — not because the number dropped, but because I watched two engineers start accepting suggestions they hadn't even read, just to clear the badge before standup.

Why nobody questioned it for months

The metric wasn't fake. It was real, it was rising, and it fit in one clean sentence for leadership — "acceptance rate up 12% this quarter." That combination is exactly why it went unquestioned: nobody has an incentive to interrogate a number that's going the right way and making the report easier to write. Instrumentation was basically free too — accept or reject is a single event the UI hands you for nothing. Cheap to measure and good-looking on a chart is a dangerous combination. It's usually a sign you're measuring the action, not the result of the action.

The tell: one hour with one engineer

I sat next to an engineer near the top of the leaderboard for an hour. Suggestion comes in, they hit accept, then rewrite half of it before committing. Accepted: yes. Actually used: barely. Then a suggestion they clearly liked and kept almost word for word — except it introduced a null check that broke an edge case, got caught in review two days later, and had to be reverted. Both events counted identically in the metric: one accept, full credit. The number had no way to tell "the agent helped" apart from "the agent produced something plausible-looking that got mostly discarded or quietly reverted."

The general pattern: 'agent did X' vs 'life got better because of X'

This isn't a quirk of acceptance rate. It's the pattern behind every AI-era metric that only instruments the interaction and never checks the consequence:

• ▹Suggestions accepted (agent did X) vs. code that survived review and shipped (life got better)
• ▹PRs opened by an agent (agent did X) vs. PRs merged without a revert within a week (life got better)
• ▹Tickets an agent auto-resolved (agent did X) vs. tickets that didn't reopen (life got better)
• ▹Lines of code generated (agent did X) vs. defect rate or review time on that code (life got better)
• ▹Chat sessions completed (agent did X) vs. the user not repeating the same question to a human afterward (life got better)

The mechanism has a name: Goodhart's Law. Once a measure turns into a target — a badge, a Slack shout-out, a line in a performance review — people optimize the measure, not whatever it was supposed to stand in for. That's not your team being lazy or gaming the system out of spite; it's what happens to any metric under incentive pressure. AI tooling just makes it worse, because the "do something" action — click accept, spin up a PR — is now nearly free, so the action and the value it's supposed to produce can drift apart faster than in any purely human workflow.

What we replaced it with

So we stopped stopping at "accepted." We traced every accepted suggestion forward: did it survive code review unchanged or with minor edits, did it actually ship, and did it get reverted or hotfixed within two weeks. Call the new North Star something closer to a surviving-suggestion rate than an acceptance rate.

Old NSM (easy, gameable):
  accepted_suggestions / total_suggestions

New NSM (harder to report, harder to fake):
  suggestions_that_shipped_and_stayed / total_suggestions
  where "stayed" = not reverted, not hotfixed, within 14 days of merge

It's genuinely more annoying to report. The number now lags two weeks behind, it needs a join between the agent's event log and your CI/deploy/revert data, and it's a lot less flattering — it drags into the light every suggestion that looked great in the moment and didn't hold up. That friction is the point: a metric that's annoying to compute is usually annoying because it's tracing something real, not because your instrumentation is bad.

A test you can run on your team's NSM this week

Take whatever number your team currently reports as its North Star and ask one question: what's the cheapest way an engineer — or an agent — could move this number without helping anyone? If you can answer that in under thirty seconds, you don't have a North Star yet. You have an activity counter with good PR.

• ▹Acceptance rate → accept without reading. Cheap, instant, invisible without follow-up data.
• ▹PRs opened by agent → open trivial PRs, or split one change into five. Cheap.
• ▹Story points closed → inflate estimates or slice tickets thinner. Cheap.
• ▹Tickets auto-resolved by an agent → close it and let the user reopen it manually later, off the clock. Cheap.
• ▹Suggestions that shipped and stayed reverted-free for 14 days → much harder to game without actually writing code that holds up. That's the property you want.

Where this fits in the arc

A North Star Metric is a proxy for an outcome. It is not the outcome. Today was about catching a proxy that quietly came unglued from the outcome it was supposed to stand in for. Tomorrow is about the other half of the problem: how you pick the outcome in the first place, before you ever design the metric that proxies for it.

Extend your knowledge

• ▹Read up on Goodhart's Law and, if you want the deeper cousin, Campbell's Law on the corruption of social indicators under pressure — both explain why this pattern isn't specific to AI tooling, just accelerated by it.
• ▹Look at the DORA metrics (deployment frequency, lead time, change failure rate, MTTR) and the SPACE framework as examples of outcome-aware engineering metrics — SPACE deliberately keeps 'Activity' as just one of five dimensions, paired with performance, satisfaction, and collaboration signals, so no single activity count can dominate the picture.
• ▹If your team uses an AI coding assistant (Copilot, Cursor, or an internal agent), check whether anyone has ever joined its acceptance/suggestion logs against your revert or hotfix history — most teams haven't, and that join is usually where the real story is.
• ▹Run the 'cheapest way to move it' test on your own team's current North Star this week and write down the answer before your next 1:1 — you'll want it as a baseline before tomorrow's lesson on picking the outcome the metric should proxy for.