Why 'Acceptance Rate' Is the Most Dangerous Number on Your AI Dashboard

Day 6 — Vanity Metrics

Somewhere on your team's dashboard is a chart that only ever goes up, and everyone loves it in standup. Most of those charts are lying to you by omission. On an AI team the lie gets expensive fast — because it can quietly hide your agents cranking out garbage faster than any human ever could.

What a vanity metric actually is

A vanity metric is any number that moves without moving you. It reports activity, not outcome. The test isn't whether the number is real — it's whether a swing in it would ever make you do something differently. If your honest answer is 'no, we'd just screenshot it for the all-hands,' you've found one.

• ▹Total signups (vs. active users who came back)
• ▹Lines of code written (vs. defects shipped)
• ▹PRs merged per week (vs. cycle time or revert rate)
• ▹Downloads / installs (vs. retention at day 30)
• ▹Page views (vs. task completion rate)

Why they exist at all

They stick around because they're cheap. A real metric — one tied to an actual decision — usually means instrumenting a whole user journey, or sitting down and actually defining what 'success' means for a task. That's work. A count pulled straight out of a database query is not. So the moment a team feels pressure to show progress, it reaches for whatever number is easiest to make go up — and nobody stops to check whether it means anything.

The AI-era version is worse, not different

Every team building with LLMs and agents right now has grown a fresh batch of numbers that look like progress and aren't. They're seductive precisely because they're new — nobody's built the instinct yet for what 'good' actually looks like at this stage, which is exactly why the easy number gets reported instead of the harder question sitting underneath it.

• ▹Copilot / AI suggestion 'acceptance rate' — accepting a suggestion isn't the same as it surviving code review, or not causing a revert two days later
• ▹Number of autonomous agents deployed — an agent fleet that's running isn't the same as one that's finishing tasks correctly
• ▹Tokens processed per day — this is a cost driver, not a value signal; it just as easily means your prompts are bloated
• ▹'% of code written by AI' — a number PhoenixDX and half of LinkedIn love to post, with zero connection to defect rate or maintainability
• ▹Model benchmark scores (MMLU, HumanEval, etc.) reported as team KPIs when nobody on the team ships against that benchmark's task distribution

When to use them (narrowly) — and when not

• ▹OK: as a top-of-funnel awareness number for marketing or exec updates — 'we shipped X AI features' — as long as nobody internally optimizes toward it
• ▹OK: as a leading indicator tracked alongside an actionable metric, never alone — agent count is fine next to task success rate, not instead of it
• ▹NOT OK: as a KPI an engineer or agent is tuned to maximize — that's when Goodhart's law kicks in and the metric stops measuring anything real
• ▹NOT OK: as the headline in a retro or planning doc used to justify the next investment — that decision needs an outcome metric, not an activity count

Common pitfalls

• ▹Goodhart's law in practice: set 'AI acceptance rate' as a target, and engineers (or agents grading agents) start accepting suggestions they'd normally reject, just to move the number
• ▹Confusing throughput with quality: an agent fleet processing more tickets per hour tells you nothing if the resolution rate or human-override rate is also climbing
• ▹Reporting model-level metrics (latency, tokens, accuracy on a static eval) as if they were product metrics — they measure the engine, not whether the car got anywhere
• ▹Letting a vanity metric survive because it's the only one already instrumented — the fix is to instrument the outcome, not to keep reporting the easy number

Extend your knowledge

• ▹Read the original 'Lean Analytics' chapter on vanity vs. actionable metrics (Croll & Yoskovitz) — still the clearest framing, pre-dates AI but the test transfers directly
• ▹Look up Goodhart's Law and read a couple of real-world cases outside tech (education, healthcare KPIs) — it makes the AI-team version easier to spot
• ▹Audit your own team's dashboard this week: for every metric, write down the decision it's supposed to inform. If you can't name one, it's a candidate to drop or pair with an outcome metric
• ▹If your team reports AI-assist metrics, pair 'acceptance rate' or 'tokens/day' with a downstream one — revert rate, defect escape rate, or human review time — before the next report