Activation metrics

Day 18: Activation — Measure the Moment Value Lands, Not the Moment Someone Clicks

Your adoption dashboard can be green while your users are quietly giving up. Pick the wrong activation event and that's exactly what happens. AI coding tools make this mistake easy, because the events that are cheapest to log are often the ones that reward people for going through the motions.

The dashboard that looked great

When we rolled out coding agents to our own engineering team, the internal dashboard looked healthy. Daily usage was climbing. Acceptance rate (the share of agent suggestions people kept) was high. We had a short list of 'top adopters', and I was about to ask them to coach everyone else.

Then I sat next to a few of them while they worked. The top adopters weren't moving faster. They were salvaging. They'd take a mediocre diff and spend hours patching it by hand, because the agent had already written it. Meanwhile, the engineer with the lowest acceptance rate on the team was shipping the most. Their sessions went like this: generate, read for thirty seconds, throw it away, re-scope the prompt, generate again, keep the third attempt. By our metric they were the least activated person on the team. In reality they were the only one using the tool the way it's meant to be used.

The concept: activation

Activation is the step between signup and retention where a user first gets the core value of the product. Product people call it the 'aha moment'. It's not the first time someone uses the product. It's the first time using it pays off. On Days 4 and 5 we defined who your user is and what value means to them. Activation is how you check whether a given user has actually received that value yet.

• ▹Signup or access: they can use the product.
• ▹First use: they tried it once. That's activity, not value.
• ▹Activation: they got the thing the product exists to deliver.
• ▹Retention: they keep coming back because of it. That's tomorrow's lesson.

The engineer's default mistake: the event that's easiest to log

Engineers pick activation events the way we pick log lines. Whatever's already emitted and easy to query wins. For AI coding tools, the usual suspects are:

• ▹First agent run. It tells you someone was curious, and nothing about whether they got value.
• ▹First merged agent PR. Someone can rewrite 90% of it by hand and still merge, so the event fires even when the agent was a net cost.
• ▹Acceptance rate (% of suggestions kept). This one rewards rubber-stamping. Accepting takes one keystroke, and the cleanup afterwards never shows up in the number.
• ▹Daily active usage goes up when people are stuck, too. A frustrated user retrying a broken workflow looks exactly like a productive one.

Acceptance rate has the same flaw as 'logged in' for a consumer app. Logging in rewards opening the app, not getting anything out of it. Acceptance rewards keeping the output, not whether it was worth keeping. Push either number up and you get more of the behaviour you were trying to avoid.

The reframe: what is the core value, really?

To find a better activation event, ask what the product actually gives people. For a coding agent, it isn't 'code appears'. Code was never the scarce thing. What the agent really gives you is cheap judgment. You can try an idea for almost nothing, see it fully written, and drop it the moment it's wrong. Before agents, trying an approach cost hours, so we argued about designs in the abstract. Now a concrete attempt costs minutes.

If that's the value, you'll see it in what people throw away as well as in what they keep. Someone who bins a bad diff in seconds, without regret, has understood that generation is cheap and their judgment is the expensive part. That understanding is the aha moment.

Two groups the dashboard counted as the same

• ▹Salvagers treat agent output as sunk cost. The diff exists, so they patch it. Sessions drag on, frustration builds, and eventually they conclude AI 'isn't there yet for our codebase'. They rarely say it out loud. They just open the tool less often. That's silent churn.
• ▹Discarders treat agent output as a disposable draft. Within seconds of spotting a wrong direction they regenerate, narrow the scope, split the task, or just write it themselves. Their acceptance rate is lower, they ship more, and months later they're still using the tool.

Our usage dashboard counted both groups as 'activated'. Both ran the agent daily. Both merged agent PRs. You could only tell them apart by watching what they did with bad output.

The new signal, made concrete

We dropped 'first merged PR' and replaced it with two measures:

• ▹Time-to-first-confident-discard: how long after someone starts using the agent before they first throw its output away quickly and move on, rather than trying to repair it. Shorter usually means they hit the aha moment sooner.
• ▹Discard-to-salvage ratio: fast discards divided by long salvage sessions, over a rolling window. If it's rising, that person is learning to treat the tool as a cheap way to try things. If it's stuck near zero, treat it as an early churn warning.

You don't need new instrumentation to start. Both can be approximated from data you already have:

• ▹Abandoned agent branches: deleted or left untouched soon after creation, with no human commits on top.
• ▹Regenerate, reject or rewind events in the tool's own logs. Many agentic IDEs and CLIs record these, so check what yours exposes.
• ▹Agent-authored PRs closed unmerged within minutes (a discard), compared with ones that take days and pile up human fix-up commits (a salvage).
• ▹Lots of human commits stacked on an agent's first commit before merge. That's a decent salvage proxy.

-- Approximate discard vs salvage behaviour per engineer.
-- Thresholds are illustrative: calibrate them against your own team.
WITH first_use AS (
  SELECT author, MIN(created_at) AS first_agent_run
  FROM agent_sessions
  GROUP BY author
),
agent_prs AS (
  SELECT author, created_at, closed_at, merged_at,
         human_commits_after_agent AS fixups
  FROM pull_requests
  WHERE origin = 'agent'
),
classified AS (
  SELECT author, closed_at,
    CASE
      WHEN merged_at IS NULL
       AND closed_at - created_at < INTERVAL '30 minutes'
       AND fixups = 0                            THEN 'fast_discard'
      WHEN fixups >= 5
        OR closed_at - created_at > INTERVAL '2 days' THEN 'long_salvage'
      ELSE 'normal'
    END AS kind
  FROM agent_prs
)
SELECT c.author,
  MIN(c.closed_at) FILTER (WHERE kind = 'fast_discard')
    - f.first_agent_run                          AS time_to_first_confident_discard,
  COUNT(*) FILTER (WHERE kind = 'fast_discard')  AS fast_discards,
  COUNT(*) FILTER (WHERE kind = 'long_salvage')  AS long_salvages
FROM classified c
JOIN first_use f USING (author)
GROUP BY c.author, f.first_agent_run;

One warning. This is a judgment signal, not a number to maximise. Put 'discards per week' on a leaderboard and people will generate junk just to throw it away, and you've rebuilt the acceptance-rate problem with a new name. Use the ratio to spot who would benefit from pairing or coaching, and to check whether onboarding teaches the right habit. Don't use it to rank people. Some long sessions are legitimate, too. A hard migration is real work, not salvaging. Read the pattern over weeks, not single events.

The general lesson: a three-question test for any activation metric

Discard-versus-salvage only applies to coding agents. The thinking behind it works for any feature you own. Before you trust an activation event, put it through three questions:

• ▹1. Does it show value received, or just activity? 'Ran a report' is activity. 'Shared a report and someone acted on it' is much closer to value.
• ▹2. Could someone hit it while getting no value? If a disengaged user, or someone whose manager told them to 'try the AI tool', can trigger it just by going through the motions, it's a vanity event.
• ▹3. Does it predict who's still around in 30 days? Split users by whether they hit the event and compare 30-day retention. If both groups retain about the same, the event isn't activation, no matter how good it looks on a chart.

Engineers tend to skip question 3, and it matters most. An activation event is a hypothesis about what drives retention. Test it like any other hypothesis, against the outcome.

Exercise for today

• ▹Pick one feature you own: an internal AI tool, a CLI, an API, or something user-facing.
• ▹Write down its current 'activation' event, whether anyone wrote it down or not. If nobody ever defined one, write down what the dashboard quietly treats as success.
• ▹Run it through the three questions. Be honest on question 2 and describe, in one sentence, how a bored user could trigger it.
• ▹Draft one alternative event that a disengaged user couldn't fake. Hint: look for behaviour that takes judgment, like discarding, customising, sharing, or reversing a default. Clicking doesn't count.
• ▹If you have the data, check question 3 for both events. Compare 30-day retention for users who hit each event against users who didn't.

Close

Activation isn't about whether someone adopted your product. It's about whether they trust it enough to use it properly. With coding agents, that means trusting it enough to throw its work away. Once someone is activated, the question becomes what keeps them coming back, and that's tomorrow: retention.

Extend your knowledge

• ▹Andrew Chen's 'The Cold Start Problem' covers how network effects get started and compound. Once you've read it, ask what 'the hard side' looks like when you're rolling out an internal AI tool.
• ▹The SPACE framework paper (Forsgren et al., 2021, ACM Queue) on developer productivity argues against single-number activity metrics. Acceptance rate falls into exactly that trap.
• ▹METR's 2025 study of experienced open-source developers using AI tools found that perceived speed-up and measured speed didn't match. Keep it in mind whenever self-reported adoption looks better than the real value delivered.
• ▹Lenny Rachitsky's newsletter has several pieces on how real companies define activation. Hold their aha-moment definitions up against the three-question test and see which ones survive.