AI agents and hand-backs

Day 29: Agents, and the Most Important Decision They Make

One of the most useful things an AI agent ever did for my team was stop before finishing its task. Every vendor right now is selling agents that “complete tasks end to end.” By the end of this lesson you'll know what an agent actually is, and you'll have a better question to ask about one than “how many tasks did it finish?”

The message that made me rethink agents

Here's what the message looked like. I've rebuilt it from memory and anonymised the details, but the structure matches what an agent on our team posted on a migration ticket at PhoenixDX:

I stopped before running the migration.

What I did:
- Updated the schema for the `orders` table and regenerated the models.
- Ran the unit tests: 214 passed.
- Checked the 6 services that read `orders`. 5 use the new field correctly.

What I could NOT verify:
- The reporting service builds its SQL from strings at runtime.
  I can't tell statically whether it reads the old column.
- There is no staging copy of production data, so I can't test
  the backfill against real row counts.

Why I stopped:
- The migration drops a column. That can't be undone without a restore.

What I need from you:
- Confirm whether reporting reads `orders.status_legacy`, OR
- Give me read access to a staging snapshot and I'll check it myself.

The engineer who picked it up wasn't annoyed. What they said was roughly “oh, good catch.” Reporting did read the old column. Confirming that and adjusting the migration took them about twenty minutes. If the agent had kept going and run it, reporting would have broken without any error, and we'd have found out when someone in finance asked why the numbers looked wrong.

A completion dashboard would count that ticket as a failure, because the task wasn't finished. It was still the most useful thing any agent did for us that month. To see why, we need a clear picture of what an agent is.

What makes something an agent: the loop

An agent is a loop, not a single call to a model. It looks at the situation, picks a step, takes that step with a tool, checks the result, and then goes around again. You already know two of the parts from earlier days. The LLM does the deciding. Tool use, which we covered in an earlier lesson, does the acting. The new part today is the loop that joins them, plus one extra decision the agent makes every time it goes around.

• ▹Observe: collect what's true right now, such as test output, file contents or an API response.
• ▹Decide: the model chooses the next step, based on what it saw and what the goal is.
• ▹Act: call a tool. That could mean running a command, editing a file, querying a database or sending a request.
• ▹Check: read what the tool returned. Did the step do what it was supposed to?
• ▹Choose: keep going, finish, or stop and hand back to a person. People tend to forget this fifth step, and it's the one that matters most.

A single model call can't do any of this. It gives you one answer and never finds out whether the answer worked. An agent sees what its own actions led to. That's what makes agents powerful, and it's also what makes them dangerous.

Why the loop multiplies mistakes

If a chatbot gives you a wrong answer, you have one wrong answer. You read it, you're suspicious of it, you move on. If an agent gets step 2 wrong, it treats that result as true and builds steps 3 to 10 on it. Each of those later steps can look fine by itself, because the mistake is underneath them where you can't see it.

Here's a quick illustration. It isn't a measurement, and it assumes every step is independent, which real steps never are. Suppose an agent gets each step right 90% of the time. The chance it gets all 10 steps right is:

Per-step accuracy    Chance all 10 steps are right
90%                  0.90^10 ≈ 35%
95%                  0.95^10 ≈ 60%
99%                  0.99^10 ≈ 90%

Being right 90% of the time on each step sounds good. The whole run is still right only about a third of the time. Don't hold on to the exact numbers. The point is that confidence compounds badly. If an agent is confident at every step, including the wrong ones, you get a long, tidy, believable chain of work with a crack near the bottom. The longer the loop runs, the more valuable it is for the agent to stop at the right moment.

The two kinds of failure: a 2x2

You can sort every agent outcome on two axes. Was the work right or wrong? And did the agent finish confidently, or hand it back? That gives you four boxes, and they cost very different amounts.

• ▹Right and confident: this is what you want. The agent did the work and the work is correct. Cost: a quick review.
• ▹Wrong and handed back: cheap. The agent got stuck or wasn't sure, and it said so. Cost on our team: usually between a few minutes and an hour of an engineer's time, while the context was still fresh.
• ▹Right but hesitant: mildly annoying. The agent stopped when it didn't need to. Cost: a few minutes to reply “yes, go ahead.” It isn't free, though. If it happens all the time, people stop reading the hand-backs.
• ▹Wrong and confident: expensive, and you can't see it. The agent said “done” and it wasn't. Cost: whatever it takes to find the error later, plus everything that got built on top of it in the meantime.

In that last box, the key question is who finds the error, and when. At PhoenixDX, the agent almost never caught its own confident wrong completions, and neither did the engineer reviewing the PR. They turned up later. It might be an integration test failing two days on, a customer report, or a teammate asking why a function behaved strangely. By that point the original context was gone, other changes had landed on top, and fixing it meant digging back through the agent's reasoning to find the step where it went wrong.

The PhoenixDX comparison

This wasn't a controlled study. These numbers are rough estimates from our tickets and my notes, not tracked metrics. Over one stretch, we compared the escalation from the start of this lesson with a handful of confident completions from the same period that later turned out to be wrong.

• ▹The escalation: about 20 minutes of one engineer's time. It was caught before anything shipped, and nothing had to be redone.
• ▹A typical confident wrong completion: found days later, by someone other than the reviewer. Each one cost roughly half a day to a day and a half once you add up finding it, tracing it back, fixing it and re-checking everything that depended on it.
• ▹The worst case: an agent “fixed” a flaky test by loosening the assertion, which hid a real bug for over a week. That one incident cost more than every escalation from the period combined.

Now look at how the scoreboard counted all this. By completion rate, the confident agents won, because they closed their tickets. The honest agent lost. Demos and benchmarks count completions, while your team pays for the errors nobody caught. Those are two separate ledgers, and only the second one shows up in your engineers' week.

Anatomy of a good hand-back

“I'm not sure” is only useful if reasons come with it. A good hand-back has four parts:

• ▹What I tried: the steps already taken, so the person doesn't repeat them.
• ▹What I verified: claims that have evidence behind them, like tests that were run, files that were checked and outputs that were observed.
• ▹What I couldn't verify, and why: the exact gap. Not “something might be wrong,” but “the reporting service builds SQL at runtime, so I can't check it statically.”
• ▹What I need from you: one small, concrete request, such as a decision, a permission or a piece of information.

Here's a bad hand-back for comparison:

I was unable to complete this task due to complexity.
Please review the requirements and try again.

It's honest, and it's no help at all. It doesn't say what was done, what's safe or where the problem is, so the person has to start from scratch. A bad hand-back doesn't make an error cheaper. It just pushes all the work back onto the human. A good hand-back gives the person the hard 10% and keeps the 90% that's already done.

How teams get agents to do this

There's no “calibration” switch you can turn on. You get this behaviour from ordinary design choices, and anyone on the team can understand them and push for them:

• ▹Explicit stop conditions. Write them down. Stop if the tests fail twice in a row. Stop if a change touches more than N files. Stop if the thing the task assumes exists can't be found. Without rules like these, the agent's default is to keep going.
• ▹Checks before irreversible actions. Deleting data, dropping columns, sending emails, deploying and paying money should all sit behind a verification step or a human approval gate. Claude Code handles this with permission modes, and many agent frameworks support approval steps.
• ▹Permission to say no, written into the instructions. Models tend to try to please you. If the system prompt says “finishing is the goal,” they'll finish. If it says “stopping with a clear reason is a valid, good outcome, and here's the format,” you'll get better hand-backs.
• ▹Verification as part of the job. Ask the agent to name the evidence behind its claim that the task is done. “Tests pass” should mean it ran them, not that it expects they'd pass.
• ▹Reward escalations in review. If engineers grumble every time an agent stops, the team will tweak prompts and settings until it stops less often, and you'll drift back into the expensive box. Count a good hand-back as a win in retros.

The metric to take away

Judge an agent by the cost of the errors it lets through, not by the number of tasks it completes. Task success rate is easy to measure and looks good on a slide. The cost of undetected errors is harder to measure, but it's what tells you whether the agent saves your team time or quietly costs it time.

Ask these three questions of any agent vendor or internal tool:

• ▹When does it stop? Ask for the actual stop conditions. “It's very reliable” is not an answer.
• ▹What does it tell me when it stops? Ask to see a real hand-back. Does it cover what was tried, what was verified, what wasn't, and what it needs?
• ▹How would I know if it was confidently wrong? What evidence does it attach to “done”? What review or checks catch the errors it doesn't flag itself?

If the answer to the third question is “it's rarely wrong,” keep pushing. Every agent is wrong some of the time. What you need to know is whether you'll find out in twenty minutes or in two weeks.

Bridge to Day 30

Today was about one agent knowing its limits. Tomorrow is the last day, and we'll look at the bigger picture: what changes when an agent has teammates, whether those are other agents or your human team? If an agent hands work to another agent instead of a person, and the work passes through several hands, who owns the result?

Extend your knowledge

• ▹Read Anthropic's engineering post “Building effective agents.” It separates workflows from agents and makes the case for starting simple, which fits well with today's loop.
• ▹Try an agent tool like Claude Code on a small, low-stakes task in a repo. Notice where it asks for permission and where it doesn't, and write down which actions you'd want gated.
• ▹Pick one agent or AI feature your team uses and fill in the 2x2 from memory for the past month. Which box cost the most engineer time?
• ▹Write three stop conditions and a hand-back template for an agent workflow you care about. Then ask the three vendor questions about it.