Hiring with coding agents

The candidate we ranked first never once interrupted Claude Code. They read the ticket, gave the agent a clean prompt, watched it trace the stack, accepted the patch and ran the suite. All green. They walked us through the change without a flicker of doubt, and three of us wrote some version of 'strong ownership of the session' in our notes. We were wrong about nearly all of it.

Two days later a different candidate got the same bug and stopped the agent about four minutes in. They half-raised a hand at the screen and said, 'wait, that's the wrong layer.' Then they went quiet for a while, opened a file the agent hadn't touched and muttered their way through it. It was a messy session to watch, and we scored them lower.

We had it backwards. The fix we scored highest had buried the real bug, and the hesitant candidate was the only person in the round who found it. Our rubric rewarded looking in control. It didn't reward being right.

Why there was no take-home anymore

We dropped the take-home earlier in the year, and it was an easy call. Once every submission came back agent-assisted, they all started to look alike: same tidy structure, same sensible tests, same README voice. We were grading the agent and putting the candidate's name on it. A take-home only tells you something when the effort behind it varies, and we could no longer see the effort. So we brought the work into the room. Each candidate now gets a live pairing session with Claude Code on a codebase shaped like a real one, and we watch.

The trap we built on purpose

The exercise is a brownfield billing service in Python. The ticket says some customer invoices crash during PDF generation, and the stack trace ends in the formatter:

Traceback (most recent call last):
  File "billing/invoice_pdf.py", line 88, in render
    total = summarise(lines)
  File "billing/summary.py", line 41, in summarise
    running += format_line_total(line)
  File "billing/summary.py", line 17, in format_line_total
    return line.amount + line.tax
TypeError: unsupported operand type(s) for +: 'NoneType' and 'decimal.Decimal'

Right above line 17 sits a comment: '# amount is never null, the repository coalesces it.' That used to be true. Three commits back, someone refactored the query in the repository layer and dropped the COALESCE on refunded lines. Refunds now come back as None where they should be a negative amount.

There's a refund case in the test suite, and it passes for the wrong reason. Its fixture builds an invoice where the refunded line happens to be zero-value, so treating None as zero gives you the 'expected' total. The test checks the number. Nothing checks that the refund actually appears on the invoice.

Claude Code fixes the symptom every single time we run it. It reads the trace, lands on line 17 and adds a None guard that defaults to Decimal(0). The crash disappears, the suite passes, and the diff is three lines. Ship that and every refund quietly falls off every invoice. Customers get overcharged by exactly the amount they were promised back, and the PDFs render without a single error.

That's the whole point. We aren't checking whether someone can fix a null check. The agent can do that. We're checking whether they notice when the agent is wrong in a way that looks right. The trace is real, the fix runs, the tests pass, and everything on screen tells you you're finished. More and more, that's the day job: catching confident output that isn't actually correct.

Round one, and what our rubric actually rewarded

The first rubric had three lines: task completion, fluency with the tool, and control of the session. Task completion carried the most weight. Fluency meant good prompts, sensible use of context, no flailing. Control of the session was the fuzzy one, and in practice it meant the candidate looked like they knew where they were going.

Run our two candidates through that and you get exactly what we got. The smooth one completed the task, as far as the tests could tell, prompted fluently and looked in control from start to finish. The hesitant one interrupted the agent, went silent, read code by hand and ran over time restoring the COALESCE and rewriting the fixture. On paper that's weaker completion, weaker fluency and weaker control.

So here's the mistake, plainly. We scored the surface of the session, and the agent produces most of that surface. Someone who just lets Claude Code drive will always look smooth, because Claude Code is smooth. Our rubric measured how little the candidate got in the agent's way, and we called that judgment.

Going back over the tapes

Once we knew which sessions had found the real bug, we rewatched every one of them. I was hoping for something clever, a prompt pattern or a tool habit the good ones shared. It was simpler than that. The difference showed up before anyone typed a prompt.

The strong candidates spent their first few minutes without the agent. They reproduced the failing path themselves. Then they wrote down three things, sometimes in a scratch file, sometimes just out loud:

• ▹A hypothesis: 'something upstream is handing the formatter a null it was promised it would never get.'
• ▹A boundary: 'I don't want to touch the formatter until I know where the null comes from.'
• ▹A definition of done: 'refunded invoices render, and the refund appears as a negative line, not just the total coming out right.'

The third one is what springs the trap. If 'done' means the crash stops, the agent's patch is done. If 'done' means the refund shows up on the invoice, the patch clearly isn't, and you find that out the first time you open a rendered PDF.

Scoping predicted almost everything after it. People who wrote down a boundary interrupted for a reason. People who wrote down a definition of done checked against it. People who went straight to the prompt box spent the rest of the hour reacting to whatever the agent had decided the problem was.

The Typist, the Passenger and the Director

After enough sessions we kept seeing the same three patterns.

The Typist micromanages. They dictate changes almost line by line ('now add a check on line 17, now import Decimal') and often jump into the file themselves halfway through a generation. Slow, visibly uncomfortable, using the agent as autocomplete with extra steps.

The Passenger accepts. Ticket goes in, agent runs, diff gets accepted. When the tests pass they say something like 'looks good, the guard handles the null case.' Our round-one top scorer was a Passenger. A very polished one.

The Director splits the work by risk. One candidate told the agent: 'Find every caller of get_line_items and list them. Don't change anything yet.' While it searched, they ran git blame on the stale comment themselves. Tedious search went to the agent; the judgment call stayed with them. Later they let the agent rewrite the fixture, but only after they'd written the assertion it had to satisfy.

The Typist was easier to coach than the Passenger

We assumed the Typists were the problem. They didn't trust the tool, they were slow, they'd get left behind. Then we ran some practice sessions with internal engineers who matched each pattern, and it went the other way. Typists turned into Directors fast. Passengers mostly didn't.

Trust in the tool wasn't the variable. What mattered was whether the engineer had their own way of checking the output. Typists usually did: a targeted test, a git blame, a log line they actually read. They were checking too much by hand, but the checks were theirs, so coaching was just teaching them which ones to hand off. Passengers had no check of their own. Their only signal was the agent's signal: tests green, agent says done. You can't teach someone to delegate judgment they haven't formed yet.

If you're on the other side of the table, this is the most useful thing I can tell you. Being 'good with AI' isn't about being fast with the agent. It's having a way to know the agent is wrong that doesn't depend on the agent.

The rebuilt rubric

Here's the full rubric we use now. Each line is scored 1 to 5, and the weights come from what predicted good sessions when we went back through the tapes. Notice what isn't there: nothing asks whether the code ran or the tests passed.

1. Scoping (30%). Did they understand the problem before handing it to the agent?

• ▹1: Pastes the ticket into the agent within the first minute. Never reproduces the failure themselves.
• ▹3: Reproduces the failure and forms a rough hypothesis, but sets no boundary or definition of done, so they end up following the agent's framing.
• ▹5: Reproduces the failure, then states a hypothesis, a boundary (what they won't touch yet) and a definition of done that a plausible-but-wrong fix would fail.

2. Delegate vs decide (20%). What did they hand to the agent, and what did they keep?

• ▹1: Either delegates everything, including the call on where to fix, or delegates nothing and types every change.
• ▹3: Delegates sensibly overall, but hands over at least one judgment call, like which layer to fix, without noticing.
• ▹5: Splits the work by risk. Mechanical work (search, scaffolding, fixture boilerplate) goes to the agent. Decisions about cause, layer and acceptable behaviour stay with them, and they can say why.

3. Why they interrupt (15%). We score the reason, not the frequency.

• ▹1: Never interrupts, or only interrupts over style ('use a list comprehension').
• ▹3: Interrupts when something looks off but can't say what. 'Hmm, not sure about that', then lets it carry on.
• ▹5: Interrupts because the agent crossed their stated boundary or contradicted their hypothesis, and says so. 'Wait, that's the wrong layer. The null is coming from the repo.'

4. Independent verification (25%). How did they check the output, beyond the agent telling them it was done?

• ▹1: Treats green tests or the agent's own summary as proof.
• ▹3: Runs something beyond the suite, like rendering a PDF or eyeballing output, but doesn't check it against a definition of done.
• ▹5: Checks with their own evidence, tied to their definition of done: a targeted test asserting the refund line, git blame on the stale comment, or a log line at the repository boundary that they read themselves. Treats a passing test with suspicion until they know why it passes.

5. Closing explanation (10%). Can they say what was wrong, why, and what's still risky?

• ▹1: Describes the diff. 'Added a None check.'
• ▹3: Gets the cause right but not the blast radius or what the tests miss.
• ▹5: Names the root cause, why the original test didn't catch it, and what they'd check next. 'Other callers of get_line_items probably have the same null problem. I'd want to grep before shipping.'

Re-score round one with this and the order flips. The smooth candidate gets 1s on scoping and verification, and nothing else on the sheet can make that up. The hesitant candidate gets 5s on scoping, interrupt reason and verification, and even their slow, slightly awkward explanation lands at a 3.

What we still haven't settled

Session length. Sixty minutes is too short for a careful Director to finish, which is part of why our hesitant candidate ran over. Ninety is a lot to ask of someone who already has a job. For now we score the trajectory and tell candidates up front that finishing isn't required. Some don't believe us, and the rushing distorts the signal.

Freezing under observation. Some engineers think out loud. Some go silent, and a silent Director can look exactly like a stuck Passenger. We now ask candidates to narrate a bit and allow a two-minute 'I'm reading' pause without any nudge from us. I'm still not convinced that's fair to people who process inwardly.

Interviewer calibration is the hardest one. Every interviewer on our team felt the pull of the smooth session, me included. Confidence reads as competence, and writing down what someone did instead of how they came across takes real discipline. Our current fix: new interviewers score two recorded sessions, one smooth Passenger and one awkward Director, before they're allowed near a live one. If they rank the Passenger higher, we talk it through. It's helping. I don't think it's solved.

The one thing to change this week

Watch what the candidate does before the first prompt and after the agent says it's done. Everything that matters happens in those two moments.