Hiring for verification

Day 25: Hiring for verification, not typing speed

Last sprint your agent probably wrote more of the shipped code than you did. In your next interview, someone will still ask you to invert a binary tree from memory while they watch. Only one of those is the job. On many teams agents now write a large share of the code, so the skill that sets engineers apart is no longer producing code. It's checking it. Whether you're about to sit on a panel or about to face one, you need to know what that skill looks like and how to test for it.

1. The near-miss

Our loop at PhoenixDX used to open with live coding: one algorithm problem, a shared editor, 45 minutes. One candidate froze. She got a partial solution, lost track of an off-by-one and ran out of time. Before the next round started, two interviewers had already written 'no hire' in their notes.

That next round was a pilot we'd only just started. We handed her a pull request our coding agent had generated against a realistic ticket and asked her to review it as if she were approving it for production. She found all three bugs we had planted. When we dry-ran the same exercise internally, two of our own senior engineers missed those same three. She also told us which one she'd block the merge on, and which test she'd write to prove it.

We hired her, but only just. If the debrief had simply counted rounds, she'd have been rejected. That left us with an uncomfortable question: what had our loop been rewarding all this time?

2. Why that happened

The old loop rewarded fast recall under pressure: how quickly you could produce a correct algorithm from memory with someone watching. That was a fair proxy back when engineers spent most of the day writing code by hand. It isn't one any more.

Day 24 was about delegation, meaning handing well-scoped work to agents and to people. Verification is the other half of that. Once you delegate, most of your day goes on reading output, questioning it and sometimes throwing it out. At PhoenixDX that's now most of what engineering time looks like. Typing speed barely matters. What matters is catching the change that looks right and is wrong.

• ▹The old bottleneck was producing correct code, so the old test was writing code live.
• ▹Now the bottleneck is judging code you didn't write. The test should be reviewing code you didn't write.
• ▹Agents don't get tired or embarrassed. They're confidently wrong in ways that look plausible, and reviewers need to be calibrated for exactly that.

3. The format: a 40-minute agent-PR review

The exercise centres on a pull request that looks like a real agent-generated change to a small but realistic service. We get an agent to write it, then go in by hand and plant specific flaws.

• ▹Setup: a small repo of a few hundred lines that the candidate can clone or open in a browser IDE. The diff is roughly 150–250 lines across 4–6 files. That's enough room to hide things and still small enough to read in 25 minutes.
• ▹Ticket context: the candidate gets the original ticket text, the prompt the agent was given, and a two-line note on the service's constraints (e.g. 'handles payment webhooks; providers retry on timeout'). Without the ticket they can't judge scope, so reading it is part of the test.
• ▹Allowed tools: whatever they'd use at work. Running tests, running the code, official docs, an AI assistant. One rule applies: they say out loud what they asked the assistant and whether they believe the answer. If they accept an AI reviewer's verdict on trust, that's a signal too.
• ▹The interviewer says: 'This PR came from our agent. Review it as if your approval ships it to production. Think out loud. I'll answer questions the way a teammate would.'
• ▹The interviewer doesn't say how many issues there are, or whether there are any at all, and gives no hints when the candidate gets close. Nothing like 'are you sure?' or 'look at line 40'. Factual questions about the codebase get straight answers, the same as they would on the job.

4. The planted bugs

Every PR carries a mix of flaws from five categories. Here's a trimmed example. The ticket asks for a webhook handler to be made idempotent, so that provider retries don't double-charge anyone.

# handlers/payments.py  (agent-generated, with planted issues)

async def handle_payment_webhook(event, store, client):
    key = event["idempotency_key"]

    # [1] race: check-then-act with no atomic guard
    if await store.exists(key):
        return {"status": "duplicate"}
    await charge_customer(event, client)
    await store.put(key, "done")

    # [5] correct but ugly: nested ifs left in on purpose
    if event.get("type"):
        if event["type"] == "refund":
            if event.get("amount"):
                await notify_finance(event)
    return {"status": "ok"}


# clients/http.py
# [2] hallucinated flag: httpx.AsyncClient has no `retry_on_status` parameter
client = httpx.AsyncClient(timeout=10, retry_on_status=[502, 503])


# migrations/0042_rename.py
# [4] scope creep: ticket said nothing about renaming this column
op.alter_column("payments", "amount", new_column_name="amount_cents")


# tests/test_payments.py
# [3] test mocks away the thing it should check
async def test_duplicate_webhook_not_charged_twice(mocker):
    store = mocker.AsyncMock()
    store.exists.return_value = True   # always 'duplicate'
    result = await handle_payment_webhook(EVENT, store, client)
    assert result["status"] == "duplicate"

• ▹A subtle race or ordering bug. Two concurrent retries can both get past `exists()` before either one calls `put()`, and the customer is charged twice. That's the exact failure the ticket asked to prevent. Reordering the lines won't fix it; you need an atomic set-if-absent or a unique constraint.
• ▹A hallucinated API or config flag. `retry_on_status` reads like a real parameter, but it doesn't exist. With httpx you get a TypeError the moment the client is constructed. A library that silently accepts unknown options would just ignore it. Either way, the retry behaviour the PR depends on isn't there. (httpx's built-in retries live on the transport and only cover connection failures, not status codes.)
• ▹A test that asserts nothing, or mocks away the thing it should check. The 'duplicate' test hard-codes `exists` to return True, so it never touches a real store and never sees concurrent retries. It passes with the race fully in place. CI shows green and the test proves nothing.
• ▹A change that quietly goes beyond the ticket. A column rename in a migration, which breaks every reader of that column, slipped into an idempotency PR.
• ▹One correct but ugly section, left in on purpose. The nested ifs work, and it's fine to flag them as a nit. If a candidate spends ten minutes here and never finds the race, they've shown us how they set priorities.

5. The rubric: what a strong review looks like

We score the review, not the number of bugs found. Two issues that are ranked correctly and come with a plan to prove them beat eight issues listed with equal weight.

• ▹They rank by severity. 'This charges customers twice' is kept apart from 'this is hard to read', and they say which issues block the merge.
• ▹They say how they'd confirm a suspicion rather than just asserting it: 'I think this races. I'd prove it by firing two concurrent requests with the same key in a test and asserting one charge.'
• ▹They know when to reject instead of patching. If the core approach (check-then-act) is wrong, five line comments are the wrong response. The right one is 'reject, redo with an atomic guard, split the migration into its own PR'.
• ▹They check claims against reality. Instead of trusting that a parameter exists because it looks plausible, they open the httpx docs or run the code.
• ▹They read the tests as hard as the code. For each test they ask what would make it fail, and they notice when the answer is nothing.
• ▹They have questions for whoever wrote the prompt. 'Did the prompt mention concurrency? Who asked for the column rename? Was the agent told which HTTP client version we use?' Strong reviewers treat the prompt as part of the change.

Here's the same bug reviewed two ways:

WEAK
> Possible race condition here? Maybe add a lock.

STRONG
> BLOCKING. exists() then put() is not atomic. Two provider
> retries arriving together both see 'not exists' and both call
> charge_customer -> double charge, which is the exact bug this
> ticket is meant to fix.
> To confirm: add a test that runs two handlers concurrently
> with the same key against a real store (not a mock) and
> asserts charge_customer was called once. I expect it to fail.
> Fix: atomic set-if-absent (e.g. Redis SET NX, or a unique
> constraint on idempotency_key) BEFORE charging. An in-process
> lock won't help across multiple workers.
> Q for prompt author: was the agent told we run >1 worker?

The weak comment isn't wrong. The problem is that it hedges, it doesn't say how bad the bug is, and its fix (an in-process lock) won't hold across workers. The strong comment states the impact, explains how to prove it, offers a fix that actually works, and asks the one question that stops the agent making the same mistake next time.

6. What went wrong with the new format

This isn't a perfect filter. If we pretended it was, it would turn into the next LeetCode.

• ▹The memorised checklist. Within a few months some candidates showed up with an 'AI bug checklist': grep the imports for hallucinated APIs, look for mocked tests, check scope. They found the surface bugs fast and then stalled on the race, because a checklist can't reason about concurrency for you. We now give more weight to the 'how would you confirm it' and 'reject or patch' signals than to spotting.
• ▹Rotating the plants. We keep a pool of base PRs and bug variants, rotate them regularly and vary the mix. Some PRs have no hallucinated API at all. Now and then one is almost clean apart from a single serious issue. Candidates who assume there are exactly five bugs start inventing problems, which tells us something as well.
• ▹Thinking out loud isn't neutral. It favours fluent speakers and extroverts, so candidates now write their review comments in silence first and talk us through them afterwards.
• ▹Interviewers slide back into counting. Left uncalibrated, panelists score by number of bugs found. To counter that, we run shadow sessions where two interviewers score the same review and compare notes.
• ▹It doesn't test everything. You learn little about system design, about how someone works with people, or about writing code from scratch when no agent can help. It replaces one round, not the whole loop.

7. If you're the candidate: how to practise this week

• ▹Review your agent's PRs as if a stranger wrote them. Before you accept any agent change this week, write at least one real review comment on it in the format above: severity, evidence, fix.
• ▹Log every agent mistake you catch, one line each: date, category (race, hallucinated API, empty test, scope creep, other), and how you caught it. A week of this shows you your agent's habits and your own blind spots, and gives you concrete stories for interviews.
• ▹For every test the agent writes, ask what would make it fail. Then break the code on purpose and check that the test catches it.
• ▹Talk through one review out loud in under 10 minutes, either recorded or with a colleague. Practise saying 'this one blocks, these are nits, here's how I'd prove it.'
• ▹Keep doing some LeetCode, because plenty of companies still ask it. Just don't let it take up all your prep time.

8. If you're joining a panel: one change for next sprint

You don't need to rebuild your loop. Take the coding question your team already uses and have an agent solve it. Plant two flaws in the solution, one serious and one cosmetic, and make that the second half of the existing round: 20 minutes writing code, then 20 minutes reviewing 'a teammate's' solution. Add one line to the scorecard: 'Did the candidate rank issues by severity and say how they'd confirm them?' Try it with two or three candidates, compare notes with the other panelists, and bring what you learned to your next hiring sync. Small changes with evidence behind them get adopted far more easily than a proposal to redesign the whole process.

Day 26 continues from here.

Extend your knowledge

• ▹Read Google's engineering practices guide on code review, especially the 'What to look for in a code review' section, and see where its priorities differ from the rubric above.
• ▹Start your agent-mistake log today and look back at it after 7 days. Which category came up most, and which one did you nearly miss?
• ▹Write a deliberately flawed PR in your own codebase with one race, one empty test and one bit of scope creep. Ask a teammate to review it cold and note what they catch.
• ▹Go through your company's current interview scorecard and mark which lines test producing code and which test judging it.