The three-PR interview

Day 30: The Three-PR Interview, or How Hiring Now Tests Judgment

The next technical interview you sit probably won't ask whether you can write the code. An agent can write it. What it will ask, often without saying so, is whether you can tell when the agent's code is wrong. Plenty of strong engineers fail that test. You can practise for it on purpose, and you can start this week. Hiring changes before job titles do.

1. Two candidates, three PRs

Earlier this year I gave the same 20-minute exercise to two candidates in the same week. The first had seven years of experience, a long list of shipped work and excellent references. The second had about two years, and most of the last one had been spent reviewing Claude Code and Cursor output at a small startup. Both got the same ticket and the same three pull requests. An agent had written all three PRs, and all three had green CI.

The ticket asked for a payment webhook handler, and one line in it was vague on purpose: "Handle retries." Below is the PR with the hidden bug. It's clean, typed and short, the kind of thing you'd approve before your coffee goes cold.

# PR 2: clean, readable, green CI
@router.post("/webhooks/payments")
async def payment_succeeded(event: PaymentEvent, db: Session = Depends(get_db)):
    order = db.get(Order, event.order_id)
    if order.status == OrderStatus.PAID:
        return {"ok": True}  # already processed -> idempotent

    # "Handle retries": retry the email up to 3 times with backoff
    await send_with_retry(email.send_confirmation, order, attempts=3)

    order.status = OrderStatus.PAID
    order.paid_at = event.created_at
    db.commit()
    return {"ok": True}


# The test that makes CI green
def test_duplicate_event_is_idempotent(client, order):
    client.post("/webhooks/payments", json=evt(order))
    client.post("/webhooks/payments", json=evt(order))
    assert order_status(order) == "paid"
    assert email_outbox.count() == 1

Nothing here is a syntax error. The bug is an assumption. The agent read "handle retries" as "retry our outbound email." With webhooks, retries usually mean the reverse: the payment provider redelivers the same event when it doesn't get a 2xx back in time, and most providers deliver at least once. That means the check-then-act guard only holds when duplicates arrive one after the other. The test sends them one after the other, so it passes. Production traffic doesn't wait its turn like that. It gets worse. The email retry loop with backoff slows the handler down, and a slow handler is exactly what makes the provider redeliver. The code that handles one reading of the vague line causes the failure under the other reading.

The first candidate worked through the PRs in order and talked mostly about style. They liked PR 2's typing and structure. PR 3 got "hard to maintain, raw SQL, would block on readability." Their ranking was PR 2, then PR 1 a close second, then PR 3. They were articulate and confident, and they got it wrong.

The second candidate went to the test files before looking at the handlers. Their first question was "Who is retrying here, us or the provider?" The next one was "These duplicate tests are sequential. What happens if two deliveries overlap?" They put PR 1 first and PR 3 second ("ugly, but it's the only other one that's actually idempotent"), and said PR 2 must not ship. Then, without being asked, they rewrote the ticket line.

In 2024 I would have hired the wrong one.

2. The quiet switch: why I dropped "How would you build this?"

When I was a CTO, my favourite interview question was "How would you build this?" It worked because a reasonable design used to be expensive to produce. It took experience, pattern recall and time. If someone could do it fluently in an interview, they could most likely do it on the job.

Today an agent gives you a reasonable-looking answer in seconds. Ask Claude, Codex or Cursor to design a webhook handler and you'll get something plausible, often in several versions. The answer is cheap now. The expensive part is knowing whether it's right for this system, under these failure modes, given everything the spec left out.

• Cheap now: code, designs, and tests that pass.
• Still expensive: catching the wrong assumption, seeing which tests prove nothing, finding the gap in the spec, deciding what ships.
• That's why at PhoenixDX I interview for the expensive part. Our job descriptions still say Python, TypeScript and Postgres because recruiters filter on keywords, but the interview itself is checking judgment.

3. The exercise in full (run it yourself)

Here's the whole exercise, so you can run it on yourself or with a friend. It takes 20 minutes. You read, you rank, you explain. You don't write any code.

## TICKET PAY-212: Mark orders paid from provider webhook

When the payment provider sends `payment_succeeded`, mark the order
as paid, record paid_at, and send the customer a confirmation email.
Handle retries.

Acceptance:
- Order status becomes PAID
- Customer receives a confirmation email
- Tests included

"Handle retries" is the vague line. It doesn't say who retries, what counts as a duplicate, or what must happen exactly once. Agents fill gaps like this with the statistically most likely guess, and that guess can land either way.

• PR 1, clean and correct. Typed models. It dedupes on the provider's event ID with a unique constraint inside one transaction, writes the email to an outbox table instead of sending it inline, and includes a test that fires two deliveries concurrently. This one ships.
• PR 2, clean and subtly wrong. That's the code above: check-then-act on order status, inline email with retry and backoff, and tests that only cover sequential duplicates. CI is green. It must not ship.
• PR 3, messy but correct. Raw dicts, inline SQL, a function called `handle`, hardly any type hints. Underneath all that, it dedupes atomically on event ID, uses an outbox, and has a concurrency test. It ships after a cleanup pass. Even without PR 1 in the picture, it would ship as-is before PR 2 ever did.

# PR 3: messy but correct
@router.post("/webhooks/payments")
def handle(e: dict, db=Depends(get_db)):
    # provider redelivers on timeout / non-2xx, so dedupe on event id
    with db.begin():
        r = db.execute(text("""
            INSERT INTO processed_events (event_id) VALUES (:id)
            ON CONFLICT (event_id) DO NOTHING RETURNING event_id
        """), {"id": e["id"]}).first()
        if r is None:
            return {"ok": True}  # seen it, ack fast
        db.execute(text("UPDATE orders SET status='paid', paid_at=:t WHERE id=:o"),
                   {"t": e["created"], "o": e["data"]["order_id"]})
        db.execute(text("INSERT INTO outbox (kind, ref) VALUES ('order_confirmation', :o)"),
                   {"o": e["data"]["order_id"]})
    return {"ok": True}

Then come four questions, always in this order:

• Q1. Which one ships? Rank them and say why.
• Q2. Which one hides a bug? Describe the input or timing that triggers it.
• Q3. What did the spec fail to say? List the decisions the agent had to guess.
• Q4. What would you change in the ticket so this couldn't happen again?

A strong Q3 answer names the missing decisions. (a) Delivery semantics: the provider delivers at least once and will redeliver. (b) The dedupe key: event ID or order ID. (c) Which effects must happen exactly once from the customer's point of view. (d) How quickly the handler has to acknowledge. A strong Q4 answer looks roughly like this:

## TICKET PAY-212 (revised)

Provider delivers `payment_succeeded` AT LEAST ONCE and redelivers
if we don't return 2xx within its timeout. Deliveries may overlap.

- Dedupe on provider event id (not order id), atomically.
- Order -> PAID and confirmation email are each exactly-once
  from the customer's view. Email goes through the outbox, not inline.
- Handler must ack fast; no inline network retries.

Acceptance:
- Test: two concurrent deliveries of the same event -> one PAID
  transition, one outbox row.
- Test: two different events for the same order -> defined behaviour (state it).

4. What separates strong answers from weak ones

• They read the tests before the code. Strong candidates ask what the tests actually prove. Weak ones see a green check and take it as evidence.
• They ask about the spec instead of critiquing style. "Who retries?" is worth more than "I'd extract this into a service."
• They pick messy-but-correct over pretty-but-wrong. You can fix readability in a follow-up PR. You can't take back a silent duplicate side effect, like a second confirmation email already sitting in a customer's inbox.
• They propose a change to the ticket, not only a patch. Patching PR 2 fixes one PR. Fixing the ticket fixes the next ten agent runs.
• They name a concrete failure. "Two overlapping deliveries of evt_123" beats "there might be race conditions."
• They separate what they know from what they suspect. "I'm sure PR 2 double-sends; I suspect PR 1's outbox needs a dispatcher we haven't seen." That's how senior reviewers talk.

5. Where the course meets the job market

Go back to the four questions. "What did the spec fail to say?" is the spec-writing work from early in this series. "Which tests prove nothing?" is evals, deciding what evidence counts before you trust a result. "Which one ships?" is the review loop, reading agent output with a model of how it tends to fail. "Change the ticket" closes the loop so the next agent run starts from a better spec. They were never four separate skills. Put together, they're one skill, judging agent work, and that's what this 20-minute interview checks. Day 30 is where the course meets the job market.

6. The uncomfortable part for mid-career engineers

I've seen solid, experienced implementers score lower on this exercise than juniors who spent a year reviewing agent output. Talent has nothing to do with it. What matters is which skills each group has practised.

• Implementers have spent years training "I'll make it work." When you write the code yourself, you made the assumptions, so you can't see them. You don't stop to check them.
• Agent reviewers spent a year training "what did it assume?" Every PR they read came from something that guesses with total confidence, so they learned to hunt for the guess.
• Experience still counts for a lot. The first candidate knew Postgres better than the second did. In this format, though, domain knowledge only helps if you aim it at someone else's assumptions instead of your own build plan.
• The good news is that this habit trains fast. In my experience, experienced engineers who practise for a couple of weeks usually pass the juniors, because once they start hunting for failure modes they recognise far more of them.

7. The 7-day practice plan

One rule applies every day: decide which solution ships and write down why before you run anything. If you run the code first, you skip the judgment, and the judgment is the thing you're training.

• Day 1: Take a real ticket from your backlog, or an open GitHub issue. Have an agent produce 2 solutions in separate branches or worktrees. Write your ship decision in 3 sentences, then run the tests and compare.
• Day 2: Do the same with 3 solutions. This time read only the tests first and predict which solution is weakest before you look at any code.
• Day 3: Pick a ticket with an obvious ambiguity and leave it alone. Let the agents guess. Log each different guess as a 'spec didn't say' entry.
• Day 4: Failure modes day: concurrency, retries, partial failure, timezones, empty inputs. For each solution, write one input that would break it.
• Day 5: Rewrite yesterday's ticket so the ambiguity is gone and rerun the agents. Did the solutions converge? If so, your spec is doing its job.
• Day 6: Swap with a friend. Each of you builds a three-PR set (correct, subtly wrong, messy-correct) for the other. Keep it to 20 minutes.
• Day 7: Read back through your log and group the entries. What do specs in your domain keep failing to say? That list of patterns is your interview material.

## What the spec didn't say: log

| Date  | Ticket   | Missing decision                 | How agents guessed           | Cost if wrong                | My ticket fix                       |
|-------|----------|----------------------------------|------------------------------|------------------------------|-------------------------------------|
| 10/28 | PAY-212  | Who retries? Dedupe key?         | 2 of 3: outbound email retry | double email / side effects  | State at-least-once + event-id dedupe |
| 10/29 | SRCH-88  | Case-sensitive match?            | split 50/50                  | missed results               | Add example table to acceptance     |

Ship decision (written BEFORE running): ______  Why: ______
Was I right after running? Y/N  What did I miss? ______

Take this log to your next interview. When someone asks "How do you work with AI tools?", skip "I use Cursor every day." Open the log and walk them through two entries. It shows how you think, and almost no other candidate will have anything like it.

8. What I'm changing next quarter

• Every engineering loop at PhoenixDX gets a three-PR round. It replaces our old take-home build exercise; it isn't an extra step on top.
• Candidates can use whatever agent they like during the round. If the agent spots the bug, I expect the candidate to explain why it's a bug. If the agent misses it, I want to see whether the candidate catches it anyway.
• Q4 (change the ticket) gets explicit weight in scoring. It's the clearest sign that a person will improve the whole team's agent output, not only their own.
• Scenarios will rotate across domains (payments, search, auth, data pipelines) so nobody can memorise a single bug.

An open invitation: run the three-PR drill above, write your answers to all four questions, and send them to me. I'll reply with how I'd score them. I'm most curious about Q3 answers that find a gap I didn't plant.

Agents write the code now. What gets you hired is judgment: knowing which code to trust, which to reject, and what the spec should have said in the first place.

Extend your knowledge

• Read Stripe's documentation on webhook best practices (duplicate events, fast acknowledgement) and Brandur Leach's Stripe engineering post on designing APIs with idempotency keys. The PR 2 bug is a textbook case of what both warn about.
• Work through the chapters on transactions and stream processing in Martin Kleppmann's 'Designing Data-Intensive Applications'. They'll give you a longer list of failure modes to check agent code against.
• Skim Google's public Engineering Practices guide (the Code Review Developer Guide). Note which advice still holds when the author is an agent, and what you'd add.
• This week, build your own three-PR set from a real ticket in your domain, swap it with a peer, and start your 'what the spec didn't say' log.