Verifier agents

Day 29: Verifier Agent

A badly built verifier is worse than no verifier at all, because its green checkmark tells everyone they can stop looking. Once agents write most of your diffs, the review step is what decides whether you ship work that's been checked or work that only looks right.

The hook: a week of green, then two incidents

We once put an audit agent behind our coding agent. For a week, every ticket came back 'pass', and the dashboard stayed green. Then we had two production incidents. When we replayed both tickets, the verifier's 'pass' rested on the same thing both times: the coder's own account of what it had done. The verifier never ran anything. It read a confident summary and agreed with it.

That replay taught us what a verifier is and what it isn't. This lesson covers both.

The definition in one breath

A verifier agent is a separate agent with one job. It checks an artifact (a diff, a migration, a report) against the original spec and shows evidence for every judgment it makes. It isn't a code reviewer with opinions. It checks claims, and it brings proof.

Banks call this the maker-checker principle, or the four-eyes rule: the person who enters a payment can't also approve it. We're applying the same rule to agents. Earlier in this series we split the orchestrator and worker roles. The verifier is a third role outside that loop, and it never builds anything.

A useful verifier has two properties:

• ▹Independence. It doesn't see the builder's reasoning, and it differs from the builder in at least one of four ways: model, context, tools or evidence.
• ▹Honesty about its limits. Instead of pass/fail, it returns verified, refuted or unverifiable.

Why the obvious version fails, part 1: anchoring

The obvious setup gives the verifier everything: the ticket, the diff, the builder's chain of thought and its final summary ('I added retry logic with exponential backoff and handled the timeout case'). That feels thorough. In practice the summary becomes an anchor, and the verifier ends up grading how convincing the story is instead of the code.

We saw approvals go up and real catches go down. A well-written explanation of a wrong change got through more easily than a short explanation of a correct one. LLM-as-judge research has documented similar biases, such as preferring longer answers and favouring outputs that resemble the judge's own. A stricter prompt won't fix this. You fix it by removing the anchor from the input.

Why the obvious version fails, part 2: correlated errors

The second failure is quieter. Our verifier ran on the same model as the builder and used the same retrieved context. The builder hallucinated a client library method that didn't exist in the version we'd pinned. The verifier read the diff and 'remembered' the same method, because it had the same training priors and saw the same misleading context. It confirmed the bug.

Layered checks only help if each layer fails for different reasons. A second opinion with the same blind spots as the first is really one opinion counted twice. It makes you more confident without making you more correct.

Rule 1: Independence

Limit the verifier's input to three things:

• ▹The ticket: the original spec and acceptance criteria, as written before any work started.
• ▹The diff: the artifact alone, without the builder's commentary, PR description or reasoning trace.
• ▹A sandbox, where it can check out the branch, run tests, call the endpoint and read logs.

Then make sure it differs from the builder in at least one of these ways. More than one is better:

• ▹Model. Use a different model family or version so their training-data blind spots don't line up.
• ▹Context. Start it fresh, with no shared retrieval results, scratchpad or conversation history.
• ▹Tools. Give it different capabilities. For example, the builder can edit files, and the verifier can only run things and observe the results.
• ▹Evidence. It judges from what it sees at runtime (test output, HTTP responses, logs, type-checker results), not from reading code.

Evidence matters most of the four, because running code tells you more than reading it. A verifier that reads a diff can be fooled by plausible-looking code in the same way the builder was. A verifier that runs `pytest -k test_retry_on_timeout` and pastes the output has made the claim something anyone can check.

Rule 2: Three verdicts, not two

With only pass and fail, the verifier has to give a false answer whenever it can't tell. Say the spec reads 'handle slow upstreams gracefully' and no fixture simulates a slow upstream. A binary verifier has to guess, and in our experience LLMs tend to guess 'pass'. So give it a third option:

• ▹verified: the claim holds, with evidence attached (command, output, log line).
• ▹refuted: the claim fails, again with evidence (the failing test, the wrong response).
• ▹unverifiable: the claim can't be confirmed or denied. The verdict says what's missing: a fixture, a log field or a precise definition in the spec.

The verifier first splits the spec into separate testable claims, then gives each claim a verdict. The ticket's status comes from the claim verdicts. If any claim is refuted, the ticket is refuted. Otherwise, if any claim is unverifiable, the ticket is unverifiable. The ticket is verified only when every claim is verified. Here's a schema you can copy. Use structured output to get the shape, and also run the response through a full JSON Schema validator, because many structured-output modes don't support conditional keywords like `allOf` and `if`/`then`:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "VerifierReport",
  "type": "object",
  "required": ["ticket_id", "claims"],
  "properties": {
    "ticket_id": { "type": "string" },
    "claims": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["claim", "spec_ref", "verdict"],
        "properties": {
          "claim":    { "type": "string", "description": "One testable statement taken from the spec" },
          "spec_ref": { "type": "string", "description": "Quote or line of the ticket this claim comes from" },
          "verdict":  { "enum": ["verified", "refuted", "unverifiable"] },
          "evidence": {
            "type": "array",
            "items": {
              "type": "object",
              "required": ["kind", "ref", "excerpt"],
              "properties": {
                "kind":    { "enum": ["test_run", "command_output", "http_response", "log_excerpt", "static_check"] },
                "ref":     { "type": "string", "description": "Exact command or file:line" },
                "excerpt": { "type": "string" }
              }
            }
          },
          "missing": { "type": "string", "description": "Required if unverifiable: what evidence would settle it" }
        },
        "allOf": [
          { "if": { "properties": { "verdict": { "enum": ["verified", "refuted"] } } },
            "then": { "required": ["evidence"], "properties": { "evidence": { "minItems": 1 } } } },
          { "if": { "properties": { "verdict": { "const": "unverifiable" } } },
            "then": { "required": ["missing"] } }
        ]
      }
    }
  }
}

Look at the `allOf` block. It rejects a 'verified' or 'refuted' verdict that has no evidence, and an 'unverifiable' verdict that doesn't say what's missing. Those two rules are what keep the verifier honest. Here's what one claim looks like in practice:

{
  "claim": "Requests to /export retry up to 3 times on upstream timeout",
  "spec_ref": "AC2: 'retry transient upstream failures, max 3 attempts'",
  "verdict": "unverifiable",
  "evidence": [],
  "missing": "No fixture simulates an upstream timeout; retry attempts are not logged, so attempt count cannot be observed"
}

The surprise: the unverifiable list wasn't about code

After a month of three-verdict reports, we read every unverifiable claim. We expected subtle code problems. Most of what we found was about how we work as an engineering team:

• ▹Vague spec sentences such as 'handle errors gracefully', 'should be fast' and 'support large files'. None of them has an observable pass condition, so nobody, human or agent, can verify them.
• ▹Missing test fixtures. We had no way to simulate a slow upstream, an expired token or a partially written file.
• ▹Logs that couldn't confirm what happened. We logged that a job started and finished but not which branch it took, so nobody could tell whether the fallback path ran.

The verifier had given us a map of where our specs, test fixtures and logging were too weak. None of these gaps were new. They existed before we had agents, and human reviewers had been covering them with judgment calls and approving anyway. The agent wouldn't guess, so it reported them.

What to do with each bucket

• ▹Verified: auto-merge, or merge after a quick human look if the change is higher risk. Put the evidence in the PR so anyone can audit it later.
• ▹Refuted: send it back to the builder with the failing evidence, not the verifier's opinion. 'Test X fails with output Y' tells the builder what to fix. 'This looks wrong' starts an argument.
• ▹Unverifiable: send it to a human along with the specific gap. The human decides whether to accept the risk for now. The gap also becomes a ticket to improve the spec template, the fixture library or the logging.
• ▹Then track one metric: unverifiable claims per ticket over time. It tells you how good your specs, fixtures and logging are. If it goes down, your system is getting easier to check. If it stays flat while you ship more, you're shipping more unchecked behavior, faster.

When to use a verifier, and when not to

• ▹Use one when agents produce changes faster than people can review them carefully, when the spec has acceptance criteria you can observe, and when you can give the verifier a real sandbox.
• ▹Don't use one instead of tests. If nothing can be checked by running it, the verifier will either mark everything unverifiable (the honest answer) or read the code and guess (the failure mode).
• ▹Skip it for questions of taste, such as naming or architecture style. Those aren't claims you can back with evidence. Use a linter, a style guide or a human instead.
• ▹Don't add a second or third verifier on the same model with the same context and call it extra safety. Checkers with the same blind spots don't make your review any more independent.

Common pitfalls

• ▹Giving the verifier the PR description 'for context'. That's the builder's summary again.
• ▹Asking the verifier to be strict ('be extremely critical') instead of requiring evidence. Without a sandbox, a strict verifier only gives you more confident opinions.
• ▹Treating unverifiable as a failure and pushing for fewer of them. If you punish that verdict, the verifier learns to say verified.
• ▹Letting the verifier fix the code. Once it edits, it's a second builder, and you no longer have a checker.

Try this week: a 4-step checklist

• ▹1. Remove the builder's reasoning from the verifier's input. Give it the ticket and the raw diff only: no chain of thought, summary or PR description.
• ▹2. Give it a sandbox. At minimum it should be able to check out the branch, run the test suite and run one targeted command per claim. Use a different model from the builder, or at least a fresh context.
• ▹3. Switch to three verdicts. Adopt the schema above and reject any verified or refuted verdict that comes without evidence.
• ▹4. Review the unverifiable log on Friday. Sort the gaps into spec, fixture and observability, and fix the biggest one before next week.

The takeaway

Being strict isn't what makes a verifier useful. Its value comes from being independent of the builder's reasoning, model and context, and from being willing to say what it can't confirm. Verified and refuted verdicts protect the current merge. The unverifiable list improves every merge after it. Tomorrow, on Day 30, we take that list back to where the gaps start and look at how to write a spec an agent can actually be held to.

Extend your knowledge

• ▹Anthropic's 'Building effective agents' essay, especially the evaluator-optimizer pattern. Read it next to this lesson and ask yourself what that evaluator is independent of.
• ▹'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena' (Zheng et al., 2023). It documents position, verbosity and self-enhancement biases, which are the same forces behind anchoring and correlated errors.
• ▹'Large Language Models Cannot Self-Correct Reasoning Yet' (Huang et al., ICLR 2024). It explains why checking without outside feedback often fails, which is why the sandbox and real execution results matter.
• ▹Exercise: take your last 10 merged agent PRs and run them again through a three-verdict verifier that gets only the ticket, the diff and a sandbox. Sort every unverifiable claim into spec, fixture or observability.