We Deleted a Reviewer Agent From Our Pipeline — Quality Went Up, Not Down

Day 5 — Few-Shot Examples Are Your Cheapest Multi-Agent Code Review

The default fix for a worker agent that drifts in format is a critic agent: read the output, check it, repair it. That's a full extra LLM call, on every single run, forever — to catch a mistake that might not even happen most of the time. Three good examples baked into the worker's own prompt usually stop the drift before it starts, for a fraction of the cost.

The day we deleted a reviewer agent and quality went up

One of our pipelines at PhoenixDX ran a 'formatter checker' agent whose entire job was catching a single recurring bug. A worker agent would occasionally wrap its JSON in a markdown code fence, tack on a friendly 'Here's the result:' — and the downstream parser would choke. The checker read the worker's output, judged whether it was clean, and bounced it back for a fix when it wasn't. It worked fine. It also doubled the token cost of that stage and added a full round trip of latency to every run — including the vast majority where nothing was ever wrong. During a cost review we cut the checker as an experiment, braced for the drift rate to creep back. It didn't. Quality held steady, and in a few batches it actually looked better — because we'd stopped patching the symptom downstream and started preventing it at the source.

What 'format drift' actually looked like

• ▹Worker occasionally wrapped valid JSON in a ```json fence plus 'Here's the result:' — a plain JSON.parse() on the other end just threw
• ▹Field names drifted across sessions: 'severity' one run, 'priority' or 'risk_level' the next, despite the schema being spelled out in the system prompt
• ▹Worker sometimes appended a trailing explanation after the JSON block, 'just to be helpful'
• ▹Drift wasn't random — it clustered on longer inputs and edge-case-y requests, which is the tell that matters

Once you stop treating the system prompt as 'instructions the model reads,' the pattern makes sense. It's one block of text competing for attention against a much stronger, RLHF-baked habit: be a helpful conversational assistant. Under a long context window or a gnarly edge case, that default persona is the path of least resistance — and a formatting rule stated once, in prose, three paragraphs up, is exactly the kind of thing attention deprioritizes. A system prompt describes the shape you want. It doesn't show it.

The instinct everyone has: bolt on a reviewer agent

Bolting on a critic/verifier hop is the obvious move, because it maps straight onto muscle memory every engineer already has: CI, linting, code review. So multi-agent pipelines get modeled the same way — a service generates, a gate checks, the gate blocks or fixes. It feels rigorous. It's also the first pattern most teams reach for the second they spot any QA problem in an agent pipeline, whether or not it's actually the right tool for that specific problem.

Why that instinct is expensive and late

A verifier hop runs on every request, not just the ones that would've drifted — so you're paying the full extra token cost and latency across the whole pipeline to catch a failure mode that might show up one run in ten. Worse, it's downstream of the mistake. By the time the reviewer sees anything, the worker has already committed to a bad generation path — its attention has already settled on the wrong token distribution. Detecting that isn't fixing it. Fixing it needs a second generation pass, and most teams never actually wire that up; the reviewer just flags or logs, and a human ends up hand-fixing format bugs, which defeats the entire point of automating the pipeline.

The fix: 3 examples in the worker's own prompt

So we pulled three real, anonymized input/output pairs straight into the worker's prompt — not a description of the schema, the actual shape of a correct output. One of the three was picked deliberately close to the input pattern that used to trigger the fence-wrapping bug, so the model would see: even on this tricky one, the output is bare JSON. Not just on the easy cases.

SYSTEM PROMPT (excerpt)
...
Output raw JSON only. No markdown, no code fences, no preamble.

Examples:

--- Example 1 (typical case) ---
Input: "Login fails silently after SSO redirect on mobile Safari"
Output: {"title":"SSO redirect silent failure on mobile Safari","severity":"high","next_step":"reproduce on iOS 17 Safari"}

--- Example 2 (long/noisy input) ---
Input: "<400 words of a rambling Slack thread with 3 people arguing about whether this is even a bug>"
Output: {"title":"Rate limit errors on batch export","severity":"medium","next_step":"confirm with API team if limit was recently lowered"}

--- Example 3 (near past failure mode) ---
Input: "<edge case that used to make the worker add a fenced explanation>"
Output: {"title":"...","severity":"low","next_step":"..."}
...

No new agent, no new hop. Same worker call, same latency budget — just a longer prompt, with demonstrations instead of more instructions.

Before / after

• ▹Reviewer-hop pipeline: worker call + checker call on every run = 2x LLM calls, 2x latency, whether or not drift would have happened
• ▹Few-shot pipeline: 1 worker call, a slightly longer prompt (a few hundred extra tokens for 3 examples) — a rounding error next to a whole second LLM call
• ▹Drift went from something we saw regularly enough to justify a dedicated agent, to rare enough that we stopped tracking it as a distinct failure mode
• ▹The failures that remained after switching were logic mistakes (wrong severity call, wrong next step) — not format problems. That's the tell that the fix worked at the right layer

Why this works mechanistically: prevention vs. detection

A reviewer agent operates after the worker has already sampled its tokens — it can only catch a bad distribution after the fact, and 'catch' isn't 'fix' unless you've also built a repair loop. Few-shot examples act before generation. They narrow the space of plausible next tokens the model is sampling from, because now the strongest local pattern in context is 'three times in a row, the output was bare JSON with these exact fields' — not the model's default helpful-assistant persona. You're not catching the mistake after it happens. You're making it statistically less likely to happen at all.

The limit: this only prevents surface-form drift

Few-shot examples fix how the output looks, not whether it's right. A worker can emit perfectly-formed JSON, correct field names and all, and still classify a critical bug as 'low severity.' Examples don't touch that — it's a reasoning error, not a formatting one, and it needs real verification: a rubric, a second model checking substance against ground truth, a human spot-check. Don't let three clean examples convince you the pipeline's output is correct. All they mean is that it's now parseable.

Takeaway

Next time you reach for another agent hop, ask one question first: is this a formatting problem that few-shot examples can solve, or a reasoning problem that needs a real check? Most 'add a reviewer agent' instincts turn out to be the first kind, dressed up as the second. Save the real verification loops for the cases that actually need them — which is exactly where Day 6 picks up: when verification loops earn their keep, and when they're just an expensive way to catch a bug you could've prevented for free.

Extend your knowledge

• ▹Audit one pipeline you run today: find a verifier/critic agent whose only job is catching a formatting bug, and check whether 3 inline examples could replace it
• ▹Log your worker's actual drift cases for a week before writing examples — anonymized real failures beat hand-crafted ones because they match the model's actual failure distribution
• ▹Read Anthropic's prompt engineering docs on few-shot/multishot prompting for the mechanics of example selection and ordering
• ▹Preview Day 6: map which of your reviewer hops are catching formatting (replaceable) vs. catching actual reasoning errors (keep and invest in)