Posts tagged LLM As Judge — AI × software engineering, made practical.
You will learn what an eval actually is, why large generated test suites miss the failures real users hit, and how to turn each incident into a small regression case with the cheapest check that catches it.
This lesson shows how to build a verifier agent that checks another agent's work independently of the builder and gives each claim one of three verdicts, verified, refuted or unverifiable, backed by evidence.
Learn why an LLM-as-judge validator needs its own tests, fail-closed defaults, and separate alerting instead of being treated as trusted plumbing.
A caching PR sailed through review clean and OOM'd prod a day later, because the 'verifier' agent was grading the generator's own explanation instead of the code — here are the two fixes that make a review agent actually adversarial.
The score I'd been shipping on for weeks wasn't measuring quality — it was measuring how much my output sounded like the judge itself, and there's a cheap afternoon experiment that exposes it before it costs you more than a prompt.
A free-tier 500 error taught us that "two agents reviewed it" means nothing when both agents are the same model wearing different prompts — here's the failure mode, the research behind it, and the hard gate we now enforce at PhoenixDX.