Zero Errors, Zero Retries, 100% Wrong: Why Circuit Breakers Miss Agent Drift

Why This Matters

Here's the uncomfortable part about circuit breakers in agent pipelines: they work exactly as designed, and that's the problem. A breaker built for HTTP calls only knows how to catch loud failure — 500s, timeouts, dropped connections. Agent pipelines don't fail loud. The call succeeds, the schema validates, latency looks great, and the output is quietly, completely wrong. Watch only status codes and your breaker will wave a broken pipeline through at 100% throughput and call it healthy.

The Six-Hour Drift

Take a pipeline that ran clean all night: zero errors, zero timeouts, zero retries, every dashboard green from midnight to 6am. Six hours in, someone finally opened the actual output and found the pipeline had drifted off-task hours earlier — a small misreading at step one fed into step two, which fed into step three, until the final result had nothing to do with the original ask. Monitoring never flagged it, because nothing technically failed. The system was healthy by every classic metric and wrong by the only metric that mattered.

Quick Refresher: What a Circuit Breaker Actually Is

Quick recap, since we're about to lean on this vocabulary: a circuit breaker trips on an unhealthy signal and cuts traffic to whatever's failing. Closed means traffic flows normally. Open means it's cut off. Half-open means you're testing the water with a trickle of requests before trusting it again. That's the whole concept — three states, one job. The real question for this lesson is narrower: what counts as "unhealthy" when the thing you're protecting isn't a service endpoint, it's a chain of LLM calls?

The Gap: Success Without Correctness

In REST-land, "unhealthy" has a short, well-worn vocabulary: 5xx, timeout, connection refused, a p99 that's crept too high. Those signals exist because failure in a distributed system is usually structural — a process died, a socket refused, a queue backed up. An agent pipeline can sail through every one of those checks — 200 OK, schema-valid JSON, 400ms response — and still be dead wrong in the way that matters: the summarizer paraphrased instead of extracting, the planner solved a different problem than the one it was handed, the coding agent "fixed" a failing test by deleting the assertion. None of that trips a classic breaker, because classic breakers were never built to read meaning. They were built to read plumbing.

The New Tripwire: Intent-Match, Not Status Code

The fix isn't a smarter HTTP check — it's a different sensor entirely. Bolt on a lightweight judge: a scoring check or a small judge-agent that periodically asks one narrow question — does this output still match what the task actually asked for? That score becomes the breaker's input, either replacing the transport-level signal or running alongside it. It doesn't need to run on every call — that's slow and expensive for no reason — it runs on a sample, at a cadence tuned to how fast drift actually compounds in your pipeline.

• ▹Classic signal: HTTP status, timeout, retry count, p99 latency — cheap, synchronous, structural.
• ▹Agentic signal: intent-match score from a judge-agent or rubric-based scorer — sampled, asynchronous, semantic.
• ▹Key difference: the classic signal answers 'did the call complete', the agentic signal answers 'was the call worth completing'.

Mechanics: Open, Half-Open, and Closed for an Intent Breaker

Same state machine as Hystrix, just re-wired to watch semantic health instead of network health.

• ▹Closed: the pipeline runs as usual, but a sampler grabs every Nth output (or one every T minutes) and runs it past the judge-agent.
• ▹Trip condition: not 'error rate > 50% over 10s' but 'intent-match score below threshold across the last K samples' — drift has to be consistent, not a single rough sample, or you'll trip on ordinary LLM variance.
• ▹Open: the whole pipeline pauses, not just individual calls — because the failure mode is cumulative drift across steps, and letting more traffic through just compounds the damage.
• ▹Half-open: instead of 'let the next request through and see if it works,' you deliberately pull a handful of fresh outputs and re-run the intent-check on them. Resume only once that batch passes — one clean call proves nothing about semantic health.

From the Field: When Every Call Looks Healthy

In multi-agent research at RMIT, and in PhoenixDX's own pipelines, the failure that's hardest to catch is never the crash — it's the run where every individual agent call comes back clean and well-formed, retries sit at zero, latency looks normal, and the only place the problem shows up is in the aggregate output quality several hops downstream. A retrieval step quietly starts citing generic knowledge instead of what it actually retrieved. A planning agent starts producing plausible-sounding subtasks that don't decompose the original goal at all. Per call, there's nothing to alert on. Only when you hold the final output up against the original task intent does the gap show up — which is exactly why the check has to look at output and meaning, not transport.

Today's Glossary Line

For agentic systems, unhealthy means intent-drift, not error-rate. Write that down before you copy a Hystrix config straight into an agent framework — it'll run fine and protect you from exactly nothing.

Extend Your Knowledge

• ▹Read the original Hystrix docs on circuit breaker states (closed/open/half-open) — it's the exact vocabulary you're now bending toward semantic health.
• ▹Look at how LLM-as-judge evaluation is described in Anthropic's and OpenAI's eval docs — same scoring pattern, just repurposed for a runtime intent-check.
• ▹If you're building this for real, prototype the sampler+judge as a side-car process reading pipeline outputs off a log or queue, not sitting in the hot path — keeps the intent-check from adding latency to every call.
• ▹Compare this to 'LLM observability' tooling — tracing frameworks built for multi-agent debugging. Most give you spans and latency; almost none give you drift-over-time out of the box. You'll probably end up wiring the judge-agent yourself.