Circuit breakers

Day 19: Circuit Breakers, From API Calls to Agent Behaviour

A retry is a bet that the failure will clear up on its own. Sometimes it won't, and then something in your system has to decide to stop. For a normal service, that decision saves threads and connections. For a coding agent, it saves your token budget, your codebase and the attention of whoever has to review what the agent produced.

Hook: the 38th patch

In our PhoenixDX agent harness, we asked a coding agent to fix one failing integration test. Every dashboard stayed green. LLM API latency was normal and the error rate was zero. The circuit breaker on the model provider never tripped. When someone finally opened the session, the agent was on its 38th patch for that one test. The test still failed. The diff had grown from a few lines to several files: a changed fixture, a widened type, a commented-out assertion. Each iteration was a little worse than the one before.

As far as our breaker could tell, nothing was broken. So today's question is: what should have stopped it?

Where this sits in the course

Day 18 covered retries and timeouts. A retry is a bet that the failure is temporary: wait a moment, try again, and it'll probably work. A circuit breaker handles the cases where that bet loses. It watches outcomes, decides the failure isn't transient, and stops trying for a while. Retries are about how to try again. The breaker is about when to stop. You usually need both, and how you nest them is a design decision you make on purpose.

The pattern in three states

A circuit breaker is a small state machine that sits between a caller and something it depends on. Michael Nygard popularised it in Release It!, and Martin Fowler's short article spread it further. It has three states:

• ▹Closed is the normal state. Calls go through and the breaker counts failures. Once the count hits the threshold, it trips to Open.
• ▹Open means fail fast. The breaker rejects calls immediately and never touches the dependency. It stays like this for a cool-down window.
• ▹Half-open comes after the cool-down. The breaker lets a few trial calls through, often just one. If they succeed, it closes again and resets the count. If they fail, it reopens and the cool-down starts over.

Every breaker makes you choose three settings. Libraries like resilience4j, Polly and pybreaker expose all three, and their defaults are rarely right for your system.

• ▹Failure threshold: the number of failures, or the failure rate, that trips the breaker.
• ▹Cool-down window: how long the breaker stays open before it allows a probe.
• ▹What counts as a failure: which outcomes add to the counter. This is the setting that matters most, and teams usually spend the least time on it.

The classic example, kept short

Say your checkout service calls a payment provider and the provider goes down. With no breaker, every request waits for a timeout and then retries a few times. Under load, that's thousands of threads or connections waiting on a dead dependency. Your connection pool runs dry, and requests that never needed payments start failing too. Their outage has become yours. Meanwhile, your retry storm keeps hammering the provider just as it tries to come back up.

A breaker protects two things. First, the caller's resources: it fails in microseconds instead of holding a thread for 30 seconds. Second, the callee's chance to recover: traffic stops while it restarts. LLM providers work the same way. If a model endpoint is returning 529s or rate-limit errors, the breaker should send traffic to a fallback model or queue the work. It shouldn't let a hundred agents keep hitting the endpoint.

The twist: what counts as a failure

Our LLM API breaker worked correctly. The problem was that it guarded the wrong layer. It counted HTTP errors and timeouts, and the agent never caused either. Every call returned a 200 with a well-formed tool call. Every patch applied cleanly. The test runner finished on time. At the transport layer, it was a perfect run.

The failure that costs you with a coding agent is semantic. It's a fix, break, fix loop where every single call technically succeeds. These are the signals worth counting:

• ▹The same test fails after a different patch. The agent tried a new approach, and that one didn't work either.
• ▹The diff keeps growing while the passing-test count stays flat or drops, so the agent is adding code without getting closer.
• ▹The agent starts editing outside the failing test's area: fixtures, mocks, the test itself. That's often a sign it has switched from fixing the bug to making the test pass.

Pitfall: if your agent breaker counts "tool call errors" or "model errors" as failures, it will stay happy while the agent burns through tokens. Define failure by the outcome you actually care about, not by what the protocol reports.

Mapping the breaker onto agent reasoning

All three states carry over to agents. The difference is what Open means. For an API, Open means "don't call". For an agent, Open means "stop writing code and explain yourself".

• ▹Closed (edit mode): the agent has its full toolset. It can read, search, edit files, apply patches and run tests. The breaker counts failed test runs on the target.
• ▹Open (diagnose-only mode): the write tools are taken out of the toolset. Asking nicely in the prompt doesn't count. The agent can still read code and run tests, but it has to write a diagnosis in three parts: what it tried, what it thinks the root cause is, and what it's unsure about. A human gets that diagnosis.
• ▹Half-open (one approved probe): a human or a planner agent reads the diagnosis and approves one targeted edit, ideally with a narrower scope such as "only touch parser.py". If the run passes, the breaker closes. If it fails, the breaker goes straight back to Open.

The design choice that matters here is enforcing Open by removing tools, not by adding instructions. If the prompt says "please stop editing", that's a suggestion, and models are good at talking themselves out of suggestions. With no write tools, the agent has no way to keep editing.

How-to: a minimal implementation

Here's a minimal version in Python. The breaker is keyed on (test_id, file set). It only counts a failed run when the patch is new, which it detects with a patch hash, and it resets on a pass. When it trips, it swaps the toolset and returns a diagnosis prompt.

import hashlib
from dataclasses import dataclass, field

EDIT_TOOLS = {"read_file", "search", "run_tests", "write_file", "apply_patch"}
DIAGNOSE_TOOLS = {"read_file", "search", "run_tests"}  # no writes

DIAGNOSIS_PROMPT = (
    "You have made {n} different fixes for {test} and it still fails. "
    "You can no longer edit files. Write a diagnosis: "
    "1) what you tried, 2) what you believe the cause is, "
    "3) what you are unsure about (spec, test, environment?)."
)

@dataclass
class Breaker:
    state: str = "closed"
    failures: int = 0
    seen_patches: set = field(default_factory=set)

class AgentBreakers:
    def __init__(self, threshold=3):
        self.threshold = threshold
        self.breakers = {}

    def _get(self, test_id, files):
        return self.breakers.setdefault((test_id, frozenset(files)), Breaker())

    def record_run(self, test_id, files, patch, passed):
        b = self._get(test_id, files)
        if passed:
            b.state, b.failures = "closed", 0
            b.seen_patches.clear()
            return EDIT_TOOLS, None
        h = hashlib.sha256(patch.encode()).hexdigest()
        if h not in b.seen_patches:      # a *different* fix that still failed
            b.seen_patches.add(h)
            b.failures += 1
        if b.state == "half_open" or b.failures >= self.threshold:
            b.state = "open"
            return DIAGNOSE_TOOLS, DIAGNOSIS_PROMPT.format(n=b.failures, test=test_id)
        return EDIT_TOOLS, None

    def approve_probe(self, test_id, files):  # called by a human or planner
        self._get(test_id, files).state = "half_open"

Why the key matters:

• ▹A per-session counter is too blunt in both directions. An agent that fixes five different tests in one session, with one honest retry each, would trip a session-wide breaker for nothing. An agent stuck on one test inside a long, otherwise productive session might never reach the threshold at all.
• ▹Keying on test_id alone lumps separate problems together. One test can fail for unrelated reasons in different parts of the code, and adding the file set keeps those apart.
• ▹Only new patches count. If the agent re-runs the same patch, you're more likely looking at a flaky test or a bad environment than a reasoning failure. Track that with its own signal so it doesn't inflate this counter.
• ▹Pitfall: touching one extra file changes the file set, which lets the agent slip out from under the key. In practice, normalise the file set to the module or directory that owns the failing test, or keep a looser per-test counter with a higher threshold as a backstop.

Optional extension: also trip the breaker when the diff grows over three runs in a row and the passing-test count doesn't go up. That catches the slow-drift loops, where every patch is different and none of them gets any closer.

Choosing thresholds honestly

I use 3, not 5, and here's my reasoning. The first failed fix tells the agent something about the problem. The second is a revised hypothesis, which is still fair. When a third, different fix fails too, the agent's model of the problem is usually wrong, and more attempts from a wrong model tend to make the diff dirtier. I got that number from watching our own sessions. It isn't a universal constant, so tune it for your codebase and your models.

• ▹Trip too early and you get annoying hand-offs. People receive diagnoses for problems the agent would have solved on attempt four, and they start ignoring the breaker. It's alert fatigue in a new form.
• ▹Trip too late and someone has to untangle a dirty diff. You've also wasted tokens, and you may get changes that look plausible, like a weakened assertion, which a tired reviewer could approve.
• ▹The costs aren't symmetric. An extra hand-off costs a few minutes, while a bad change that lands on main can cost far more. If you're unsure, trip earlier, and make the diagnosis good enough that people want to read it.

Cool-down works differently for agents. For a payment service it's measured in seconds, because you're waiting for the dependency to recover. An agent doesn't recover by waiting, and the model won't be any smarter in 30 seconds. For an agent, cool-down is measured in human attention. The breaker stays open until someone reads the diagnosis and then approves a probe, fixes the spec or test, or reassigns the task. A timer-based half-open on an agent just restarts the loop.

What changed after shipping it

• ▹It cut thrash loops more than any prompt rewrite we tried. We'd already added instructions like "if you are stuck, stop and ask", but models rarely notice from the inside that they're stuck. An outside counter that can take tools away worked where those instructions didn't.
• ▹The diagnoses turned out to be useful in their own right. They often pointed at something other than bad code: an ambiguous or contradictory spec, a flaky test, a fixture that didn't match production. A human needed to fix those anyway, and no amount of patching by the agent would have got past them.
• ▹Reviews got easier. When someone picks up an open breaker, they start from a short written account of what was tried instead of reverse-engineering a long diff.
• ▹It didn't catch agents that make the test pass the wrong way, for example by over-mocking. The breaker only sees failure, so for test-gaming you still need review and diff-scope checks.

Exercise for today

• ▹Pick one place in your system: an API client, an LLM provider call or an agent loop.
• ▹Write a one-sentence definition of failure for that place. If it's an agent loop and your sentence mentions HTTP status codes, try again.
• ▹Pick a threshold and write one line on why you chose that number.
• ▹Decide what Open actually does. It could fail fast, fall back to another model, queue the work, or revoke write tools and ask for a diagnosis. Then decide who or what moves it to half-open.

Coming up on Day 20: a breaker on one node only protects that node. When many agents share a dependency, such as a model endpoint, a test environment or a planner, one node's failure can still take down the whole graph. Tomorrow covers bulkheads and how to contain failure in multi-agent systems.

Extend your knowledge

• ▹Read the Circuit Breaker and Bulkheads sections in the Stability Patterns chapter of Michael Nygard's "Release It!" (2nd edition). It's the book that popularised the pattern, and it sets up Day 20.
• ▹Read Martin Fowler's short "CircuitBreaker" article on martinfowler.com. Then compare its settings with a real library: resilience4j (Java), Polly (.NET) or pybreaker (Python).
• ▹Get the retry layer right before you wrap a breaker around it. The AWS Builders' Library article "Timeouts, retries, and backoff with jitter" is a good guide.
• ▹Before you ship a breaker in your own agent harness, log every test run for a week with (test_id, files touched, patch hash, pass/fail). Replay the log with thresholds of 2, 3 and 5 and see where each one would have tripped.