Load shedding for agent systems

Day 22: Load Shedding for Agent Systems: Decide What Not to Do

When an agent fleet is overloaded, some of its work is going to fail. You can't change that. What you can change is which work fails. If you don't pick, the failures land at random. Usually everything slows down together and nothing finishes.

The Tuesday incident

One Tuesday at PhoenixDX, one integration test started failing on and off. We had about 40 coding agents running tasks, and every task ended the same way: run the test suite, then open a PR. We also had retries at three layers. The orchestrator retried a failed task. The subagent retried a failed step. The test-runner tool retried a failed run. Each layer allowed 3 attempts, so that's 3 × 3 × 3 = 27 attempts per task.

Nothing was down. Every dashboard was green. The API was healthy, the queues were moving, the model provider was answering. But the day's token budget was gone by early afternoon, and nothing had merged. Every task got a fair share of a resource that had run out, so none of them got enough to finish. Nobody had decided what to give up, so the system gave up a little of everything.

The one-sentence definition

Load shedding is choosing which work fails so the rest succeeds.

Here's how it sits next to the tools from Days 20 and 21:

• ▹Rate limiting caps what comes in: 'no more than N per second.'
• ▹Backpressure slows the sender down: 'wait, I'm full.'
• ▹Load shedding drops work: 'this task won't run, so that one can finish.'

Rate limiting and backpressure both assume all the work gets done eventually. Shedding is the only one of the three that accepts some work is worth zero right now. That's why it feels uncomfortable. It's also why it works.

Why agents make overload worse

In a classic web system, overload usually comes from outside, like a traffic spike or a launch that goes viral. In agent systems it usually comes from inside, and the main cause is nested retries. Here's the tree for one task:

Now multiply it out:

• ▹Orchestrator: 3 attempts per task
• ▹× Subagent: 3 attempts per step = 9
• ▹× Tool: 3 runs per call = 27 attempts at the flaky test, for each task
• ▹× 40 agents = 1,080 test runs. Each one sits inside LLM calls that read logs, re-plan and patch code.
• ▹Most LLM SDKs also retry by default, including Anthropic's and OpenAI's Python clients. Count that layer and you multiply again.

Check any one layer by itself and it looks fine. Three retries is a sensible default. That's exactly why nobody notices what they add up to.

Agents add a second problem on top of the multiplication. In microservices, a retry sends the same request again with the same payload, ideally with an idempotency key, and it costs about what the first try cost. An agent retry is a different thing. The agent re-plans, reads different files, takes a different path, spends new tokens, and may call tools it never touched the first time. Every retry is a new trajectory. Agent retries are not idempotent. When a retry does more work than the original attempt, it amplifies load.

Retries also get bigger. The second attempt carries the failure context from the first ('the test failed, here's the log'), so its prompt is longer. It often costs more than the first attempt did.

Design rule 1: Shed at the orchestrator, not the edge

The traditional advice is to shed at the edge: the load balancer or API gateway returns a 503 or 429 before any work starts. You still need that to protect the model provider and your own APIs. But in an agent system the edge can't make good shedding decisions, because it has no idea what the work is.

• ▹The API gateway sees a 429 on a POST to the LLM endpoint. That's all it sees.
• ▹The orchestrator knows that call belongs to a low-priority docs refactor with no deadline, and that the call right next to it belongs to a hotfix blocking a release.

Smart shedding needs two inputs: the task's priority and how much of its deadline is left. Only the orchestrator has both. Put the shedding decision where that information lives.

Design rule 2: Propagate a deadline budget down the tree

gRPC is a good model here. A client sets a deadline. When the server passes its context along, downstream calls can inherit the time that's left. A server that sees the deadline has already passed can skip the work. Do the same thing with agents, but carry time and tokens together, because you run out of both.

The rule for children is simple. If you can't finish inside the budget you were given, fail fast and don't start. Starting a task you can't finish is the most expensive way to fail.

import time
from dataclasses import dataclass, replace

class BudgetExceeded(Exception):
    """Raised before work starts. Never retried below the orchestrator."""

@dataclass(frozen=True)
class Budget:
    deadline: float      # absolute time, not a duration
    tokens: int
    priority: int        # 0 = must ship ... 3 = nice to have
    may_retry: bool      # only the orchestrator's budget has True

    def seconds_left(self) -> float:
        return self.deadline - time.time()

    def split(self, share: float, reserve_s: float = 10.0) -> 'Budget':
        # Keep a reserve so the parent can still verify and merge.
        usable = max(0.0, self.seconds_left() - reserve_s)
        return replace(
            self,
            deadline=time.time() + usable * share,
            tokens=int(self.tokens * share),
            may_retry=False,   # children fail fast; they never retry
        )

def run_subagent(step, budget: Budget):
    est = step.estimate()  # rough: seconds + tokens from past runs
    if budget.seconds_left() < est.seconds or budget.tokens < est.tokens:
        raise BudgetExceeded(
            f'{step.id}: needs ~{est.seconds}s/{est.tokens} tok, '
            f'has {budget.seconds_left():.0f}s/{budget.tokens} tok'
        )
    ...  # do the work, passing budget.split(...) to any tool or sub-subagent

Three details matter. First, pass an absolute deadline, not a timeout duration, so time spent waiting in queues counts against the budget. Second, have each parent keep a reserve for verification and merging. Third, children inherit priority and can't change it.

Design rule 3: Degrade the reasoning before you drop the task

Agents have an option most services don't: they can think less. Reflection, self-critique and deep planning can eat a good part of an agent's token spend. They improve quality at the margin, and they're the first thing to cut under pressure. Go down this ladder one rung at a time, in order:

• ▹Rung 1: Cut reflection loops and self-critique. One pass, no 'review your own answer' step. Cost: the output is a bit rougher. Saves: often a large share of tokens per step.
• ▹Rung 2: Reduce planning depth. Use a short plan instead of a multi-level breakdown, and do fewer exploratory reads. Cost: a less optimal approach. Saves: fewer sub-calls.
• ▹Rung 3: Use a cheaper, faster model for non-critical steps like summarising logs, formatting and classification. Keep the strong model for code changes and decisions. Cost: lower quality on side tasks. Saves: latency and spend.
• ▹Rung 4: Defer low-priority tasks to a 'not now' queue. They aren't failed. They're parked with their context. Cost: those tasks finish later, or never. Saves: whole trajectories.
• ▹Rung 5: Reject new tasks at admission unless they're top tier. Cost: people see the refusals. Saves: the system itself.

Leave tool calls that verify work alone until you hit the bottom of the ladder. That means running tests, type-checking and diffing against the spec. Verification is what makes agent output trustworthy. An agent that skips it to save tokens turns out fast, confident, unchecked work, and that's worse than no work. Cut the thinking about the work before you cut the checking of it.

If a task can't finish with verification intact, drop the whole task. Don't ship half of it. A half-finished PR costs a human reviewer more than a task that never started.

Design rule 4: Retry at one layer only

Nested retries multiply, so stop nesting them. Pick one layer to own retries. That's usually the orchestrator, because it can see priority and budget. Every layer below it fails fast with a clear, typed error: BudgetExceeded, ToolFlaky, ModelOverloaded. This includes your LLM SDK. Set its max_retries to 0 below the orchestrator, or to a value you chose on purpose and wrote down.

Then replace per-call retry counts with a retry budget. Across the fleet, retries can't go above a fixed fraction of first attempts, say 10%. When the system is healthy, you never hit the limit. When something breaks, retries stay capped as a share of real work instead of growing with it. The Google SRE book describes this pattern at the client level, and Envoy and Finagle ship retry budgets as a built-in feature.

class RetryBudget:
    """Fleet-wide: retries <= ratio * first attempts, over a sliding window."""
    def __init__(self, ratio: float = 0.10, min_retries: int = 5):
        self.ratio, self.min_retries = ratio, min_retries
        self.first_attempts = 0
        self.retries = 0      # reset/decay per window in a real implementation

    def record_attempt(self, is_retry: bool):
        if is_retry:
            self.retries += 1
        else:
            self.first_attempts += 1

    def can_retry(self, budget: Budget) -> bool:
        allowed = max(self.min_retries, self.ratio * self.first_attempts)
        return budget.may_retry and budget.seconds_left() > 0 and self.retries < allowed

With this in place, the Tuesday incident costs at most about 1.1 attempts per task on average across the fleet, not 27. The flaky test still fails. It just fails cheaply, and you can see it.

What we changed at PhoenixDX

After that Tuesday we made four changes:

• ▹Every agent task gets a priority tier. Tier 0 is release-blocking. Tier 1 is feature work for the current sprint. Tier 2 covers refactors and test improvements. Tier 3 is docs and cleanup.
• ▹Every task gets a deadline budget: an absolute deadline plus a token cap, passed down with the split() pattern above.
• ▹Retries live only at the orchestrator. SDK and tool retries are off below it, and a fleet-wide retry budget sits on top.
• ▹Reflection switches off under pressure. Once token burn rate crosses a threshold, self-critique loops turn off for tiers 1 to 3 first.

The next time a test went flaky, things went differently. Retries stopped quickly because the retry budget ran out. The orchestrator flagged the test as the common failure point. Tier 0 and tier 1 tasks kept their budget and merged. Some tier 2 and tier 3 tasks went to the 'not now' queue.

Here's the honest cost: some of those low-priority tasks never ran. They aged out of the queue and nobody noticed. That doesn't mean the design failed. It means the tiers were right. If nobody misses a task when it's dropped, it really was low priority. Sometimes shedding shows you work you shouldn't have been doing in the first place.

The part the code can't decide

Everything above is mechanism. The policy, meaning which tasks count as tier 0, is a business decision. Say an engineer sets the tiers at 2am in a YAML file in the middle of an incident. That's an organisational decision nobody actually made. In the next incident the system will shed a customer-facing task because someone once labelled it 'cleanup'.

• ▹Give the priority tiers a named owner. That's usually product and engineering leadership together, not whoever happens to be on call.
• ▹Write the shedding policy up as an Architecture Decision Record. It should cover the tiers, the ladder order, the triggers and who can override them.
• ▹Review it like any other architecture decision: when priorities change, after every incident where shedding kicked in, and at least once a quarter.
• ▹Make the default tier for unlabelled tasks a low one. Unlabelled work should be the first thing shed, not the last.

Architect's checklist for your next design review

• ▹Where do retries happen, and across how many layers? Multiply them out loud. If it's more than one layer, ask which layer owns retries.
• ▹What does each task's deadline budget look like, and does it get passed down? Look for an absolute deadline, a token cap and children that fail fast.
• ▹What's the shedding ladder, in order? Write the rungs down and confirm verification is the last thing to go.
• ▹Who owns the priority tiers? Name a person or a group, and point to the ADR.
• ▹What signal triggers shedding? Options include queue depth, token burn rate against budget, upstream 429 rate, and p95 task latency against deadline. Pick signals that move before the budget runs out.

Bridge to Day 23

Shedding protects the system while it's overloaded. When the pressure lifts, everything you held back comes back at the same moment: the 'not now' queue, the deferred tasks, every client that backed off. Day 23 is about recovery. We'll cover refilling capacity gradually, draining deferred work without setting off a new thundering herd, and deciding when it's safe to climb back up the ladder.

Extend your knowledge

• ▹Read 'Using load shedding to avoid overload' by David Yanacek in the Amazon Builders' Library. It covers priority-based shedding and why you should drop work early.
• ▹Read 'Handling Overload' and 'Addressing Cascading Failures' in Google's Site Reliability Engineering book. Spend the most time on client-side throttling and retry budgets.
• ▹Read the gRPC docs on deadlines and deadline propagation, then map what you read onto the Budget object in this lesson.
• ▹Exercise: draw the retry tree for your own agent setup and count every layer, SDK defaults included. Then write a one-page shedding ADR that covers tiers, ladder, triggers and owner.