Bulkhead pattern

Day 24: Bulkheads, and why your context window needs them

Your agent didn't get dumber halfway through the task. One tool call flooded its context window, and nothing told you. Every system has shared resources, and when one noisy consumer drains a shared pool, the healthy parts go down with the broken one. In multi-agent systems the pool that usually runs dry first is the context window. When it does, no error is raised.

The hook: the orchestrator that 'got dumb'

You've probably built this setup already, or you will soon. An orchestrator agent plans a feature change and hands out work to tools and sub-agents: edit the code, run the tests, update the docs. The test run fails, and the tool pipes its raw output straight into the orchestrator's context. That output is a full CI log with the same stack trace repeated across every parameterised test. It comes to about 40k tokens, and almost all of it is noise.

[orchestrator context]
system prompt + plan ............... ~6k tokens
task history ....................... ~10k tokens
run_tests() output ................. ~40k tokens   <-- one tool call
  FAILED tests/test_billing.py::test_proration[case-001]
  Traceback (most recent call last):
    ... 180 lines ...
  FAILED tests/test_billing.py::test_proration[case-002]
  Traceback (most recent call last):
    ... same 180 lines ...
  (x 200 more)

Three bad decisions follow, one after another:

• ▹It 'fixes' a line from the middle of a stack frame because that's the line it happened to attend to. That line isn't the root cause.
• ▹It loses track of a constraint from the original plan ('do not change the public API'), which is now buried under the log, and renames a method.
• ▹The docs sub-task gets marked as done, because the earlier status messages have been pushed far back in its context.

So what actually failed? The model didn't crash, and the tools did exactly what they were told. One consumer flooded a shared pool, the orchestrator's working attention, and every other task that relied on that pool sank with it. This is a capacity isolation failure, and the pattern that fixes it already has a name.

The ship picture in 60 seconds

A ship's hull is split into watertight compartments by walls called bulkheads. If the hull is breached, water floods that one compartment, the bulkheads keep it out of the others, and the ship stays afloat. Without bulkheads, one hole fills the entire hull.

The whole pattern in one sentence: partition shared capacity so failures stay where they start.

The classic version: thread pools

Take a service that calls two downstream dependencies, Payment and Search, through one shared pool of 200 request threads. Payment slows down, so each Payment call now holds a thread for seconds instead of milliseconds. New Payment requests keep arriving and each one takes another thread. Soon all 200 threads are waiting on Payment. Search requests have a perfectly healthy dependency but no thread to run on. One slow dependency has taken out the whole service.

The bulkhead fix is to give each dependency its own pool. Payment can only ever use its own threads. Once they run out, Payment calls fail fast or degrade, and Search keeps serving from its separate pool without noticing.

This is the mechanism behind Hystrix thread pool isolation and resilience4j's ThreadPoolBulkhead and SemaphoreBulkhead. It connects to Day 23: a circuit breaker stops you calling a failing dependency, and a bulkhead limits the damage while you're still calling it. You want both. A breaker takes time to trip, and the bulkhead protects you until it does.

Three design decisions every bulkhead forces

• ▹What is the shared resource? Threads, DB connections, memory, rate-limit quota, GPU slots, LLM tokens-per-minute, the context window. Be concrete. You can't partition a pool you can't name.
• ▹How will you partition it? Per dependency (payment vs search), per tenant (so one big customer can't starve the rest), or per priority (interactive traffic vs batch jobs).
• ▹When a compartment fills up, what happens? You can reject (fail fast with a clear error), queue (bounded and with a timeout, never unbounded), or degrade (serve a cached or partial answer). Make this choice up front. If you skip it, the default behaviour decides for you, and the default is usually 'hang'.

Here's a worked sizing example using Little's Law: concurrent slots needed = arrival rate × time each request holds the slot.

Search:   300 req/s x 0.1 s  = 30 threads at peak  -> pool of 45 (50% headroom)
Payment:   40 req/s x 0.5 s  = 20 threads at peak  -> pool of 30

Now Payment degrades and calls hang until the 10 s timeout:
Payment:   40 req/s x 10 s   = 400 threads needed

Shared pool of 200: Payment takes all 200, Search gets 0. Total outage.
Bulkheaded:         Payment caps at 30, rejects the rest fast
                    ("payment pending, we'll email you").
                    Search still has its 45. Partial outage, contained.

Each compartment is sized for normal peak plus headroom, not for the failure case. The bulkhead isn't there to absorb a failure. Its job is to make the failure cheap.

The new shared pool: the context window

In an agentic system, threads aren't the most fragile shared pool. The orchestrator's context window is: the finite set of tokens the model reasons over on every step. The mapping is almost one to one:

• ▹Threads become tokens. Every tool result, message and sub-agent report takes a slice of a fixed budget.
• ▹The slow dependency becomes a verbose tool output, such as a CI log, a full HTML page, a 5,000-row SQL result, or a whole file someone 'just read to be safe'.
• ▹Thread starvation becomes reasoning starvation. The plan, the constraints and the state of the other sub-tasks are technically still in the window, but they get crowded out. Long-context degradation (often called 'context rot', for example in Chroma's research on it) means models attend less reliably to the relevant details as irrelevant tokens pile up, and this starts well before the hard limit.

The real danger is how quiet this failure is. A starved thread pool gives you timeouts, 503s and alerts. A flooded context window gives you no exception and no 4xx. The agent keeps working with full confidence and makes worse decisions. You find out in code review if you're lucky, or in production if you're not.

How to bulkhead context, in four steps

• ▹1. Give each sub-agent its own context and token budget. The test runner, the code editor and the docs writer each work in a separate window, so a flood in one can't reach the others. Claude Code subagents work this way, and Anthropic's write-up on its multi-agent research system describes the same idea: sub-agents explore in their own context windows and return condensed results.
• ▹2. Hard-cap any tool output before it enters a context. Truncate it (head + tail + error lines), or write the full output to a file and pass a reference instead. If the agent needs more, it can grep the file. It should never be handed 40k tokens by default.
• ▹3. Only summaries cross the boundary. Sub-agents report to the orchestrator in a fixed schema, for example status, key finding and artifact path. That schema is your bulkhead door: fixed size, fixed shape, and raw logs can't get through it.
• ▹4. Decide the overflow behaviour before you need it. When a sub-agent's budget runs out, it can summarise (compact its own history and keep going), drop (abandon the sub-task and report it as failed), or escalate (stop and ask a human). These are the agentic versions of reject, queue and degrade.

from dataclasses import dataclass
from pathlib import Path

TOOL_OUTPUT_CAP = 4_000   # tokens allowed into any context per tool call
SUMMARY_CAP = 300         # tokens allowed across the sub-agent -> orchestrator boundary

@dataclass
class SubAgentReport:
    status: str            # "ok" | "failed" | "blocked" | "over_budget"
    key_finding: str       # one or two sentences, no raw output
    artifact_path: str     # where the full detail lives

def cap_tool_output(raw: str, run_id: str, count_tokens) -> str:
    """Bulkhead door for tool output: never let a flood into the context."""
    if count_tokens(raw) <= TOOL_OUTPUT_CAP:
        return raw
    path = Path(f"artifacts/{run_id}.log")
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(raw)
    lines = raw.splitlines()
    errors = [l for l in lines if "Error" in l or "FAILED" in l][:20]
    return "\n".join([
        f"[output truncated: {count_tokens(raw)} tokens, full log at {path}]",
        *lines[:20],
        "... error lines ...",
        *errors,
        "... tail ...",
        *lines[-20:],
    ])

def to_orchestrator(report: SubAgentReport, count_tokens) -> str:
    """Only the summary crosses the boundary. Enforce the size, don't trust it."""
    msg = f"status={report.status}; finding={report.key_finding}; artifact={report.artifact_path}"
    if count_tokens(msg) > SUMMARY_CAP:
        raise ValueError("summary exceeds bulkhead cap: re-summarise in the sub-agent")
    return msg

Two details matter in this code. First, the caps live in code, not in the prompt. 'Please be concise' is a hope, and a hope won't hold back a flood. Second, nothing gets thrown away. The full detail moves to a file that any agent can open deliberately, so 'always in context' becomes 'available on demand'.

Before and after: the same failing build, contained

This time the test sub-agent takes the 40k-token log into its own window, reads it, groups the repeated failures, and sends back this:

status=failed; finding=3 distinct failures in test_billing.py, all proration cases;
root cause likely off-by-one in days_in_period() after the leap-year change;
artifact=artifacts/ci-1842.log

That's about 80 tokens. The orchestrator's plan, constraints and sub-task states are still at the front of its attention. It delegates a targeted fix to the code sub-agent, leaves the public API alone, and keeps the docs task open. The failure is the same as before, but now it stays in one compartment.

The costs, stated plainly

• ▹Summaries lose detail. The summarising sub-agent can drop the one line that mattered. That's why every summary should carry an artifact path, so the detail can be retrieved.
• ▹Extra hops cost latency and money. Each sub-agent is another model call, often several, each with its own system prompt. Multi-agent setups burn noticeably more tokens than a single agent, and you pay for that on every run, not just the ones that fail.
• ▹Budgets set too tight cause starvation by design. A sub-agent that can't see enough of the codebase will make the same bad decisions you were trying to prevent. Size budgets from real traces rather than guesses, as you would with thread pools.
• ▹My rule of thumb: partition when the work streams are independent, such as tests, docs, or search across separate sources. Share context when the work genuinely needs joint reasoning, like a tricky refactor where the edit and the test failure only make sense together. If you split tightly coupled reasoning across agents, you trade a context problem for a coordination problem.

Architect's checklist, agentic or not

• ▹List your shared pools: threads, connections, rate-limit quotas, LLM tokens-per-minute, GPU slots, and every context window that more than one source writes into.
• ▹For each pool, name its noisiest consumer: the slowest dependency, the biggest tenant, the chattiest tool.
• ▹Then ask: 'If this consumer floods, what else sinks?' If the answer is 'everything', that pool needs a bulkhead.
• ▹Write down what each bulkhead does when it's full: reject, bounded queue, degrade, summarise, drop, or escalate.
• ▹Make every compartment observable. Emit tokens-used-per-agent and truncation events the same way you emit thread-pool saturation. A failure this quiet only shows up if you emit a signal for it.

Bridge to Day 25

Bulkheads limit the damage, but a flooded compartment is still flooded. On Day 25 we look at how a system notices that a compartment has failed (health checks, saturation signals, a sub-agent that keeps hitting its budget) and recovers on its own, without paging a human.

Extend your knowledge

• ▹Read the stability patterns chapter in Michael Nygard's 'Release It!' (2nd edition), which covers Bulkheads. It's still the clearest treatment of the classic pattern.
• ▹Go through the resilience4j docs on SemaphoreBulkhead vs ThreadPoolBulkhead. Then configure one in a toy service and load-test it against a deliberately slow dependency.
• ▹Anthropic's engineering posts 'How we built our multi-agent research system' and 'Effective context engineering for AI agents' show how production agents isolate context and condense results.
• ▹Exercise: pick one of your own agent workflows and log the token count of every tool result for a single run. Find the largest one, add a cap with file spill-over, and compare decision quality on the same task before and after.