Posts tagged Multi Agent Systems — AI × software engineering, made practical.
You'll learn to evaluate each handoff between agents in a multi-agent pipeline by checking for dropped or softened constraints, invented claims and message sufficiency, and by injecting a fault into a single message to find where nothing catches errors.
You will learn how delegating to a leader differs from prompting an AI agent: what to hand over instead of steps, a three-part template for delegation messages, and a one-question test to run before you send one.
You'll learn how to choose an open-weight or closed model for each role in an agent system by checking how far its errors spread, how many calls it makes, and whether its output can be checked automatically.
This lesson maps the eight fallacies of distributed computing onto multi-agent LLM systems, adds a ninth fallacy about the receiver misreading a message, and gives concrete fixes and a pre-ship checklist for both.
You will learn how to measure where tokens go in an orchestrator-plus-sub-agents fleet, and how to cut re-read context with cache-friendly prompt prefixes, a shared context ledger and decision-based handoffs.
You'll learn how to attach provenance IDs to every unit of context in a RAG or multi-agent pipeline, check model citations, and use Postgres recursive queries to trace a wrong answer back to its source and find everything that came from a document.
This post shows why a lease alone cannot stop a paused coding agent from overwriting another agent's work, and how fencing tokens checked at git, the database and the file system, together with a distinct FENCED error that makes the agent re-plan, prevent it.
You'll learn why chained agent pipelines can fail with every stage reporting success, and how an owner of last resort and explicit handoff contracts fix the accountability gap.
You'll learn how a coder agent and a reviewer agent can drift into satisfying each other instead of shipping working code, why per-agent evals miss it, and which signals and design changes catch it.
You'll learn why stronger prompts and guard classifiers can't stop prompt injection, and how to contain it with taint tracking, privilege drop, quarantined readers, typed channels and provenance logging.
You will learn how to model, measure and reduce the tokens each agent spends reading, validating and summarising the same shared data, and how to price that spend next to your warehouse bill.
You will learn how to treat the human in a multi-agent pipeline as a slow, rate-limited agent and build the protocol around those limits with escalation budgets, async queues, decision packets, timeout defaults and two metrics that show whether the setup is working.
You will learn how the bulkhead pattern isolates shared capacity, starting with classic thread pools and then applying it to LLM agents through per-sub-agent context budgets, tool output caps and summary-only boundaries.
This lesson shows how to build a verifier agent that checks another agent's work independently of the builder and gives each claim one of three verdicts, verified, refuted or unverifiable, backed by evidence.
You'll learn how to record and replay multi-agent runs step by step, covering message ordering, logical clocks, event-sourced traces, exact and counterfactual replay, and how replay results can decide how much autonomy an agent gets.
You'll learn why burnout in senior engineers and new tech leads is usually a team-design problem, and how to fix it by delegating decision authority, writing context down, and making sure the team keeps running when you're away, with each fix tied to the same failures in single-orchestrator multi-agent systems.
You'll learn what an orchestrator agent does in a multi-agent system, why it should be limited to decomposing, dispatching and accepting work, and when a smarter planner is worth adding.
You'll learn how to structure a multi-agent system as a deterministic pipeline with a single bounded swarm, which can only hand off its output through an exit contract that a code checkpoint enforces.
You will learn how to decide which agent work to drop under overload, using deadline budgets, a shedding ladder, retries at a single layer, retry budgets and priority tiers with a named owner.
You'll learn how to partition parallel agent work and design the merge step (output contracts, shared-file ownership, seam reviewers and reducer budgets) so that fanning out actually saves time, and when you shouldn't fan out at all.
You will learn how prompt injection hides inside agent-to-agent handoffs and how to stop it with provenance-tagged handoffs, authority checks that only the principal can pass, and limits on free-text fields.
This post explains why exponential backoff with jitter fails when many coding agents share one API key, and how to replace it with a token-based lease coordinator, DAG-weighted queueing and checkpointed agent turns.
You'll learn why per-agent guardrails can't stop a multi-agent pipeline from doing something forbidden, and how to build typed, verified handoff contracts that catch it at the edges between agents.
You'll learn where streaming belongs in an agentic system: buffer complete outputs between agents and verifiers, define an explicit commit point, and stream progress events to users instead of unverified tokens.
You'll learn when a reflect or critique step actually improves an agent's correctness, and how to replace the ones that don't with verifiers that use real evidence and can block a change.
You'll learn how prompt caching keys on an exact prefix, why a one-line edit to a shared tool schema silently invalidates it across every agent that imports it, and what to monitor so it doesn't cost you a week of spend before anyone notices.
You'll learn why coding agents working on the same file diverge even with no network partition, and how to design an arbitration rule so merges stop silently picking a winner.
You'll learn why heartbeat-based liveness checks can't tell a stuck agent from a working one, and how to add a separate progress signal that catches the difference.
Learn what churn is, how to split it into voluntary, involuntary, logo, and revenue churn, and why AI products create new churn patterns — trust churn, silent churn, cost-driven churn, and task churn — that classic SaaS metrics miss.
A plain explanation of why teams and AI agents really reach for CQRS — not to scale reads, but because different consumers of a shared model disagree on what 'current state' means — and how to tell that apart from an actual load problem.
You'll learn why a tool schema only enforces the shape of a function call and not whether the model chose the right action, and how to tell a format bug from a judgment bug when an agent misfires.
Event-driven architecture never actually removes the dependency between your services — it just deletes the paper trail, and the same blind spot is already sitting inside your multi-agent pipelines.
We went from 4 subagents to 16 expecting more throughput — p95 latency got worse instead, and the culprit turned out to be the same failure mode database engineers have fought for decades.
Six agents ran clean, every span passed, and the pipeline still quietly wrecked its own output — the postmortem question that exposed it is one logs, metrics, and traces were never built to answer.
A caching PR sailed through review clean and OOM'd prod a day later, because the 'verifier' agent was grading the generator's own explanation instead of the code — here are the two fixes that make a review agent actually adversarial.
There's a third causal category sitting between "human error" and "agent error" that most incident templates have no slot for — and naming it is the difference between fixing the gate and re-running the same incident next sprint.
The engineer who gets promoted usually isn't the best coder in the room — they're just the node everyone's routing table happens to point to, and that's a structure you can deliberately build.
I said one sentence at turn 6, the agent obeyed it for thirty turns, then compaction quietly summarized it out of existence and the agent ran the exact command I'd banned — here's the fix that isn't "write a better CLAUDE.md."
A billing anomaly forced us to trace one request through a reasoning loop with twelve hidden calls inside it — and exposed the blind spot every API gateway has the moment your agents start calling each other behind its back.
If your code-review agents share a base model and a prompt, the majority vote you trust isn't canceling error — it's laundering it into false confidence, and the math says so.
A multi-agent PR review pipeline gave a bug five clean approvals before it took auth down at 3am, and the failure wasn't a bad prompt — it was a gap no single agent was ever shown.
A rising cache-hit ratio and a growing pile of 'site feels slow' tickets can both be true in the same quarter — here's the arithmetic behind that blind spot, and why the exact same trap is now showing up in LLM prompt caching and multi-agent pipelines.
Majority vote feels like triple-checking your agents, but when three instances share one blind spot, unanimous agreement is just the same wrong answer said three times — here's the asymmetric tiebreaker pattern that actually catches it.
If your agent fleet keeps stalling on one PR while everyone else waits, you didn't get unlucky — you sharded your work the same way databases have been getting it wrong for twenty years, and the fix is just as old.
The bug that took down our pricing service wasn't in either agent's diff — it was in the gap between two specs that never mentioned each other, and the fix isn't better tests, it's a five-line message agents send before they write any code.
A cached number ninety seconds old looks byte-for-byte identical to a fresh one — until an agent stakes an irreversible decision on it, and nothing in your stack was ever built to notice the difference.
A sub-agent's "yes, I'm prepared" can quietly evaporate the second its context gets truncated — and most orchestrators have no way to notice until two agents have already clobbered the same resource.
Consistent hashing was built to make moving a cache key cheap — but an agent session isn't a cache key, and treating it like one turns every routine scale-up into a silent, expensive case of amnesia.
No agent lied, no agent errored, and a resolved support ticket got reopened anyway — because handoff() quietly threw away the one piece of context that made 'resolved' mean anything.
A three-week 'flaky model' postmortem turned out to be a one-character field rename nobody validated — here's the fix that turns that class of bug into a single stack trace.
The 'flaky' bug where Agent B redoes work Agent A already fixed isn't a timing race at all — it's a consistency choice your context store made for you without asking, and naming it changes everything you reach for next.
Two agents answered the same support ticket with two contradicting replies, no error ever fired, and the postmortem verdict was scarier than a bug: the system worked exactly as designed — nobody had just agreed on what that design was.
Every failure you're currently blaming on prompt engineering — the bottleneck agent, the re-plan that breaks on scale, the handoff where two agents end up disagreeing — already has a name and a fix, and distributed database engineers wrote it up over a decade ago.
A PR at PhoenixDX passed review clean — no errors, no crash, no merge conflict — and it was still wrong, because two agents had quietly overwritten each other's memory of the codebase and nothing in the system was built to notice.
A Lamport clock will always tell you which event came first — even when that's a lie — and the fix for it, vector clocks, turns out to be exactly what your multi-agent pipeline needs before a merge quietly erases someone's work.
A vanishing task flag in a two-agent system isn't a merge bug — it's Brewer's CAP theorem showing up uninvited, and your orchestration framework already picked a side without asking you.
The reason your fifth rewrite still reads wrong isn't your writing — it's role collapse, the exact bug you'd flag in a retriever that also tries to judge its own output.
A comment vanishes, support says 'data loss,' the logs say otherwise — and the same tiny lag that explains it is quietly wrecking your multi-agent pipelines too.
A timeout, a retry, and zero broken code is all it takes for a multi-agent pipeline to double-charge a customer — here's the decade-old fix almost nobody carried over into agent orchestration.
Your subagents are re-planning your tasks mid-execution, off the books, and the constraint that mattered most is the first thing they drop.
Months after handing off a domain, an ex-CTO got tagged into a Slack thread as the person who had to say go — and the diagram had no box for why.
A `sleep(random())` hack cut down on duplicate dispatch bugs but nobody could say why — turns out Raft solved this exact standoff a decade ago, and it's missing two pieces the hack never had.
One question — 'can you point to the seam?' — decides whether adding a person, a server, or an AI agent makes you faster or just adds coordination tax.
A verifier that only outputs PASS or FAIL isn't automating review — it's quietly training your engineers to stop looking, and here's the schema fix we run at PhoenixDX instead.
Run more than one coding agent against shared infrastructure and you're not orchestrating a pipeline anymore — you're refereeing a game, and game theory has some uncomfortable things to say about where it settles.
Borrowing Raft's leader election into a multi-agent LLM pipeline makes the design doc look rigorous — right up until you notice it can't stop the one thing that actually goes wrong: two confident agents disagreeing.
An agent pipeline can post a spotless uptime dashboard for six straight hours while quietly solving the wrong problem entirely — here's the sensor a classic circuit breaker doesn't have, and the one you actually need.
Why the reasoning text your agent streams out isn't a log of what it decided — and how trusting it almost let a force-push slip past a test gate.
Quorum voting only earns its fault-tolerance guarantee when failures are independent — and most "multi-agent" code review setups quietly skip that check before a single vote gets cast.
The cache you turned on is working exactly as designed — it's just guarding the one line item on your invoice that was never the problem.
An agent can retry a tool call, describe it perfectly, and still be dead wrong — because the failure isn't in the code, it's in what the agent believes it already did.
The obvious fix for a worker agent that keeps breaking its own output format is another agent to catch it — here's why that instinct is usually wrong, and what actually killed the drift for a fraction of the cost.
A promotion backed by every number on the page quietly slowed my team down — and the blind spot behind it turns out to be the same one that makes most multi-agent observability stacks credit the wrong agent.
A planner prompt change that looked like a two-line efficiency tweak quietly broke every downstream executor — and revealed why plans need the same versioning discipline as any API.
An agent wrote a flawless commit message for a fix that broke production six weeks later, and when I asked why the fix existed, nobody — not the engineer, not me, not the agent — could say, because the pipeline that writes these messages was never built to hold an answer.
That double-booking bug you filed under "LLM hallucination" was actually your orchestrator making an unwritten CAP tradeoff — here's how to catch it before the next incident does.
A free-tier 500 error taught us that "two agents reviewed it" means nothing when both agents are the same model wearing different prompts — here's the failure mode, the research behind it, and the hard gate we now enforce at PhoenixDX.
One 429 turned into a fan-out storm in under ten minutes — here's the assumption baked into every rate limiter that agentic callers blow straight through, and the three fixes that actually hold.
A billing agent and a policy agent both answered correctly, and the orchestrator still shipped a wrong response to the customer — because nobody had built the one piece that decides what happens when confident agents contradict each other.
A network hiccup forces every distributed system — including the fleet of LLM calls hiding behind a single "agent" — into a choice between a fast wrong answer and a slow right one, and most teams never notice they made it.
An on-call team burned two hours blaming a 'flaky' order service, when the real story was two agents independently patching the same order through a gateway that had no way to notice.
Four agents, two weeks of handoff tuning, and 3.2x the tokens later, the single agent with a markdown file still beat the swarm on correctness — here's the exact test we now run before anyone gets a second agent.
The LLM call you're treating as "just reasoning" is doing the exact job a dbt model does, except nobody's diffing it, versioning it, or noticing when ten agents quietly disagree on what "churn" means.
Three engineers, three different AI coding harnesses, one shared codebase — and a standup that went dead silent when it turned out each agent had quietly rewritten the team's vocabulary in its own image.
If your first multi-agent run came back with confident garbage, the fix isn't a better model — it's the same delegation skill you already learned the hard way with your first junior hire.
A team spent two weeks bulletproofing a prompt that told an agent never to issue a refund — and the agent did it anyway, because the rule was sitting in the one place that was always up for negotiation.
The two prompts looked almost identical; one let a bug hide in production for two days, the other made the agent stop and confess exactly where it was stuck.
A three-day PR review at PhoenixDX exposed that the real blocker to delegating work to agents isn't task size or risk — it's whether anyone wrote down what 'correct' means before the work started.
A whiteboard test — and a real production refund bug — for telling an actual agent apart from a chatbot wearing a trenchcoat.
Running three parallel agents on the same ticket and trusting the 2-of-3 majority almost cost us a customer relationship — because agreement between correlated agents isn't confirmation, it's the same blind spot showing up twice.
A real Tuesday — including an eleven-minute incident where an agent executed a flawed plan perfectly across 600 tenants — shows the senior engineer's job shifting from writing code to designing the harness that decides whether an agent's conclusions are safe to act on.