Posts tagged LLM Agents — AI × software engineering, made practical.
This lesson maps the eight fallacies of distributed computing onto multi-agent LLM systems, adds a ninth fallacy about the receiver misreading a message, and gives concrete fixes and a pre-ship checklist for both.
You'll learn how to design a run record that captures every non-deterministic input to an agent run so you can replay failures exactly, and what to look for when you build one or buy one.
You'll learn how feature-store ideas (entities, event_time vs ingested_at, point-in-time reads, one write path and TTLs) let you find and fix leakage and skew in LLM agent memory.
You'll learn why prompt injection can't be fixed inside the model, and how to limit the damage by separating agents that read untrusted content from the code that holds the permissions to act.
You'll learn why retrying a side-effecting tool call in an LLM agent loop can cause a new action instead of a repeat, and how stable idempotency keys, cancel-and-confirm on timeout, a shared budget and failure classification keep each action to one run.
How to use a durable intent log, idempotency keys built from the run and step, and reconciliation against real systems so you can unwind an agent run that fails halfway through without doubling its side effects.
You'll learn how to split an agent run's token allowance by decision type (explore, plan, edit, verify), give each type its own stop rule, and pick the shares from your own trace data.
You will learn what poison messages are, how dead-letter queues work in SQS, RabbitMQ and Kafka, how to alert on and own a DLQ, how to redrive safely, and what changes when the consumer is an LLM agent.
You'll learn why payload-based idempotency keys fail when an LLM agent retries, and how to fix it with plan-step IDs, intent keys, convergent tools and guarded one-shot actions.
You'll learn why idempotency keys built from a hash of the payload fail when an LLM agent retries, and how to key side-effecting tool calls on intent (task_id + step_id), with the key minted by the orchestrator.
You'll learn a quick test for telling whether a ReAct agent's Thought steps are genuine reasoning or just narration over an action space that only ever had one legal move.
You'll learn how prompt caching keys on an exact prefix, why a one-line edit to a shared tool schema silently invalidates it across every agent that imports it, and what to monitor so it doesn't cost you a week of spend before anyone notices.
You'll learn how images get compressed into a fixed token budget before an LLM ever reasons over them, and how to tell an encoder bug from a reasoning bug.
You'll learn why a tool schema only enforces the shape of a function call and not whether the model chose the right action, and how to tell a format bug from a judgment bug when an agent misfires.
This lesson explains why editing an LLM tool schema's wording is a breaking API change that tests, type checks, and code review won't catch, and how to version, review, and snapshot-test schemas the way you would any service contract.
Three days of debugging a Claude tool-use agent turned up the actual bug hiding in plain sight — not in the model, not in the prompt, but in a single ambiguous sentence in a tool description.
A PhoenixDX agent passed every trace check with flying colors and still shipped the wrong PR — here's the layer of observability that would have caught it at turn 18 instead of turn 40.
A multi-agent PR review pipeline gave a bug five clean approvals before it took auth down at 3am, and the failure wasn't a bad prompt — it was a gap no single agent was ever shown.
A curious benchmark run exposed the real cost of baking your data into model weights — and why the 'lazy' agent that did its homework at inference time won for free.
The bug wasn't a hallucination and it wasn't sloppy prompting — it was a structural gap between what the agent believed about the saga's state and what was actually true, and no amount of better prompting closes that gap.
A cached number ninety seconds old looks byte-for-byte identical to a fresh one — until an agent stakes an irreversible decision on it, and nothing in your stack was ever built to notice the difference.
Every failure you're currently blaming on prompt engineering — the bottleneck agent, the re-plan that breaks on scale, the handoff where two agents end up disagreeing — already has a name and a fix, and distributed database engineers wrote it up over a decade ago.
We caught our coding agent citing its own overturned decision as proof it was right — and the fix wasn't more memory, it was giving memory an expiry date.
Two rows, one identical idempotency key, both attempts legitimate — the 3am postmortem that made us realize our retry wrapper wasn't resending a request, it was re-asking the model what it wanted to do.
Your agent isn't running out of room; it's rotting from the inside, and the transcript will lie to your face about it.
Why the reasoning text your agent streams out isn't a log of what it decided — and how trusting it almost let a force-push slip past a test gate.
Quorum voting only earns its fault-tolerance guarantee when failures are independent — and most "multi-agent" code review setups quietly skip that check before a single vote gets cast.
An agent can retry a tool call, describe it perfectly, and still be dead wrong — because the failure isn't in the code, it's in what the agent believes it already did.
It read like the model going flaky, but the real bug was a type error your JSON schema was never positioned to catch — here's the tell that separates the two, and the fix that isn't 'write a better prompt.'
That double-booking bug you filed under "LLM hallucination" was actually your orchestrator making an unwritten CAP tradeoff — here's how to catch it before the next incident does.
Four agents, two weeks of handoff tuning, and 3.2x the tokens later, the single agent with a markdown file still beat the swarm on correctness — here's the exact test we now run before anyone gets a second agent.
Your schema validator has one job — checking shape, not truth — and constrained decoding quietly turns that blind spot into a hallucination wearing a green checkmark.
A prompt can run perfectly every single time and still be silently wrong for a week — here's the one question that catches it before your users do.
Your dashboards say every node is healthy, but one of them is quietly swallowing every expensive tool-use chain — because round-robin and least-connections were never built to know what an LLM request actually costs.
A whiteboard test — and a real production refund bug — for telling an actual agent apart from a chatbot wearing a trenchcoat.