Posts tagged Observability — AI × software engineering, made practical.
You'll learn how to design a run record that captures every non-deterministic input to an agent run so you can replay failures exactly, and what to look for when you build one or buy one.
You'll learn how to attach provenance IDs to every unit of context in a RAG or multi-agent pipeline, check model citations, and use Postgres recursive queries to trace a wrong answer back to its source and find everything that came from a document.
You'll learn why LLM consumers stop sending bad input to your dead letter queue, and how a refusal tool, a provenance envelope and abstention monitoring bring it back.
You'll learn how to record and replay multi-agent runs step by step, covering message ordering, logical clocks, event-sourced traces, exact and counterfactual replay, and how replay results can decide how much autonomy an agent gets.
You'll learn how prompt caching keys on an exact prefix, why a one-line edit to a shared tool schema silently invalidates it across every agent that imports it, and what to monitor so it doesn't cost you a week of spend before anyone notices.
You'll learn why heartbeat-based liveness checks can't tell a stuck agent from a working one, and how to add a separate progress signal that catches the difference.
Learn why an LLM-as-judge validator needs its own tests, fail-closed defaults, and separate alerting instead of being treated as trusted plumbing.
A bloom filter can degrade from 1% wrong to 80% wrong without throwing a single error, and I only found ours because three near-identical tickets landed in the same hour at 2am.
A message queue will hide a dead consumer from you for as long as it takes — here's the six-hour timeline of exactly how, the one metric that would've paged you in minutes instead, and why the same failure is now waiting inside your LLM agent worker pool.
A PhoenixDX agent passed every trace check with flying colors and still shipped the wrong PR — here's the layer of observability that would have caught it at turn 18 instead of turn 40.
Six agents ran clean, every span passed, and the pipeline still quietly wrecked its own output — the postmortem question that exposed it is one logs, metrics, and traces were never built to answer.
A billing anomaly forced us to trace one request through a reasoning loop with twelve hidden calls inside it — and exposed the blind spot every API gateway has the moment your agents start calling each other behind its back.
Our eval score sat flat and green at 94% for three releases while support tickets tripled — the agent hadn't drifted, the test had.
A rising cache-hit ratio and a growing pile of 'site feels slow' tickets can both be true in the same quarter — here's the arithmetic behind that blind spot, and why the exact same trap is now showing up in LLM prompt caching and multi-agent pipelines.
A dead letter queue nobody reads isn't a safety net, it's a data-loss incident running on a delay — here's the replay path most teams never build, and why agentic pipelines make triage harder, not easier.
Your subagents are re-planning your tasks mid-execution, off the books, and the constraint that mattered most is the first thing they drop.
An agent pipeline can post a spotless uptime dashboard for six straight hours while quietly solving the wrong problem entirely — here's the sensor a classic circuit breaker doesn't have, and the one you actually need.
A queue promises your message will arrive eventually — it never promised you could prove what happened to it, and that gap is what pages you when a delivery goes quietly, successfully wrong.
An on-call team burned two hours blaming a 'flaky' order service, when the real story was two agents independently patching the same order through a gateway that had no way to notice.
Your CDN dashboard and your origin server are both telling the truth about the same traffic — which is exactly how a 95% hit ratio and rising origin CPU coexist, and the same trick is now hiding inside your LLM inference stack.