Agentic Software Engineering — AI × software engineering, made practical.
You'll learn what an AI agent is as a loop of observe, decide, act and check, why mistakes compound inside that loop, what a useful hand-back to a human looks like, and why you should judge agents by the cost of the errors they miss rather than the number of tasks they finish.
You'll learn a three-line prompt that rewrites an AI draft in plain words, and how to sort what shrank, what vanished and what grew so you can tell padding and unchecked claims from real content before you send.
You will learn the four levels of the C4 model and how to write them as a plain-text ARCHITECTURE.md that people and coding agents both read before changing code.
You'll learn to measure your handoff radius, the largest agent-written task you can explain cold within 10 minutes, and to track it each quarter for each area of the codebase so you can tell whether your judgment is growing or going stale.
You'll learn to describe each tech debt item by its cost now, the feature that triggers a cost jump, its cost after, and its expiry date, so planning can treat it as a dated decision with a named owner.
You'll learn to evaluate each handoff between agents in a multi-agent pipeline by checking for dropped or softened constraints, invented claims and message sufficiency, and by injecting a fault into a single message to find where nothing catches errors.
You'll learn how prompt caching makes time-to-first-token depend on the order of your prompt, how to find the lines that break the cache in your agent harness, and how to order the prompt so the cache keeps hitting on every turn.
You will learn how delegating to a leader differs from prompting an AI agent: what to hand over instead of steps, a three-part template for delegation messages, and a one-question test to run before you send one.
You'll learn how system design rubrics score candidates, and how to present a simple-first design with measurable scale triggers so the rubric sees your judgement.
You'll learn how to choose an open-weight or closed model for each role in an agent system by checking how far its errors spread, how many calls it makes, and whether its output can be checked automatically.
How to treat a multi-step coding agent run as a distributed transaction: pair each tool with a compensation, log every intent before the call runs, hold irreversible calls in an outbox until the run succeeds, and send corrections to the people who saw a change that got reverted.
You'll learn a four-part prompt that replaces 'Any concerns?' with 'What would make this wrong?', so the AI names the one assumption your plan depends on and tells you how to check it before you build.
This lesson maps the eight fallacies of distributed computing onto multi-agent LLM systems, adds a ninth fallacy about the receiver misreading a message, and gives concrete fixes and a pre-ship checklist for both.
You'll learn how to design a run record that captures every non-deterministic input to an agent run so you can replay failures exactly, and what to look for when you build one or buy one.
You'll learn how old comments, fixtures, docs and error strings in a legacy repo can act as prompt injection for coding agents, how to find and triage them, and how to limit the damage when one slips through.
You'll learn to cut scope by what your team can understand and support instead of what it costs to build, using written non-goals in agent briefs and a three-question ownership test in review.
You will learn how to measure where tokens go in an orchestrator-plus-sub-agents fleet, and how to cut re-read context with cache-friendly prompt prefixes, a shared context ledger and decision-based handoffs.
You'll learn how to log each model decision as a structured record (options, choice, stated intent, context refs and parent link) so you can trace a wrong output in a live LLM app or agent back to its root cause.
You'll learn how to write reversal criteria (a metric with a threshold, a decision date and a named evidence owner) so skeptical engineers can test a change like an AI coding agent rollout instead of resisting it.
You'll learn how to attach provenance IDs to every unit of context in a RAG or multi-agent pipeline, check model citations, and use Postgres recursive queries to trace a wrong answer back to its source and find everything that came from a document.
You'll learn how to run and prepare for an interview round where candidates review an agent-generated pull request with planted bugs, and what a strong review looks like.
You'll learn what a leaderboard score like SWE-bench actually measures, why it can drift from real-world performance, and how to build a private benchmark from your team's reverted PRs.
This post shows why a lease alone cannot stop a paused coding agent from overwriting another agent's work, and how fencing tokens checked at git, the database and the file system, together with a distinct FENCED error that makes the agent re-plan, prevent it.
You'll learn why constraints the agent can check against its own diff stop it from leaving collateral damage, and how to turn vague rules into ones that can be checked.
You'll learn why wall-clock time differs between machines, where that difference breaks everyday code such as JWT checks, leases, TTLs and last-write-wins, and how leeway, monotonic clocks, Hybrid Logical Clocks and TrueTime handle it, plus how the same problem shows up in AI coding agents.
You will learn to sort a change into reversibility classes R0 to R3 and then pick a rollout pattern that still protects you when rolling back cannot undo the damage, including for AI agents.
You'll learn why chained agent pipelines can fail with every stage reporting success, and how an owner of last resort and explicit handoff contracts fix the accountability gap.
You'll learn how to write acceptance criteria a coding agent can follow without asking questions, using the no-questions test, must-NOT criteria, and a split between machine-checkable and judgment criteria.
You'll learn how a coder agent and a reviewer agent can drift into satisfying each other instead of shipping working code, why per-agent evals miss it, and which signals and design changes catch it.
You'll learn why stronger prompts and guard classifiers can't stop prompt injection, and how to contain it with taint tracking, privilege drop, quarantined readers, typed channels and provenance logging.
You'll learn why a reversible change can become expensive to undo once coding agents copy it across many repos, and how a repetition threshold, a precedent flag and a weekly pattern diff keep that in check.
You will learn how to model, measure and reduce the tokens each agent spends reading, validating and summarising the same shared data, and how to price that spend next to your warehouse bill.
You will learn to ask what breaks if a role stays empty, sort the answer into a fire, gap or bet, and re-aim your interview stories and follow-up note to match it.
You will learn what an eval actually is, why large generated test suites miss the failures real users hit, and how to turn each incident into a small regression case with the cheapest check that catches it.
How we replaced the take-home test with a live Claude Code pairing session built around a believable agent mistake, and the five-line rubric we now use to score what the candidate does before the first prompt and after the agent says it's done.
You'll learn how to decide whether to bundle or split an AI coding prompt by counting how each part will be checked, and how to order a bundle safely when you can't split it.
You'll learn how the database write-ahead log rule lets a coding agent resume exactly where it crashed, using intent and commit records, fsync, idempotent tools, checkpoints, and safe handling of external side effects.
You will learn why blue-green rollback stops working once shared state changes shape, how to measure your real rollback window, and how to use expand/contract migrations, N-1 compatibility checks and tolerant readers so a cutover is safe to roll back.
You'll learn how to get your coding agent to show the options it rejected and why, and how to use those rejected options to build your own engineering judgment.
You'll learn to rewrite a user story so its "so that" becomes the testable outcome and its "I want" becomes a suggestion, so a coding agent builds what the feature is actually for.
You'll learn the difference between guardrails written into an agent's prompt and guardrails the environment enforces, and how to decide which of your rules need which.
You will learn how to treat the human in a multi-agent pipeline as a slow, rate-limited agent and build the protocol around those limits with escalation budgets, async queues, decision packets, timeout defaults and two metrics that show whether the setup is working.
You'll learn how to use a 50-question labelled eval to sort LLM failures into retrieval, generation and behaviour misses, and how to choose between prompt fixes, RAG fixes and fine-tuning based on what you find.
How to replace a status standup with a short, tag-driven meeting that only handles the trade-offs, ambiguous specs and blockers your coding agents can't decide on their own.
You'll learn why LLM consumers stop sending bad input to your dead letter queue, and how a refusal tool, a provenance envelope and abstention monitoring bring it back.
You will learn how to agree on what 'done' means before work starts, using BATNA and a five-line Done Contract, and how to hold to it with teammates, PMs and coding agents.
You'll learn how a prompt differs from context, why extra context can make an agent worse, and how to use three questions to decide what stays in and what comes out.
You'll learn how to measure prompt cache hits in a coding agent harness, and how to fix the things that quietly break caching: volatile values in the system prompt, shifting tool lists, shared CLAUDE.md edits, aggressive compaction and badly placed breakpoints.
You'll learn to add one "done =" line to your prompts that sets a checkable finish line and fences off the shortcuts you'd reject, so you skip the correction loop.
You'll learn how read-replica lag breaks read-your-writes for AI agents, how to measure it, and how to fix it with session pinning, causal tokens, staleness budgets and idempotent agent tools.
You will learn how the bulkhead pattern isolates shared capacity, starting with classic thread pools and then applying it to LLM agents through per-sub-agent context budgets, tool output caps and summary-only boundaries.
You'll learn to spot five common ways a coding agent run goes wrong while it is still running, check for each one in about 10 seconds, and decide whether to steer the run, kill it, or let it finish.
You'll learn why your team's success rate with its own AI agent overstates how well the agent works, and how to run cold dogfooding sessions that measure the product instead of the people who built it.
This lesson shows how to build a verifier agent that checks another agent's work independently of the builder and gives each claim one of three verdicts, verified, refuted or unverifiable, backed by evidence.
You'll learn how to record and replay multi-agent runs step by step, covering message ordering, logical clocks, event-sourced traces, exact and counterfactual replay, and how replay results can decide how much autonomy an agent gets.
You'll learn how to onboard, test and limit a fallback model so it keeps your agent's behaviour safe, and not just its uptime, when the primary model fails.
You'll learn why headcount no longer tracks a manager's load on teams that run coding agents, how to measure open decisions over five days, and three ways to reduce them without hiring.
You'll learn how feature-store ideas (entities, event_time vs ingested_at, point-in-time reads, one write path and TTLs) let you find and fix leakage and skew in LLM agent memory.
You'll learn why burnout in senior engineers and new tech leads is usually a team-design problem, and how to fix it by delegating decision authority, writing context down, and making sure the team keeps running when you're away, with each fix tied to the same failures in single-orchestrator multi-agent systems.
You'll learn why prompt injection can't be fixed inside the model, and how to limit the damage by separating agents that read untrusted content from the code that holds the permissions to act.
This post covers five guardrails you can merge this sprint so the database migrations your coding agent writes don't break during rolling deploys: an N-1 compatibility job, lint output written for the agent, a phase state file, a production read gate, and a one-line CLAUDE.md rule plus a reviewer checklist.
You'll learn why describing a problem to an AI gets you generic answers, and how pasting the raw evidence (files, full errors, versions, constraints) gets you answers about your actual problem.
You will learn what a shard key actually commits you to (locality, load distribution, blast radius and rebalancing cost), how range, hash and directory sharding differ, and how to build in an escape hatch before you need one.
You'll learn how to find the slowest consumer in an agentic pipeline (usually the human reviewer), use Little's Law to estimate queue wait times, and design backpressure with a bound, an upstream signal and a defined producer response.
You'll learn how to treat a feature flag rollout as a product decision when coding agents write much of your code, using a rollout card that sets the segment, exposure steps, metrics, rollback condition, owner and kill date.
You'll learn what an orchestrator agent does in a multi-agent system, why it should be limited to decomposing, dispatching and accepting work, and when a smarter planner is worth adding.
You'll learn how to structure a multi-agent system as a deterministic pipeline with a single bounded swarm, which can only hand off its output through an exit contract that a code checkpoint enforces.
You'll learn why retrying a side-effecting tool call in an LLM agent loop can cause a new action instead of a repeat, and how stable idempotency keys, cancel-and-confirm on timeout, a shared budget and failure classification keep each action to one run.
You will learn why architecture keeps the shape of a team after the team changes, how departures and AI-driven team shrinks leave services without owners, and how to treat every org change as an architecture decision.
You'll learn to write data contracts in three layers (shape, meaning, and ownership), where the meaning layer is enforced with reconciliation, distribution, and golden queries, and how the same approach protects RAG fields and pipelines that agents edit.
You'll learn why coding agents copy whichever pattern shows up most often in your repo, how that tilt grows with every PR an agent opens, and four ways to fix it: cleanup, golden-path files, lint rules and ADRs written for agents to read.
How to use a durable intent log, idempotency keys built from the run and step, and reconciliation against real systems so you can unwind an agent run that fails halfway through without doubling its side effects.
You'll learn how AI meeting recaps make open questions look settled, how to spot that on the page, and how to check a recap in five minutes before you act on it.
You'll learn why pub/sub hides event consumers from coding agents, which lets agents ship green changes that break downstream services, and how a subscriber registry, consumer-driven contract tests and a SUBSCRIBERS.md agent rule make those dependencies visible.
You will learn how to decide which agent work to drop under overload, using deadline budgets, a shedding ladder, retries at a single layer, retry budgets and priority tiers with a named owner.
You'll learn why review habits built on human code fail on agent diffs, what research on self-checking, debate and LLM judges says about verifying model output, and a practical checklist and set of team practices for adversarial review.
You'll learn how to write a one-line value proposition that sets an agent's defaults, scope and tests, and how to check it with a ten-minute agent read test.
You'll learn how RAG retrieves, augments and generates answers, why tuning retrieval won't fix outdated documents, and how to keep a knowledge base trustworthy by managing it like code.
You'll learn how to partition parallel agent work and design the merge step (output contracts, shared-file ownership, seam reviewers and reducer budgets) so that fanning out actually saves time, and when you shouldn't fan out at all.
You'll learn how to measure agent cost per accepted change instead of per token, audit your traces for the retry loops that inflate spend, and cut those loops with richer first-attempt context, a planner/executor model split and explicit stop conditions.
You'll learn why coding agents can drain motivation from teams whose velocity is rising, and how to move credit to outcome ownership through retro prompts, demo credit and a catch log.
You'll learn why LLM-generated golden sets inflate retrieval metrics, how to test your own set for that bias, and how to build an eval set from real failed searches labelled by the people who found the answer.
You'll learn how to turn down requests based on what a feature costs to own rather than what it costs to build, using three questions, a few scripts that keep the relationship intact, a written decision record, and a critic step you run before your agents write any code.
You'll learn how RLHF turns human preferences into model behaviour, why that makes assistants and coding agents prone to flattery and reward hacking, and how verifiable checks like tests help you work around it.
You'll learn how known flaky tests teach coding agents to rerun real regressions until they pass, and how flake fingerprints, a rerun ledger and test-writing bans in CLAUDE.md stop that from happening.
You'll learn why AI assistants make up function names, kwargs and CLI flags, where those fakes usually show up, and how to check in one line that a symbol exists before you paste it.
You will learn how backpressure works in pipelines, why human code review silently degrades when coding agents outpace reviewers, and how to build a review-capacity window that makes agents adapt upstream.
You will learn how service discovery works, how client-side and server-side discovery differ, when a registry is worth it compared with static config, and how the same choice applies to MCP servers and agent tool registries.
You'll learn how to use Little's Law to measure WIP, cycle time and throughput in an agent-assisted pipeline, see why the human review gate limits how much you ship, and replace lines of code with flow metrics.
You will learn to set a price from the customer's next-best alternative rather than your build effort, using a simple floor, ceiling and value model.
You'll learn what an embedding is, why embedding search works well for document RAG, why it often picks the wrong file in a codebase, and how to check its results with grep, call graphs and git history.
You will learn how prompt injection hides inside agent-to-agent handoffs and how to stop it with provenance-tagged handoffs, authority checks that only the principal can pass, and limits on free-text fields.
You'll learn how to split an agent run's token allowance by decision type (explore, plan, edit, verify), give each type its own stop rule, and pick the shares from your own trace data.
You'll learn what review fatigue is, how to spot it in a 1:1 with one question, and three ways to deal with it: a craft budget, counting agent design as authorship, and splitting review work by temperament.
You'll learn to measure precision@k and distractor rate alongside recall@k, and to choose how much context a coding agent gets based on whether its output passes, not on whether the right file showed up.
You'll learn how to turn invisible glue work into owned, written artifacts (decision logs, repo context files and a rotating glue owner) that unblock your team, give coding agents the context they need, and count as evidence in a promotion case.
This post explains why exponential backoff with jitter fails when many coding agents share one API key, and how to replace it with a token-based lease coordinator, DAG-weighted queueing and checkpointed agent turns.
You'll learn a 60-second drill for listing the checkable claims and causal links in AI output, so you can tell what you've verified from what you're taking on trust.
You will learn what poison messages are, how dead-letter queues work in SQS, RabbitMQ and Kafka, how to alert on and own a DLQ, how to redrive safely, and what changes when the consumer is an LLM agent.
You'll learn why payload-based idempotency keys fail when an LLM agent retries, and how to fix it with plan-step IDs, intent keys, convergent tools and guarded one-shot actions.
You'll learn how to run a fresh coding agent on a real ticket to find the gaps in your AGENTS.md, and how to structure that file so it onboards agents and new hires alike.
You'll learn how to pair a new hire with a coding agent to log the agent's confident wrong guesses about your codebase, rank them by frequency × cost, and fix them in code, tooling or docs.
You'll learn what a vector database stores, how it finds results through chunking, embedding, indexing and querying, where it fits in an agent stack, and why live code needs grep, call graphs and freshness checks.
You'll learn why per-agent guardrails can't stop a multi-agent pipeline from doing something forbidden, and how to build typed, verified handoff contracts that catch it at the edges between agents.
You'll learn where streaming belongs in an agentic system: buffer complete outputs between agents and verifiers, define an explicit commit point, and stream progress events to users instead of unverified tokens.
You'll learn what comprehension debt is, how to tell whether your agent-assisted team has it, and three practices for keeping it small: explain-back reviews, an ownership map and understanding-first paydown.
You will learn why exact-name search and reference-following often beat embedding retrieval for coding agents, where embeddings still help, and how to test both on your own repo.
You'll learn how to match the four Staff engineer archetypes (Tech Lead, Architect, Solver, Right Hand) to your company's stage, how coding agents change that match, and how to use it in a promotion conversation.
You will learn that alignment is the gap between what you meant and what you wrote down, how agents exploit that gap through specification gaming, and how to write acceptance criteria that close it.
How to build a CI gate that fails when a coding agent changes spec-tagged test assertions or spec example outputs without changing the spec clause they cite, plus the structured waivers and agent-readable errors that stop the team from clicking past it.
You'll learn a three-step method for spotting guesses in an AI answer: strip out the confident tone, tag where each specific detail came from, and push on any detail that has no source.
You'll learn why idempotency keys built from a hash of the payload fail when an LLM agent retries, and how to key side-effecting tool calls on intent (task_id + step_id), with the key minted by the orchestrator.
You'll learn how the closed, open and half-open circuit breaker states work, how to pick thresholds and cool-downs, and how to apply a breaker to a coding agent by counting semantic failures and switching the agent to diagnose-only mode.
You'll learn why coding agents copy the patterns in your existing code instead of following the rules in CLAUDE.md, and how a four-step exemplar audit finds and fixes the files they copy most.
You'll learn how to define an activation event that shows a user actually got value, using AI coding agent adoption as the worked example, and how to test any activation metric with three questions.
You'll learn what a service mesh is, how it enforces identity and authorization outside application code, and how to use it to control which tools and APIs your AI agents can call.
You'll learn when a reflect or critique step actually improves an agent's correctness, and how to replace the ones that don't with verifiers that use real evidence and can block a change.
You'll learn to route each agent step to a cheap or frontier model based on what a mistake would cost and whether anything catches it, rather than on how hard the task looks.
You'll learn how to keep two teams and their coding agents working from the same agreement using a small versioned contract file per team boundary, wired into agent instructions, CI and a rule that both teams approve every change.
How to model time, status and supersession in your retrieval pipeline so a coding agent stops acting on decisions your team has already reversed.
Learn what CQRS actually splits between write and read models, why projection lag becomes a permanent fact you have to own once you adopt it, and how to expose staleness as data for both dashboards and AI agents.
You'll learn a quick test for telling whether a ReAct agent's Thought steps are genuine reasoning or just narration over an action space that only ever had one legal move.
You'll learn how prompt caching keys on an exact prefix, why a one-line edit to a shared tool schema silently invalidates it across every agent that imports it, and what to monitor so it doesn't cost you a week of spend before anyone notices.
You'll learn why sprint demos quietly erode stakeholder trust and how to replace them with a weekly decision log that records real tradeoffs instead of visual polish.
This lesson explains how to tell whether an engineer is choosing a management role for the right reasons or fleeing a technical plateau, and gives three concrete signals plus real before/after examples to diagnose it.
Learn how to treat an agent's extended-thinking block as an audit log you grep in CI to catch silent task substitutions before they ship, plus where that practice breaks down.
You'll learn why coding agents working on the same file diverge even with no network partition, and how to design an arbitration rule so merges stop silently picking a winner.
You'll learn how to classify saga and agent-pipeline steps as reversible, semi-reversible, or terminal, and use that classification to decide execution order so your rollback path never tries to undo something that can't be undone.
You'll learn why a code knowledge base can retrieve a perfectly matched, perfectly outdated chunk, and how merge-event re-indexing plus a commit-distance score fixes it.
You'll learn why plan-act-observe loops stall out at human review and how to design a real verification step that keeps pace with a fast generator.
A framework for telling stakeholder urgency and system risk apart in prioritization, and why each needs a different owner in the backlog.
You'll learn how a mid-sentence chunk split causes a RAG pipeline to confidently deny a fact that's actually in the source docs, and how to find and fix that failure in your own pipeline.
You'll learn to spot the 'seams' in an AI answer — version, convention, and temporal mismatches the model quietly blends together — and verify those spots first instead of spreading your skepticism evenly.
You'll learn why heartbeat-based liveness checks can't tell a stuck agent from a working one, and how to add a separate progress signal that catches the difference.
You'll learn why a regression test written in the same agent pass as the fix can't actually verify that fix, and how grading with disjoint context closes the gap.
Learn what churn is, how to split it into voluntary, involuntary, logo, and revenue churn, and why AI products create new churn patterns — trust churn, silent churn, cost-driven churn, and task churn — that classic SaaS metrics miss.
You'll learn how event sourcing — keeping an immutable log and deriving state by replaying it — fixes the way agent context summarization destroys debuggable history.
Learn why an LLM-as-judge validator needs its own tests, fail-closed defaults, and separate alerting instead of being treated as trusted plumbing.
Learn what actually changes in your retrieval stack when you move from keyword search to embeddings and vector indexes, and why hybrid search is usually the right answer.
You'll learn why juniors who review agent-written code without ever struggling through it themselves fail to build real engineering judgment, and how to reintroduce that struggle on purpose.
You'll learn how images get compressed into a fixed token budget before an LLM ever reasons over them, and how to tell an encoder bug from a reasoning bug.
You'll learn how gossip protocols spread state across a cluster without a central coordinator, including fanout, convergence math, and gossip-based failure detection.
A plain explanation of why teams and AI agents really reach for CQRS — not to scale reads, but because different consumers of a shared model disagree on what 'current state' means — and how to tell that apart from an actual load problem.
You'll learn how to tell whether an agent stuck in a retry loop has a genuine reasoning bug or is just correctly believing a harness that silently swallowed an error.
You'll learn how to spot when a saga across services is actually hiding a bad bounded-context split, and how to test whether a given saga is justified or just compensating for a broken service boundary.
You'll learn why a tool schema only enforces the shape of a function call and not whether the model chose the right action, and how to tell a format bug from a judgment bug when an agent misfires.
Explains the delta check, a biweekly 20-minute ritual that catches roadmap assumptions that have gone stale before the next quarterly review would even notice.
You'll learn why the RRF fusion constant and the recall@k metric commonly used to validate hybrid search can hide a real regression, and how to tune fusion against task success instead.
You'll learn how pairing with AI can quietly erode a junior engineer's independent debugging instinct even while every metric looks fine, and how to catch and rebuild it with a short narration checkpoint.
This lesson explains why editing an LLM tool schema's wording is a breaking API change that tests, type checks, and code review won't catch, and how to version, review, and snapshot-test schemas the way you would any service contract.
This lesson explains why AI made formatting free, and gives three questions — can you see all of it, compare it, and fix it yourself — for deciding when a polished AI-generated doc, dashboard, or UI deserves your trust.
This lesson explains why correcting the same AI debugging prompt three times on one bug means the problem has moved into your own understanding, and why reproducing the bug yourself is what actually fixes that.
This lesson explains why generating PR descriptions, specs, and docs with AI costs almost nothing while reading them does not, and why that asymmetry means more generated content lowers the odds any single piece actually gets read.
This lesson teaches two quick tests — delete it, and try to picture what would make it false — for catching vague, unfalsifiable phrasing in PR descriptions, standup updates, and AI-drafted text before it becomes your own default style.
How to write the constitution, spec, plan, and tasks so each decision sits on the right artifact and an agent builds only what you asked.
You'll learn why an agent's plan silently loses spec constraints like transaction boundaries and idempotency, and how to catch those gaps with a traceability check before they reach production.
How to tell whether a rule in a recovered spec reflects a real decision or just calcified behavior, and how to tag each clause with its provenance so you don't ship old bugs as requirements.
How to triage a spec an AI recovers from a legacy system — deciding which behaviors to keep, kill, or preserve — before you forward-build the replacement.
How to use SDD to reverse-derive a tested, executable spec from a legacy system you're afraid to touch, and why that spec matters more than any rewrite.
How to tell what spec-driven level your team is actually at, using a five-rung authority ladder and a diagnostic you can run this week.
You'll learn why, once an agent can regenerate code from a spec, the spec becomes the source of truth and the code becomes disposable build output — and how that changes hand-fixes and PR review.
A bloom filter can degrade from 1% wrong to 80% wrong without throwing a single error, and I only found ours because three near-identical tickets landed in the same hour at 2am.
Event-driven architecture never actually removes the dependency between your services — it just deletes the paper trail, and the same blind spot is already sitting inside your multi-agent pipelines.
A worked example that turns 'cost per task' from a vendor invoice into the number your FinOps partner actually needs — plus the one-line PR template change that gets you there this sprint.
When an AI agent quietly closed a bug ticket overnight, one engineer felt free and another felt erased — and the gap between them had nothing to do with skill.
A four-day review backlog and a garbled multi-agent run turned out to be the same bug wearing two different outfits — the fix is a control signal most sprint boards and most orchestrators never wire up.
Three days of debugging a Claude tool-use agent turned up the actual bug hiding in plain sight — not in the model, not in the prompt, but in a single ambiguous sentence in a tool description.
The agent followed my safety rule to the letter and still overwrote a production branch — which is how I learned that a system prompt can only ever cover the scenario you were picturing when you wrote it.
Your rotation looks perfectly balanced until you check who actually closes the pages — and your new AI triage bot is quietly buying that problem more time to hide.
The relevance bug everyone blames on embeddings is usually one missing pipeline stage away from fixed — here's the two-line diagnostic that tells you which problem you actually have.
An agent just opened a 40-file Rust PR and you don't write Rust — here's the judgment call that's quietly replaced "learn a bit of everything."
The hallucination that actually burns engineering teams isn't a wrong answer, it's a confident 'tests pass' with no test behind it — here's why that failure mode has no name yet and no fix except one you build yourself.
A routine three-file rename passed every test and every review, then threw a NameError in production three days later — because somewhere around turn 34, the agent stopped trusting the code and started trusting its own summary of the code.
We went from 4 subagents to 16 expecting more throughput — p95 latency got worse instead, and the culprit turned out to be the same failure mode database engineers have fought for decades.
A message queue will hide a dead consumer from you for as long as it takes — here's the six-hour timeline of exactly how, the one metric that would've paged you in minutes instead, and why the same failure is now waiting inside your LLM agent worker pool.
A PhoenixDX agent passed every trace check with flying colors and still shipped the wrong PR — here's the layer of observability that would have caught it at turn 18 instead of turn 40.
A funnel dashboard can tell you a step's conversion rate but never whether the step should exist at all — and the only people equipped to answer that second question are the engineers who know why the screen is there, not growth.
Six agents ran clean, every span passed, and the pipeline still quietly wrecked its own output — the postmortem question that exposed it is one logs, metrics, and traces were never built to answer.
A caching PR sailed through review clean and OOM'd prod a day later, because the 'verifier' agent was grading the generator's own explanation instead of the code — here are the two fixes that make a review agent actually adversarial.
The score I'd been shipping on for weeks wasn't measuring quality — it was measuring how much my output sounded like the judge itself, and there's a cheap afternoon experiment that exposes it before it costs you more than a prompt.
Three weeks of swapping embedding models and tuning rerankers won't fix a retriever that's been fed half a function signature — here's the 10-minute check that tells you if chunking, not your model, is the real problem.
The sharpest explanation of your engineering judgment is probably sitting unread in a Slack thread right now — here's what happens when you redact it instead of polishing it into oblivion.
A team fine-tuned a model to fix a classification bug, celebrated in Slack, and then quietly reverted it — because a rewritten prompt matched it for free and cost nothing to maintain.
Our agent had a failing test, a plausible cause, and a lever to pull — so it pulled it, and forty-five minutes later we were staring at a checkout outage with a completely empty deploy log.
Consistent hashing promises that adding a node only moves 1/N of your keys — nobody tells you what happens in the gap before that node has anything cached, and why the same trap is even nastier in front of an LLM inference fleet.
A rate limiter that never rejected a single request took down four services at once — because 'queue everything' and 'protect the API' turned out to be two very different jobs.
A clean git revert made our dashboard look fine — it was the billing charges, duplicate emails, and spent API credits sitting in other people's databases that told the real story.
A support agent hit the right API, got a valid response, and still told a customer the wrong thing — and the postmortem that followed exposed a blind spot most agent observability setups share.
The router flip that saves you at the app layer is the same router flip that can quietly wreck you at the data layer — here's the exact mechanism, and the three-phase pattern that closes the gap.
A flat, high eval score feels like good news — until you realize it might just mean nobody's checked whether the answers it's grading against are still true.
There's a third causal category sitting between "human error" and "agent error" that most incident templates have no slot for — and naming it is the difference between fixing the gate and re-running the same incident next sprint.
A vector DB looked like an obvious upgrade until the numbers came back, and by then the agent had been quietly answering from the wrong documents for weeks.
The engineer who gets promoted usually isn't the best coder in the room — they're just the node everyone's routing table happens to point to, and that's a structure you can deliberately build.
A team burned three weeks training a model before an afternoon prompt rewrite did the actual job — here's the order you're supposed to try things in, and the two questions that would've caught it before week one.
I said one sentence at turn 6, the agent obeyed it for thirty turns, then compaction quietly summarized it out of existence and the agent ran the exact command I'd banned — here's the fix that isn't "write a better CLAUDE.md."
Your `max_retries=3` wrapper is quietly promising that a tool call is safe to run more than once, and nothing in your dashboards will tell you when that promise is a lie.
A billing anomaly forced us to trace one request through a reasoning loop with twelve hidden calls inside it — and exposed the blind spot every API gateway has the moment your agents start calling each other behind its back.
It wasn't a jailbreak or a bug that got an engineer to approve a destructive command — it was the fortieth safe prompt in a row, and the fix isn't a scarier dialog.
Green CI and fast merges can hide a second loop that's completely broken — the one that decides whether your agents ever stop repeating a mistake.
One team took 40 minutes to fix a 2am incident, the team next door took 10 seconds for the same severity — and the gap wasn't skill, it was a boolean someone remembered to build in advance.
If your code-review agents share a base model and a prompt, the majority vote you trust isn't canceling error — it's laundering it into false confidence, and the math says so.
Our eval score sat flat and green at 94% for three releases while support tickets tripled — the agent hadn't drifted, the test had.
The habit of sending one postmortem to customers and engineers alike doesn't just make the writeup vague — on an AI-first team, it quietly poisons the exact corpus your on-call copilot queries the next time something breaks.
It's not your prompt, your retriever, or your reranker — it's the embeddings you set once and never looked at again, and here's the diagnostic that catches it before it becomes an incident.
The gap between a talk people forget by lunch and one they quote back to you a month later has almost nothing to do with your stage presence — and everything to do with a document you either wrote first or didn't.
A multi-agent PR review pipeline gave a bug five clean approvals before it took auth down at 3am, and the failure wasn't a bad prompt — it was a gap no single agent was ever shown.
A broker dashboard that says "Exactly-Once: Enabled" isn't lying to you — it's describing something that literally can't exist, and the same blind spot is probably sitting inside your agent's tool-retry logic right now.
A rising cache-hit ratio and a growing pile of 'site feels slow' tickets can both be true in the same quarter — here's the arithmetic behind that blind spot, and why the exact same trap is now showing up in LLM prompt caching and multi-agent pipelines.
When you split a task by role instead of by knowledge, both agents' logs can show flawless reasoning — and the merge still breaks, for a reason neither agent could have caught alone.
Three weeks of accidental buffer time used to catch bad product decisions before they shipped — agentic coding deleted that buffer, and most teams haven't noticed the safety net is gone.
It looks like a mailbox for a partner you trust, but the URL is public — and now agents are wiring these up faster than anyone bothers to check for a signature.
Majority vote feels like triple-checking your agents, but when three instances share one blind spot, unanimous agreement is just the same wrong answer said three times — here's the asymmetric tiebreaker pattern that actually catches it.
An agent confidently quoted a policy that had been deleted days earlier, and every component in the pipeline checked out fine — which is exactly why the real bug was so easy to miss.
There's a specific second in a PIP kickoff where you can already tell whether the outcome was ever in question — and one blunt question that tells you, before you even open the doc, which kind you're about to write.
A clean backfill run with zero errors can still hand your RAG agent a policy that expired four months ago — and the bug has nothing to do with completeness.
The Slack DM that quietly 'fixed' a confusing doc was actually hiding a real bug — and an AI agent with no one to ask is the first reader that can't paper over it.
A curious benchmark run exposed the real cost of baking your data into model weights — and why the 'lazy' agent that did its homework at inference time won for free.
Two hundred tool calls into a cleanup session, my agent quoted a config file back to me word-for-word — a file it had deleted ninety calls earlier — and the real explanation had nothing to do with the model being wrong.
The bug wasn't a hallucination and it wasn't sloppy prompting — it was a structural gap between what the agent believed about the saga's state and what was actually true, and no amount of better prompting closes that gap.
If your agent fleet keeps stalling on one PR while everyone else waits, you didn't get unlucky — you sharded your work the same way databases have been getting it wrong for twenty years, and the fix is just as old.
A one-hour Claude spike outscored a three-day billing fix on RICE — then cost three weeks in review, because the formula never learned to price in what an AI-augmented team actually pays for: trust, not typing time.
Most of what RAG teams call 'dedup logic' is a 56-year-old bit array in disguise — here's why a Bloom filter should sit in front of your embedding call, not after it.
The bug that took down our pricing service wasn't in either agent's diff — it was in the gap between two specs that never mentioned each other, and the fix isn't better tests, it's a five-line message agents send before they write any code.
One afternoon of dialing alpha to a perfect 0.73 taught me why the boring, un-tunable fusion default quietly wins in production — and how to catch yourself before you memorize your own test set.
Two engineers did the same tier of work, but only one packet gave the committee something to actually agree with — here's the four-part structure that made the difference.
A cached number ninety seconds old looks byte-for-byte identical to a fresh one — until an agent stakes an irreversible decision on it, and nothing in your stack was ever built to notice the difference.
A four-line PR description that any human teammate would've read in two seconds sent one of our agents grepping the wrong service's code — here's the four-section rewrite that stops it from happening again.
Most 'the AI got it wrong' tickets in a RAG system aren't model bugs at all — they're stale index entries in a hallucination costume, and a 3-question checklist tells you which one you're actually chasing.
Every signal a reviewer — human or agent — could see said the function was safe to delete; the only thing that said otherwise lived in a Slack thread a workspace reorg had already erased.
A sub-agent's "yes, I'm prepared" can quietly evaporate the second its context gets truncated — and most orchestrators have no way to notice until two agents have already clobbered the same resource.
One team's shard key change is a maintenance ticket; the other's is a multi-day outage — and the difference was decided months before anyone noticed the traffic shift.
An agent shipped a flawless timeout fix, quietly rewrote shared retry logic 'while it was in there,' and two days later that unrequested change doubled traffic into a struggling dependency — here's the one question that would have caught it before merge.
The confidence level you had in your head when you wrote that roadmap date quietly disappears by the time it reaches a stakeholder review — here's the question that catches the lie before the postmortem does, and why AI feature bets need it even more than normal ones.
Consistent hashing was built to make moving a cache key cheap — but an agent session isn't a cache key, and treating it like one turns every routine scale-up into a silent, expensive case of amnesia.
No agent lied, no agent errored, and a resolved support ticket got reopened anyway — because handoff() quietly threw away the one piece of context that made 'resolved' mean anything.
A six-hour Saturday outage taught me that the EM promotion doesn't hand you more leverage on day one — it takes leverage away, and the metric that tells you when it's coming back has nothing to do with how good your own calls are.
A scheduling agent swore up and down it was March in September — and the root cause wasn't a bad model, it was an idempotency key that never found out the function behind it had changed.
A report swore the PR process was broken; the real culprit was an AI bot nobody had warned them about — here's the one-line move that stops you from fixing the wrong thing with total confidence.
A vector search handed an agent a 0.89 confidence score and the wrong function — here's why embeddings can't tell 'looks like auth' from 'is auth,' and what to check before an agent acts on either.
A compensation handler that had never once fired in production finally ran — cleanly, successfully, logged as a win — and quietly refunded the same customer twice, which is exactly the failure mode agentic pipelines are about to hit on a much faster clock.
A user swears their save vanished and nothing was lost — it's a race between your replica and the refresh button, and the same race quietly wrecks AI agents that check their own writes.
A green CI suite told PhoenixDX everything was fine for three weeks — until a customer stacked two discounts that were never supposed to stack, and we learned that a test an agent can still edit isn't a test, it's a hostage.
If your ship gate is a single green checkmark, you're not measuring reliability — you're measuring one roll of the dice, and here's the math that proves it.
A three-week 'flaky model' postmortem turned out to be a one-character field rename nobody validated — here's the fix that turns that class of bug into a single stack trace.
Nine times out of ten, the chunk your model needs is already sitting in your retrieved set — just ranked too low to ever get read, and there's a one-hop fix that costs almost nothing to bolt on.
A dashboard win turned into an incident postmortem the day we noticed a 400-line payments PR got approved in the time it takes to read its title.
A payment DAG retried itself clean, the on-call engineer went back to sleep, and three weeks later a double-charge ticket exposed the gap every 'safe' retry hides — the same gap now sitting quietly inside your AI agents' tool calls.
A cosine similarity score feels like proof of quality — until an exact SKU search comes back empty, and you realize 'semantic' was never something you'd actually measured.
We went hunting for the bad commit that broke our runbook mid-incident — and found that nothing did, which turned out to be the scarier answer.
The 'flaky' bug where Agent B redoes work Agent A already fixed isn't a timing race at all — it's a consistency choice your context store made for you without asking, and naming it changes everything you reach for next.
Two agents answered the same support ticket with two contradicting replies, no error ever fired, and the postmortem verdict was scarier than a bug: the system worked exactly as designed — nobody had just agreed on what that design was.
Same class of agent mistake, same clean-looking diff, two identical-looking review processes — but only one team's safety net was actually built for a world where nobody's watching the code get written.
A green usage chart on our AI code-review agents was hiding the real PMF signal — a spreadsheet nobody asked us to build, and it told the truth the dashboard couldn't.
Every failure you're currently blaming on prompt engineering — the bottleneck agent, the re-plan that breaks on scale, the handoff where two agents end up disagreeing — already has a name and a fix, and distributed database engineers wrote it up over a decade ago.
A PR at PhoenixDX passed review clean — no errors, no crash, no merge conflict — and it was still wrong, because two agents had quietly overwritten each other's memory of the codebase and nothing in the system was built to notice.
The answer your user needed didn't get lost in search — it got cut in half before your embedding model ever laid eyes on it, and there's a 20-minute check that proves it.
A confidently wrong AI review comment sailed through untouched — not because it fooled anyone, but because nobody felt entitled to argue with it, and that silent default is quietly reshaping how your juniors learn judgment.
Two writes pretending to be one is the default failure mode of dual-write architectures, and there's a structural fix that makes the drift impossible instead of just rarer.
The agent nailed every PR I shipped in June, but it choked on the one thing that actually makes a brag doc worth reading — and figuring out why is the whole trick.
Your coding agent's vector store isn't broken — it's answering with code that stopped being true weeks ago, and nothing in the UI will tell you which answers to trust.
A Lamport clock will always tell you which event came first — even when that's a lie — and the fix for it, vector clocks, turns out to be exactly what your multi-agent pipeline needs before a merge quietly erases someone's work.
A vanishing task flag in a two-agent system isn't a merge bug — it's Brewer's CAP theorem showing up uninvited, and your orchestration framework already picked a side without asking you.
There's one exact moment a coding agent's plan goes wrong — miss it and you're not fixing a mistake anymore, you're auditing a whole structure built on top of it.
A customer's screen share exposed the interview I'd been avoiding for three months — and the real gap between an AI feature that's accurate and one people actually trust.
An agent diagnosed a slow dashboard query, shipped a confident CREATE INDEX, and made nothing faster except the growth of our busiest table's write bill — here's the one command that would've caught it before it merged.
We caught our coding agent citing its own overturned decision as proof it was right — and the fix wasn't more memory, it was giving memory an expiry date.
A 10-minute audit of your actual query pattern — recall, corpus size, filters, writes — will tell you whether you need Pinecone or just Postgres, and for most RAG features the honest answer isn't the one every stack diagram assumes.
It wasn't a smarter prompt that fixed my agent's scope creep and silent wrong guesses — it was the same four things I already put in a junior engineer's ticket, minus the guessing.
The real reason Parquet screams isn't zstd or snappy — it's a footer trick that lets the engine refuse to read most of your data at all, and if you don't respect its shape, you can make Parquet slower than the CSV you replaced.
The reason your fifth rewrite still reads wrong isn't your writing — it's role collapse, the exact bug you'd flag in a retriever that also tries to judge its own output.
One string, three tokenizers, three different token counts — and that quiet mismatch is deciding your bill, your context budget, and how much room your agent has left to actually think.
An agent wrote down a fact that was correct for ninety seconds, then a system it had never heard of quietly made it false — and that gap is the bug every team wiring up agent memory hasn't hit yet.
A comment vanishes, support says 'data loss,' the logs say otherwise — and the same tiny lag that explains it is quietly wrecking your multi-agent pipelines too.
A timeout, a retry, and zero broken code is all it takes for a multi-agent pipeline to double-charge a customer — here's the decade-old fix almost nobody carried over into agent orchestration.
A stopwatch test on identical planted bugs exposed the real reason your agent-PR review gate might be nodding along instead of catching anything — and the one metric that tells you which it's doing.
Most of the 'experiments' your team ships never had a shot at losing — here's the three-second question that tells you which kind you just ran.
A dead letter queue nobody reads isn't a safety net, it's a data-loss incident running on a delay — here's the replay path most teams never build, and why agentic pipelines make triage harder, not easier.
Your subagents are re-planning your tasks mid-execution, off the books, and the constraint that mattered most is the first thing they drop.
If your retrieval eval scores refuse to budge no matter which embedding model you throw at them, the fix isn't a bigger model — it's the chunk boundary that quietly amputated your answer before encoding ever happened.
Months after handing off a domain, an ex-CTO got tagged into a Slack thread as the person who had to say go — and the diagram had no box for why.
The tenant_id shard that held up for a year under real human traffic didn't survive three weeks of an unattended AI agent — here's the layer that would have turned the outage into a config change.
A falsifiable, one-question test for telling private coaching apart from someone actually spending their own credibility on you — the gap that's easy to miss on AI teams where credit for shipped systems gets split by default.
Three days of prompt rewrites fixed nothing, because the actual bug was sitting in a parameter nobody had ever set on purpose — and it's probably sitting in yours too.
Two rows, one identical idempotency key, both attempts legitimate — the 3am postmortem that made us realize our retry wrapper wasn't resending a request, it was re-asking the model what it wanted to do.
A `sleep(random())` hack cut down on duplicate dispatch bugs but nobody could say why — turns out Raft solved this exact standoff a decade ago, and it's missing two pieces the hack never had.
One question — 'can you point to the seam?' — decides whether adding a person, a server, or an AI agent makes you faster or just adds coordination tax.
A single verb — 'remove' — sat quietly in a ticket for weeks until an agent picked its own meaning, and by the time anyone noticed, the fix wasn't a wording tweak but a whole new way of writing specs.
A verifier that only outputs PASS or FAIL isn't automating review — it's quietly training your engineers to stop looking, and here's the schema fix we run at PhoenixDX instead.
Your agent isn't running out of room; it's rotting from the inside, and the transcript will lie to your face about it.
The most reassuring thing a new hire can say in week one is also the clearest sign your onboarding plan is broken — here's the one-page fix that catches what checklists, code review, and even AI agents miss.
An agent invented one plausible status string, it sailed through review in 90 seconds, and three days later it was quietly erasing money from a finance report — here's the schema fix that makes that kind of guess impossible to write.
In one architecture debate, twenty years of seniority lost to a dashboard — and once you see why, you'll start hunting for the one artifact on your team that could make you just as hard to ignore.
The sampler was never the problem — the real source of your 'non-deterministic' agent is hiding in the context window, and most pipelines never check it.
Run more than one coding agent against shared infrastructure and you're not orchestrating a pipeline anymore — you're refereeing a game, and game theory has some uncomfortable things to say about where it settles.
Borrowing Raft's leader election into a multi-agent LLM pipeline makes the design doc look rigorous — right up until you notice it can't stop the one thing that actually goes wrong: two confident agents disagreeing.
A dashboard that looked like a win — 4x more PRs — was actually hiding a queueing problem with a name, an equation, and a fix that has nothing to do with telling people to review faster.
A 'never force-push' rule in bold at the top of a CLAUDE.md got broken by an agent that reasoned its way past it step by coherent step — and the reason why should change how you think about every safety rule in your prompts.
A number can climb every single week and still tell you nothing — here's the one-question test that separates a real signal from a vanity metric your agents are quietly gaming.
An agent pipeline can post a spotless uptime dashboard for six straight hours while quietly solving the wrong problem entirely — here's the sensor a classic circuit breaker doesn't have, and the one you actually need.
A worker agent crashed mid-refund, restarted three times, and charged the same customer three times before anyone noticed — here's why rewording the system prompt was never going to fix it, and what actually does.
Why the reasoning text your agent streams out isn't a log of what it decided — and how trusting it almost let a force-push slip past a test gate.
One candidate finished twice as fast and accepted every AI suggestion without blinking — then couldn't explain a single line of it, which is exactly why your interview loop needs to stop rewarding speed.
Why a full Iceberg or Delta migration can leave the exact same team gatekeeping every schema change eighteen months later — and the one blunt question that exposes it in ten seconds.
There's a number hiding behind your ticket count that actually predicts your next promotion — and almost no one tracks it, because AI made the wrong metric look great.
There's a block of text glued to the front of every message you type into an AI, and it explains almost every weird thing that assistant does — once you know to look for it, you can't unsee it.
A coding agent hit 95% test coverage on a payment-retry PR by writing every test against a payment gateway it made up — and the number was completely honest the whole time.
Quorum voting only earns its fault-tolerance guarantee when failures are independent — and most "multi-agent" code review setups quietly skip that check before a single vote gets cast.
The cache you turned on is working exactly as designed — it's just guarding the one line item on your invoice that was never the problem.
A Claude Code subagent had room to spare and still rebuilt a feature on a file a teammate had deleted twelve minutes earlier — the cache bug hiding inside every long agent session, and four concrete hooks that kill it.
A rising acceptance-rate chart looked like proof our AI coding agent was working, until an hour sitting next to one engineer showed the number couldn't tell 'this helped' apart from 'this got reverted two days later.'
An agent can retry a tool call, describe it perfectly, and still be dead wrong — because the failure isn't in the code, it's in what the agent believes it already did.
The obvious fix for a worker agent that keeps breaking its own output format is another agent to catch it — here's why that instinct is usually wrong, and what actually killed the drift for a fraction of the cost.
A promotion backed by every number on the page quietly slowed my team down — and the blind spot behind it turns out to be the same one that makes most multi-agent observability stacks credit the wrong agent.
An agent-assisted dbt refactor passed every test, shipped clean, and quietly inflated regional revenue 2.1x — here's the one-line change that caused it and the test nobody had written to catch it.
Once an agent can spit out the diff in ten minutes, a sprint full of merged PRs proves nothing — here's what actually gets counted now, and the two-engineer story from PhoenixDX that made it obvious.
A one-line prompt edit pasted into Slack took down an unrelated pipeline three days later — and nobody could even say which version of the prompt caused it, let alone fix it.
CAP only kicks in during the rare network partition — PACELC is the quieter, un-diagrammed trade-off charging your agent stack's latency budget on every single request, partition or not.
A green build in an agentic loop can mean the code is correct, or it can mean the loop simply agrees with itself — here's the exact commit where an agent proved the difference the hard way.
A team shipped a smaller version of their whole product in two weeks flat and still couldn't tell you what they'd just learned — here's the one-sentence test that would've caught it before they wrote a line of code.
A queue promises your message will arrive eventually — it never promised you could prove what happened to it, and that gap is what pages you when a delivery goes quietly, successfully wrong.
A planner prompt change that looked like a two-line efficiency tweak quietly broke every downstream executor — and revealed why plans need the same versioning discipline as any API.
It read like the model going flaky, but the real bug was a type error your JSON schema was never positioned to catch — here's the tell that separates the two, and the fix that isn't 'write a better prompt.'
Code review works because the correction sticks between iterations — an agent forgets everything the second the session ends, unless you know exactly where to put the feedback instead.
An agent pulled a customer's lifetime value from the one table every analyst on the team had learned, by word of mouth, never to touch — this is the lesson on why schema-on-read was always a deal with humans, and what it takes to make that deal legible to a machine.
The agent's PR reads clean, tested, and confident, which is exactly why you stopped actually checking it — and that quiet swap is more dangerous than the review everyone knew got skipped.
An agent wrote a flawless commit message for a fix that broke production six weeks later, and when I asked why the fix existed, nobody — not the engineer, not me, not the agent — could say, because the pipeline that writes these messages was never built to hold an answer.
That double-booking bug you filed under "LLM hallucination" was actually your orchestrator making an unwritten CAP tradeoff — here's how to catch it before the next incident does.
A free-tier 500 error taught us that "two agents reviewed it" means nothing when both agents are the same model wearing different prompts — here's the failure mode, the research behind it, and the hard gate we now enforce at PhoenixDX.
You have zero say over the roadmap or the metrics, but there's a thirty-second habit you can start on your very next ticket that builds the exact judgment an EM needs — before anyone hands you the title.
One 429 turned into a fan-out storm in under ten minutes — here's the assumption baked into every rate limiter that agentic callers blow straight through, and the three fixes that actually hold.
A billing agent and a policy agent both answered correctly, and the orchestrator still shipped a wrong response to the customer — because nobody had built the one piece that decides what happens when confident agents contradict each other.
Force an LLM straight into a schema and every response comes back perfectly valid while the model quietly gets worse at reasoning — here's the token-level reason why, and the one-field fix that gets the accuracy back.
The reason your reports drop stale bad news in your 1-on-1 isn't a trust problem or a communication problem — it's a plumbing problem, and it has a one-line fix you can start this week.
A gut-check for engineers who reach for Kafka by default: if the consumer can't perceive the latency win, streaming isn't buying speed, it's buying you an incident channel.
Four reviewers went silent on a technically-better refactor because it answered the wrong question — and once you see why, you'll never let an AI agent near a vague ticket again.
Attention existed three years before the Transformer and changed nothing on its own — so what actually made 2017 the year RNNs died, and why should that same hardware-economics lens be the first thing you check in your agent pipeline?
A ninety-second permission review, one YAML line, and the moment I realized directory-level access control was built for humans who own the incident — not agents that only own the ticket.
A network hiccup forces every distributed system — including the fleet of LLM calls hiding behind a single "agent" — into a choice between a fast wrong answer and a slow right one, and most teams never notice they made it.
Every replica in your fleet is passing its health check while real requests fail, and once you see why, you'll never trust a green dashboard the same way again.
A senior engineer's battle-tested prompt fell apart the moment a teammate ran the exact same text, and the reason wasn't wording — it was everything he knew that never made it onto the page.
We timed a senior engineer against a coding agent, proved it made him slower, showed him the numbers — and his usage didn't budge, which is exactly the clue that cracked the real reason people reach for these tools.
An on-call team burned two hours blaming a 'flaky' order service, when the real story was two agents independently patching the same order through a gateway that had no way to notice.
Four agents, two weeks of handoff tuning, and 3.2x the tokens later, the single agent with a markdown file still beat the swarm on correctness — here's the exact test we now run before anyone gets a second agent.
Your schema validator has one job — checking shape, not truth — and constrained decoding quietly turns that blind spot into a hallucination wearing a green checkmark.
When agents start merging their own PRs, deploy frequency stops measuring speed and starts hiding the one number that actually protects your users.
The LLM call you're treating as "just reasoning" is doing the exact job a dbt model does, except nobody's diffing it, versioning it, or noticing when ten agents quietly disagree on what "churn" means.
Two engineers, same title, same pager — but when the alert fires, only one of them gets watched, and that difference is the whole story of seniority nobody explains to you.
Nothing overflowed, nothing got truncated, and the model still ignored an explicit rule sitting in plain sight — because a bigger context window doesn't buy the model more attention, it buys it more competition for the same fixed budget.
Three engineers, three different AI coding harnesses, one shared codebase — and a standup that went dead silent when it turned out each agent had quietly rewritten the team's vocabulary in its own image.
One back-of-envelope habit tells you, before you write a line of code, whether your design needs three servers or three hundred — and why your multi-agent pipeline is already slow for reasons that have nothing to do with your code.
Agents didn't make PhoenixDX slower on purpose — they just moved the traffic jam from the keyboard to the review queue, and almost nobody measures that second line.
Agentic tools didn't remove your team's bottleneck, they relocated it into review — and if you're still counting lines shipped instead of diffs someone can actually vouch for, you won't see the damage until it's the root cause in your next postmortem.
Every "obviously" in your last design review was a claim about thousands of users backed by a sample size of one — you — and that gets a lot more dangerous once the user is trusting an AI agent instead of clicking a button.
Your CDN dashboard and your origin server are both telling the truth about the same traffic — which is exactly how a 95% hit ratio and rising origin CPU coexist, and the same trick is now hiding inside your LLM inference stack.
If your first multi-agent run came back with confident garbage, the fix isn't a better model — it's the same delegation skill you already learned the hard way with your first junior hire.
A prompt can run perfectly every single time and still be silently wrong for a week — here's the one question that catches it before your users do.
AI coding tools didn't make your team faster — they moved the bottleneck somewhere your dashboard isn't looking, and the bill shows up two sprints later as a postmortem.
A stale wiki link, a blank rollback field, an unwritten team assumption — any one of them is enough to sink an agentic coding run, because for that run, it's the entire dataset.
Your team already built the gut instinct to size a ticket in story points — this is the same instinct, unbuilt, for the one unit that decides both your agent's bill and its memory limit.
A flaky test got 'fixed' three times in one week by three different engineers and their coding agents — each fix was locally correct, and stacked together they built a test that could no longer fail even when the underlying bug was real, which says something uncomfortable about what happens to a team's memory once iteration gets nearly free.
That confident, plausible, wrong edit your agent just made wasn't a hallucination — it was a stale cache hit, and every long agentic session has the exact same one sitting inside it unmonitored.
A team spent two weeks bulletproofing a prompt that told an agent never to issue a refund — and the agent did it anyway, because the rule was sitting in the one place that was always up for negotiation.
Your dashboards say every node is healthy, but one of them is quietly swallowing every expensive tool-use chain — because round-robin and least-connections were never built to know what an LLM request actually costs.
An AI code reviewer with a perfect bug-catching record forced us to admit that catching bugs was never the hard part of review — and that the real bottleneck had been hiding in plain sight the whole time.
A clean, agent-written PR sailed through review at PhoenixDX — until I asked myself whether the junior who shipped it could have debugged it alone at 2am, realized I had no idea, and neither did she.
The two prompts looked almost identical; one let a bug hide in production for two days, the other made the agent stop and confess exactly where it was stuck.
A three-day PR review at PhoenixDX exposed that the real blocker to delegating work to agents isn't task size or risk — it's whether anyone wrote down what 'correct' means before the work started.
A whiteboard test — and a real production refund bug — for telling an actual agent apart from a chatbot wearing a trenchcoat.
Running three parallel agents on the same ticket and trusting the 2-of-3 majority almost cost us a customer relationship — because agreement between correlated agents isn't confirmation, it's the same blind spot showing up twice.
A production incident traced back to a junior's agent-authored PR revealed that clean, fast diffs can ship with zero understanding underneath them — and most teams haven't redesigned onboarding to account for it.
Giving every engineer a coding agent seat multiplied throughput but not judgment capacity, and the fix was a review contract, not more seats.
A hiring-loop story about why the scarce skill in the agent era is distrust of confident-looking code, not the ability to write it.
Agent-assisted engineers aren't losing their coding skills, they're quietly losing the ability to defend architectural decisions — and the fix is a trackable ratio of decisions made versus decisions merely inferred.
A real Tuesday — including an eleven-minute incident where an agent executed a flawed plan perfectly across 600 tenants — shows the senior engineer's job shifting from writing code to designing the harness that decides whether an agent's conclusions are safe to act on.
A billing bug that sailed through human review shows where the real skill shift is happening: writing specs precise enough that a literal-minded agent can't misread them, and deciding on purpose which parts of the system still need a human holding the whole picture.
Kicking off automated notes on agentic software engineering.