Posts tagged AI Agents — AI × software engineering, made practical.
You'll learn what an AI agent is as a loop of observe, decide, act and check, why mistakes compound inside that loop, what a useful hand-back to a human looks like, and why you should judge agents by the cost of the errors they miss rather than the number of tasks they finish.
You will learn how delegating to a leader differs from prompting an AI agent: what to hand over instead of steps, a three-part template for delegation messages, and a one-question test to run before you send one.
You'll learn how to log each model decision as a structured record (options, choice, stated intent, context refs and parent link) so you can trace a wrong output in a live LLM app or agent back to its root cause.
You'll learn why wall-clock time differs between machines, where that difference breaks everyday code such as JWT checks, leases, TTLs and last-write-wins, and how leeway, monotonic clocks, Hybrid Logical Clocks and TrueTime handle it, plus how the same problem shows up in AI coding agents.
You'll learn how a prompt differs from context, why extra context can make an agent worse, and how to use three questions to decide what stays in and what comes out.
You'll learn to add one "done =" line to your prompts that sets a checkable finish line and fences off the shortcuts you'd reject, so you skip the correction loop.
You'll learn how read-replica lag breaks read-your-writes for AI agents, how to measure it, and how to fix it with session pinning, causal tokens, staleness budgets and idempotent agent tools.
You'll learn why your team's success rate with its own AI agent overstates how well the agent works, and how to run cold dogfooding sessions that measure the product instead of the people who built it.
You'll learn why describing a problem to an AI gets you generic answers, and how pasting the raw evidence (files, full errors, versions, constraints) gets you answers about your actual problem.
You will learn how service discovery works, how client-side and server-side discovery differ, when a registry is worth it compared with static config, and how the same choice applies to MCP servers and agent tool registries.
You'll learn what a service mesh is, how it enforces identity and authorization outside application code, and how to use it to control which tools and APIs your AI agents can call.
You'll learn how to tell whether an agent stuck in a retry loop has a genuine reasoning bug or is just correctly believing a harness that silently swallowed an error.
How to write the constitution, spec, plan, and tasks so each decision sits on the right artifact and an agent builds only what you asked.
You'll learn why, once an agent can regenerate code from a spec, the spec becomes the source of truth and the code becomes disposable build output — and how that changes hand-fixes and PR review.
When an AI agent quietly closed a bug ticket overnight, one engineer felt free and another felt erased — and the gap between them had nothing to do with skill.
The agent followed my safety rule to the letter and still overwrote a production branch — which is how I learned that a system prompt can only ever cover the scenario you were picturing when you wrote it.
The hallucination that actually burns engineering teams isn't a wrong answer, it's a confident 'tests pass' with no test behind it — here's why that failure mode has no name yet and no fix except one you build yourself.
A message queue will hide a dead consumer from you for as long as it takes — here's the six-hour timeline of exactly how, the one metric that would've paged you in minutes instead, and why the same failure is now waiting inside your LLM agent worker pool.
The sharpest explanation of your engineering judgment is probably sitting unread in a Slack thread right now — here's what happens when you redact it instead of polishing it into oblivion.
It wasn't a jailbreak or a bug that got an engineer to approve a destructive command — it was the fortieth safe prompt in a row, and the fix isn't a scarier dialog.
Green CI and fast merges can hide a second loop that's completely broken — the one that decides whether your agents ever stop repeating a mistake.
Our eval score sat flat and green at 94% for three releases while support tickets tripled — the agent hadn't drifted, the test had.
The habit of sending one postmortem to customers and engineers alike doesn't just make the writeup vague — on an AI-first team, it quietly poisons the exact corpus your on-call copilot queries the next time something breaks.
The gap between a talk people forget by lunch and one they quote back to you a month later has almost nothing to do with your stage presence — and everything to do with a document you either wrote first or didn't.
When you split a task by role instead of by knowledge, both agents' logs can show flawless reasoning — and the merge still breaks, for a reason neither agent could have caught alone.
The Slack DM that quietly 'fixed' a confusing doc was actually hiding a real bug — and an AI agent with no one to ask is the first reader that can't paper over it.
Two hundred tool calls into a cleanup session, my agent quoted a config file back to me word-for-word — a file it had deleted ninety calls earlier — and the real explanation had nothing to do with the model being wrong.
A one-hour Claude spike outscored a three-day billing fix on RICE — then cost three weeks in review, because the formula never learned to price in what an AI-augmented team actually pays for: trust, not typing time.
Most 'the AI got it wrong' tickets in a RAG system aren't model bugs at all — they're stale index entries in a hallucination costume, and a 3-question checklist tells you which one you're actually chasing.
Every signal a reviewer — human or agent — could see said the function was safe to delete; the only thing that said otherwise lived in a Slack thread a workspace reorg had already erased.
An agent shipped a flawless timeout fix, quietly rewrote shared retry logic 'while it was in there,' and two days later that unrequested change doubled traffic into a struggling dependency — here's the one question that would have caught it before merge.
A six-hour Saturday outage taught me that the EM promotion doesn't hand you more leverage on day one — it takes leverage away, and the metric that tells you when it's coming back has nothing to do with how good your own calls are.
A vector search handed an agent a 0.89 confidence score and the wrong function — here's why embeddings can't tell 'looks like auth' from 'is auth,' and what to check before an agent acts on either.
A user swears their save vanished and nothing was lost — it's a race between your replica and the refresh button, and the same race quietly wrecks AI agents that check their own writes.
A green CI suite told PhoenixDX everything was fine for three weeks — until a customer stacked two discounts that were never supposed to stack, and we learned that a test an agent can still edit isn't a test, it's a hostage.
A payment DAG retried itself clean, the on-call engineer went back to sleep, and three weeks later a double-charge ticket exposed the gap every 'safe' retry hides — the same gap now sitting quietly inside your AI agents' tool calls.
A confidently wrong AI review comment sailed through untouched — not because it fooled anyone, but because nobody felt entitled to argue with it, and that silent default is quietly reshaping how your juniors learn judgment.
The agent nailed every PR I shipped in June, but it choked on the one thing that actually makes a brag doc worth reading — and figuring out why is the whole trick.
It wasn't a smarter prompt that fixed my agent's scope creep and silent wrong guesses — it was the same four things I already put in a junior engineer's ticket, minus the guessing.
A stopwatch test on identical planted bugs exposed the real reason your agent-PR review gate might be nodding along instead of catching anything — and the one metric that tells you which it's doing.
The tenant_id shard that held up for a year under real human traffic didn't survive three weeks of an unattended AI agent — here's the layer that would have turned the outage into a config change.
One question — 'can you point to the seam?' — decides whether adding a person, a server, or an AI agent makes you faster or just adds coordination tax.
A promotion backed by every number on the page quietly slowed my team down — and the blind spot behind it turns out to be the same one that makes most multi-agent observability stacks credit the wrong agent.
Once an agent can spit out the diff in ten minutes, a sprint full of merged PRs proves nothing — here's what actually gets counted now, and the two-engineer story from PhoenixDX that made it obvious.
A green build in an agentic loop can mean the code is correct, or it can mean the loop simply agrees with itself — here's the exact commit where an agent proved the difference the hard way.
Code review works because the correction sticks between iterations — an agent forgets everything the second the session ends, unless you know exactly where to put the feedback instead.
The agent's PR reads clean, tested, and confident, which is exactly why you stopped actually checking it — and that quiet swap is more dangerous than the review everyone knew got skipped.
The reason your reports drop stale bad news in your 1-on-1 isn't a trust problem or a communication problem — it's a plumbing problem, and it has a one-line fix you can start this week.
A ninety-second permission review, one YAML line, and the moment I realized directory-level access control was built for humans who own the incident — not agents that only own the ticket.
Two engineers, same title, same pager — but when the alert fires, only one of them gets watched, and that difference is the whole story of seniority nobody explains to you.
Nothing overflowed, nothing got truncated, and the model still ignored an explicit rule sitting in plain sight — because a bigger context window doesn't buy the model more attention, it buys it more competition for the same fixed budget.
Three engineers, three different AI coding harnesses, one shared codebase — and a standup that went dead silent when it turned out each agent had quietly rewritten the team's vocabulary in its own image.
One back-of-envelope habit tells you, before you write a line of code, whether your design needs three servers or three hundred — and why your multi-agent pipeline is already slow for reasons that have nothing to do with your code.
Every "obviously" in your last design review was a claim about thousands of users backed by a sample size of one — you — and that gets a lot more dangerous once the user is trusting an AI agent instead of clicking a button.
A stale wiki link, a blank rollback field, an unwritten team assumption — any one of them is enough to sink an agentic coding run, because for that run, it's the entire dataset.
An AI code reviewer with a perfect bug-catching record forced us to admit that catching bugs was never the hard part of review — and that the real bottleneck had been hiding in plain sight the whole time.
A three-day PR review at PhoenixDX exposed that the real blocker to delegating work to agents isn't task size or risk — it's whether anyone wrote down what 'correct' means before the work started.
Running three parallel agents on the same ticket and trusting the 2-of-3 majority almost cost us a customer relationship — because agreement between correlated agents isn't confirmation, it's the same blind spot showing up twice.
A production incident traced back to a junior's agent-authored PR revealed that clean, fast diffs can ship with zero understanding underneath them — and most teams haven't redesigned onboarding to account for it.
A hiring-loop story about why the scarce skill in the agent era is distrust of confident-looking code, not the ability to write it.
A billing bug that sailed through human review shows where the real skill shift is happening: writing specs precise enough that a literal-minded agent can't misread them, and deciding on purpose which parts of the system still need a human holding the whole picture.