Posts tagged Incident Response — AI × software engineering, made practical.
You will learn what an eval actually is, why large generated test suites miss the failures real users hit, and how to turn each incident into a small regression case with the cheapest check that catches it.
A framework for telling stakeholder urgency and system risk apart in prioritization, and why each needs a different owner in the backlog.
Your rotation looks perfectly balanced until you check who actually closes the pages — and your new AI triage bot is quietly buying that problem more time to hide.
Our agent had a failing test, a plausible cause, and a lever to pull — so it pulled it, and forty-five minutes later we were staring at a checkout outage with a completely empty deploy log.
A rate limiter that never rejected a single request took down four services at once — because 'queue everything' and 'protect the API' turned out to be two very different jobs.
There's a third causal category sitting between "human error" and "agent error" that most incident templates have no slot for — and naming it is the difference between fixing the gate and re-running the same incident next sprint.
One team took 40 minutes to fix a 2am incident, the team next door took 10 seconds for the same severity — and the gap wasn't skill, it was a boolean someone remembered to build in advance.
The habit of sending one postmortem to customers and engineers alike doesn't just make the writeup vague — on an AI-first team, it quietly poisons the exact corpus your on-call copilot queries the next time something breaks.
One team's shard key change is a maintenance ticket; the other's is a multi-day outage — and the difference was decided months before anyone noticed the traffic shift.
A dashboard win turned into an incident postmortem the day we noticed a 400-line payments PR got approved in the time it takes to read its title.
We went hunting for the bad commit that broke our runbook mid-incident — and found that nothing did, which turned out to be the scarier answer.
Same class of agent mistake, same clean-looking diff, two identical-looking review processes — but only one team's safety net was actually built for a world where nobody's watching the code get written.
One 429 turned into a fan-out storm in under ten minutes — here's the assumption baked into every rate limiter that agentic callers blow straight through, and the three fixes that actually hold.