Your Docs Have Been Broken for Years. AI Agents Are the First Thing That Can't Hide It.

Day 11: The Documentation Test Nobody Ran Until Agents Showed Up

Your documentation has been broken for years and nobody knew, because every time it failed, someone just quietly DMed a teammate and the failure vanished before it could be measured. Hand the same doc to an AI agent and there's no one to DM. That's why an agent is one of the first tools that will actually show you where your docs break — not because it's smarter than your engineers, but because it has nowhere to hide the gap.

Where the agent stalls

I handed a coding agent one of our internal onboarding docs and gave it one job: add a new endpoint, following the pattern already written down. Three steps in, it stopped cold on a line that read something like 'configure the service using the standard env setup' and asked me, flat out, what 'standard env setup' meant. Here's the part that got me: every engineer who'd onboarded at PhoenixDX in the past year had hit that exact same sentence. Every single one had done what humans do — pinged someone on the platform team and moved on. The agent couldn't do that. It stalled in public, on the exact line, with a timestamp attached.

Why this is a mechanism, not a coincidence

A human reading a bad doc has an escape hatch: ask someone. That escape hatch never shows up in any dashboard you're tracking — no ticket, no comment on the doc, just a two-minute Slack thread that dissolves the second it's answered. The gap never becomes a cost anyone can see, so nobody fixes it, so the next person hits the exact same wall. An agent with zero tribal knowledge has nobody to ask. It either guesses — silently, sometimes wrong — or it stops and asks you directly. Either way, you get a signal you never had before. That's the real value here: the agent isn't replacing the doc. It's an honest instrument for measuring whether the doc actually works without a human standing next to the reader.

The career skill hiding inside this

The skill underneath Day 11 is writing so your context survives the room without you in it. That's not a writing tip — it's a career lever. The engineers who get promoted fastest aren't the ones carrying the most tribal knowledge around in their heads. They're the ones whose docs let other people move without needing to track them down. Tribal knowledge puts a hard ceiling on your leverage: it caps out at the number of people who can physically reach you. Docs that actually work remove that ceiling. And on an AI-era team, that ceiling gets tested constantly, because agents are now part of the population trying to get things done using nothing but what you wrote down.

The concrete test you can run today

• ▹Pick one doc — ideally the one that gets pinged about the most, not the one you're proudest of.
• ▹Hand it to a coding agent with zero extra context: no chat history, no 'also go check with X,' just the doc.
• ▹Give it one real task the doc claims to enable.
• ▹Watch for two moments: where it guesses without telling you, and where it stops to ask a clarifying question.
• ▹Both are debt markers, but the stop-and-ask is the gift — it points at the exact sentence, not a vague 'this doc feels off.'

A war story from PhoenixDX

We had a doc describing how a background job should handle retries. Three different engineers had reviewed it over its lifetime, and nobody ever flagged a problem — because every one of them already knew, from having lived through it, which retry path was the real one in production. The doc itself described two options that both sounded reasonable and never said which one was the default. I pointed a coding agent at it to build a related feature, and it picked the wrong option — the one that was technically consistent with what the doc said, but wrong in practice. That version would have shipped clean, sailed through review from anyone without the tribal context, and quietly caused a duplicate-processing bug. The doc wasn't unclear. It was clear enough for people who already knew the answer — a special kind of doc that looks perfectly fine in every human review and falls apart the moment the reader doesn't already carry the context the doc is silently assuming.

Why this beats 'clarity' as a metric

Clarity is subjective, and it's almost always judged by people who already have the context — which makes your reviewers structurally the worst people on the team to catch this kind of gap. Task completion doesn't have that problem. It's binary: the agent either finishes the task correctly using nothing but the doc, or it doesn't. No rubric required. Just watch where it stops.

How to apply it this week

• ▹Pick the doc your team pings you or a teammate about most — that's your highest-signal candidate.
• ▹Run the agent test exactly as above. No extra context fed in on the side.
• ▹Fix only the exact spot where it stalled. Don't rewrite the whole doc — that's a different project with a different payoff, and it drowns out the signal you just got.
• ▹Check back in two weeks and see if the human pings on that doc dropped too. They usually do — the stall point was the same gap people were already routing around.

The tell

When two readers with nothing in common except zero prior context — one human, one agent — stall on the exact same paragraph, the paragraph is the bug. Not the reader. Not their attention span. Not 'people don't read docs anymore.' Tomorrow picks up from here: how to turn that fix into a habit instead of a one-time cleanup.

Extend your knowledge

• ▹Run the agent doc test on your team's most-pinged runbook or onboarding doc this week, and write down the exact line it stalled on — not just 'it got confused.'
• ▹Grab a teammate and have them read the same doc cold, no prior context. See if they stall at the same paragraph the agent did.
• ▹Pull up your last postmortem that traced back to a misread doc. Check if it fits the 'clear enough for people who already knew the answer' pattern.
• ▹If you haven't read Day 10 yet, go back — this kind of context-survival writing builds directly on it.