I Fed My Agent 40 Pages of Spec — It Missed the One Line That Mattered

Day 2: The Context Window Is an Attention Budget, Not a Storage Bin

More tokens in the context window feels like more room to work with — like upgrading from a studio to a warehouse. It isn't. Every token you add competes with every other token for the model's attention, and past a point, more context makes reasoning worse, not better. Nothing overflowed. Nothing got truncated. The model just stopped listening to the part that mattered.

The incident

I had an agent doing a narrow job: generate a CRUD endpoint from a short instruction. Give it "all list endpoints must paginate, max page size 50" and it nailed the constraint every single time. So — "for completeness" — I started feeding it the full 40-page product spec alongside that same instruction. More context, more understanding of the system, right?

Wrong. The generated endpoint came back with zero pagination — `SELECT * FROM orders WHERE user_id = ?`, returned as one unbounded array. The pagination rule was sitting right there in the spec, verbatim, page 12. Nothing was cut off. Nothing was truncated. The instruction was just... not acted on.

What actually happened (it's not what you think)

• ▹Not overflow — the spec plus instructions fit well inside the context window, with plenty of room to spare.
• ▹Not forgetting — the text was physically sitting in the prompt, unmodified, the entire time.
• ▹What actually happened: attention got diluted. Forty pages of feature descriptions, edge cases, and background rationale were all competing for the model's focus, and the one line that mattered got the exact same weight as every other line. And lost.

Mental model: a table, not a warehouse

Stop picturing the context window as a warehouse where you stack boxes and only worry about running out of shelf space. Picture a table you're talking across instead. Every document you drop on that table competes for eyes — the reader's attention, not square footage. A warehouse with 500 boxes is fine as long as there's room. A table with 500 documents spread across it means the one memo that actually matters gets glanced at, not read.

That's why the industry calls it "attention," not "memory." A transformer literally computes a weighted attention score across every pair of tokens. Add more tokens and you're not adding shelf space — you're adding more competitors for the same fixed attention budget, per layer.

The tell that separates this from truncation

Here's how you tell dilution apart from a length or truncation bug: ask the model to find and quote the instruction back to you.

You: What does the spec say about pagination for list endpoints?

Agent: "All list endpoints must paginate, max page size 50."

You: Then why did the endpoint you just wrote return everything unpaginated?

Agent: You're right, that's a bug — let me fix it.

(...regenerates code, ignoring the constraint again on the next unrelated request)

If the model can quote the buried instruction word-for-word on demand but doesn't spontaneously apply it while reasoning through the actual task, that's not truncation — the text hasn't gone anywhere — and it's not a comprehension failure either, since it clearly understood the sentence. It's an allocation failure. The instruction lost the fight for attention at generation time, even though it wins easily the moment you point a spotlight straight at it.

Why '1M tokens!' doesn't fix this — and can make it worse

Bigger context windows get marketed like bigger storage tiers, so the instinct is: this is a capacity problem, buy more capacity. But if the failure mode is dilution and not overflow, widening the window just gives you a bigger table with the same number of eyes reading it. You haven't fixed the signal-to-noise ratio — in practice you've often made it worse, because now it's even cheaper to dump in irrelevant material "just in case," diluting attention further on what actually matters. Researchers have documented this exact failure as "lost in the middle" behavior, and more recently under a broader name: "context rot" — model quality on a task degrading as irrelevant context grows, regardless of whether the window technically has room.

For anyone building agents, this shows up constantly: RAG pipelines that retrieve 20 chunks "to be safe," multi-agent handoffs that forward the entire conversation history instead of a summary, tool-calling agents that keep every previous tool result sitting in context. All of it is the same mistake I made with that 40-page spec — trading precision for the illusion of completeness.

Today's practical takeaway: a one-test diagnosis

Before you conclude "the model can't handle this task," run this test to tell a context-length problem apart from a context-quality problem:

• ▹If it fails even on the minimal prompt: you've got a real capability or length problem — the model genuinely can't do the task with what it's given.
• ▹If it succeeds minimal but fails once you add everything back in: you've got a dilution problem, and the fix is curation, not a bigger window.
• ▹Rule of thumb while prototyping agents: every time you're tempted to add a document "for completeness," ask whether it's on the table because the task needs it — or because it was just lying around and convenient to include.

Bridge to Day 3

If attention is the scarce resource, the real question isn't "how much context can I fit" — it's "what do I do about the fact that it's scarce." That's tomorrow: the actual techniques — curation, structure, summarization, retrieval design — for spending an attention budget instead of just filling one.

Extend your knowledge

• ▹Read 'Lost in the Middle: How Language Models Use Long Contexts' (Liu et al.) — the original empirical study showing accuracy drops for information buried in the middle of long contexts.
• ▹Look up Chroma's 'Context Rot' research report — it documents how retrieval and generation quality degrade as irrelevant context volume grows, independent of window size.
• ▹Try a needle-in-a-haystack style eval (the long-context recall test popularized by Greg Kamradt and since adopted industry-wide, Anthropic included) on your own agent's real prompts — not a synthetic benchmark — to see exactly where your pipeline starts dropping instructions.
• ▹Audit one agent or RAG pipeline you're running today: count how many retrieved chunks or forwarded messages are actually load-bearing for the current task, versus included 'just in case.'