Day 16: Lost in the Middle, or why your long paste didn't count
You pasted the whole thing: the log, the spec, the 2,000-line file. The one rule that mattered was in there, and the model can even quote it back to you. Then it breaks that rule anyway, and it doesn't mention that anything went wrong. That's usually not you phrasing it badly, and the model isn't being dumb. It's position. Where something sits in a long paste changes how much it counts, and once you can see that, you can fix it with layout.
The migration-table story
We gave an agent a paste of about 2,000 lines: schema notes, migration history, team conventions. Around line 900 sat the rule that mattered. On the big tables, you never rename a column in place. You add the new column, backfill it, then drop the old one, and each step is its own migration.
We asked, "what does the convention around line 900 say?" It quoted the rule word for word. Then we asked it to write the migration, and it renamed the column in place.
This kept happening. At PhoenixDX we lost about a week to failed agent runs, and every one of them traced back to the same habit: paste everything, put the ask at the bottom, and assume that because it's in there, it'll get used.
The question everyone secretly has
"I included it, so why didn't it count?" Most people blame their wording or the model. Usually it's neither. You're looking at two states that look exactly the same from the outside:
- It's in the context. The text was sent, the model can see it, and it could quote it if you asked.
- It shaped the answer. The text actually changed what the model produced while it was doing the work.
- Your chat window only ever shows you the first one. You only learn about the second when the output breaks the rule.
The one mental model you need: attention is a U-shape
No maths needed. Picture your paste as a building the model walks through. It pays close attention at the front door (the start) and again on the way out (the end). In the long hallway in between, it barely looks around. Instructions left in the hallway are the ones that get politely ignored.
Researchers call this "lost in the middle", after the 2023 paper by Nelson Liu and colleagues. To be precise, the paper measured answer accuracy depending on where the relevant information sat, not attention weights, but the U-shape is the same. For daily work, two things matter. It gets worse as the paste gets longer. And it shows up long before you hit the advertised context window. A model that accepts a very long input doesn't use all of it equally.
Why the benchmarks reassure you wrongly
Long-context models usually get sold on "needle in a haystack" scores. The test hides one odd sentence in a huge document and asks the model to find it. Modern models do very well at this, and that's why people trust a big paste. The catch is that the test measures one skill and your work depends on another:
- Recall: "Find the line that says X." Needle tests measure this, and it's the number vendors advertise.
- Use: "Apply the rule on page 3 while you change something on page 40." Your migration, your contract review and your log triage all depend on this one.
- In the migration story the agent passed recall and failed use. That isn't a contradiction. The two tests measure different things.
- Newer benchmarks like RULER and NoLiMa exist partly because simple needle tests make models look better at using long context than they really are.
Why we keep falling into it
Mostly it comes down to how we write. The model's architecture is a smaller part of it:
- We write prompts like documents: background, then details, then the question at the bottom. A human reader would thank you for that. A model reading a long paste won't.
- When you paste first, the critical constraint ends up wherever it happened to be in the source, which is usually somewhere in the middle.
- Bigger windows make it worse. If everything fits, nobody feels any pressure to edit, so pastes get longer and so does the hallway.
- Agents make it worse again. A coding agent that reads ten files and a long tool output builds an enormous middle, and your original instruction ends up a long way from both ends.
The fix: Ask → Evidence → Ask again
A bigger model or a bigger window won't fix this. Layout will, and the layout has three parts:
- TOP: the task and the non-negotiables, in your own words.
- MIDDLE: the log, file, spec or transcript, clearly labelled as reference material.
- BOTTOM: the task again, plus the one constraint that must not break.
The vendors' own guidance agrees. OpenAI's GPT-4.1 prompting guide recommends putting instructions at both the beginning and the end of long context. Anthropic's long-context tips recommend putting long documents above the query, so the ask comes after the material. Here's the same prompt both ways. The words don't change. Only the order does, plus one repeated line at the bottom.
=== BEFORE (document-style: the constraint is buried) ===
Here's our schema notes and migration conventions for context.
[ ~2,000 lines pasted ]
...
line ~900: Large tables (orders, events): never rename a column in place.
Use add-new -> backfill -> drop-old, as separate migrations.
...
Write a migration that renames orders.cust_id to orders.customer_id.=== AFTER (Ask -> Evidence -> Ask again) ===
TASK: Write a migration that renames orders.cust_id to orders.customer_id.
NON-NEGOTIABLE: Large tables (orders, events): never rename a column in place.
Use add-new -> backfill -> drop-old, as separate migrations.
<reference: schema notes and migration conventions>
[ the pasted material, trimmed to the relevant sections ]
</reference>
TASK (repeated): Write a migration that renames orders.cust_id to orders.customer_id.
Before you answer, check it against this rule: orders is a large table,
so no in-place rename. Use add-new -> backfill -> drop-old, as separate migrations.Three small habits that go with it
- Trim the paste to the part that matters. If you can't say which 200 of the 2,000 lines matter, the model can't either. Trimming is half the thinking, not a shortcut.
- Label your sections: <logs>, <spec>, <contract section 4>, or just "REFERENCE ONLY". Then you can point the model at the middle ("use the rule in <conventions>") and stop hoping it notices.
- Pull critical rules out of the paste and restate them in your own words at the top. The original stays in the paste as evidence. Your restated version is the instruction.
When not to bother: for short prompts, say a paragraph or two plus a small snippet, the U-shape barely matters, so write however feels natural. Switch to this layout once the paste is long enough that you'd have to scroll to read it, or when an agent is piling up context over many steps.
The 2-minute test to run on your own context
This tells you whether your layout is the problem, so you don't have to argue about which model is smarter:
- 1. Pick a rule that sits in the middle of your usual paste, or plant one. For example: "All dates in output must be ISO 8601."
- 2. Don't mention that rule anywhere else in the prompt.
- 3. Give a task that should trigger it, like "summarise the incidents with their dates."
- 4. Check the output. If the rule was broken, your layout lost it.
- 5. Rerun with Ask → Evidence → Ask again. If the rule holds now, the layout was the fix. Keep this as a reusable check for your team's standard prompts and agent instructions.
- One trap: don't make step 3 "what does the date rule say?" That only tests recall, and recall was never the problem.
Looking ahead
The same middle-loss happens between agents. When one agent hands work to another, the constraint that mattered tends to get buried in the handoff. That's the next lesson.
The takeaway
When our regressions stopped, it wasn't because the model got smarter. We'd just stopped putting the important thing in the one place it doesn't look. The rule: ask first, evidence in the middle, then the ask again at the end with the one constraint that must not break. After that, test whether the model actually uses a rule from the middle when nobody reminds it.
Extend your knowledge
- Read "Lost in the Middle: How Language Models Use Long Contexts" by Nelson F. Liu et al. (2023), the paper that named the effect. You can skip the methods section and the charts still make sense.
- Look up the RULER benchmark (NVIDIA) and NoLiMa to see how researchers test whether models actually use long context, not just find a needle in it.
- Open the long-context sections of Anthropic's prompt engineering docs and OpenAI's GPT-4.1 prompting guide side by side and compare where each one puts instructions relative to the documents.
- Exercise: take one prompt or agent instruction file your team reuses every week, run the 2-minute test on it, then rewrite it as Ask → Evidence → Ask again.