Knowledge decay

Day 22: Knowledge Decay — Teaching Retrieval That Some Documents Are No Longer True

Your retrieval can find exactly the right document and still hand your agent the wrong answer. The document was correct once. Then the team changed its mind, and nobody told the index. If you don't model time and supersession, your coding agent will keep bringing back decisions you've already reversed. It will also cite your own docs as its justification.

The failure that started this

An agent opened a PR on one of our services. It added a pattern we had overturned months earlier, and the PR description cited our own ADR as the reason. Nobody had deleted that ADR or marked it superseded. The reversal lived in a Slack thread and in the heads of three engineers. Retrieval did exactly what it was built to do. The query matched, and the old ADR was the most similar and best-written document in the corpus. The answer was still wrong.

Nothing in the retrieval stack was broken. What it lacked was an idea, and that idea is today's lesson.

Name the failure: relevance is not truth

Similarity search answers one question: which chunks look most like this query? Time doesn't come into it. The embedding of a carefully written 2024 ADR has no way of knowing that a quick 2026 Slack decision made it obsolete. In practice the polished old document usually wins. It's longer, better structured and full of the exact vocabulary the query uses. The new decision is three lines of shorthand in a chat channel that probably isn't indexed.

Think back to Day 21. We tuned hybrid search and added a cross-encoder re-ranker to get better at 'most similar'. Each of those steps made the system better at finding the old ADR, and none of them checked whether it was still valid. Ranking quality and truthfulness are separate axes. Each one needs its own signal.

• ▹Relevance: is this chunk about what the query is about? Embeddings, BM25 and re-rankers handle this.
• ▹Validity: is what this chunk says still what we believe? That takes status, supersession, review dates and ownership.
• ▹Most RAG pipelines only model relevance. Today we add validity.

Why agents make this worse than humans do

Teams put a lot of effort into capturing knowledge, but reversals do more damage than gaps. A human who opens an old Confluence page sees the 'last edited 2 years ago' banner, the author who has since left, and screenshots of a UI that no longer exists. They get suspicious and ask someone. An agent sees none of that. The chunk arrives without its page chrome and sits in the context window with the same authority as everything else around it, and the model acts on it.

HUMAN READING THE OLD ADR                AGENT RETRIEVING THE OLD ADR
-------------------------------------    -------------------------------------
1. Search wiki: 'data access layer'      1. Query: 'how do we structure data access'
2. Opens ADR-014 (2024)                  2. Top chunk: ADR-014 section 3 (sim 0.89)
3. Notices: author left, page is old     3. No date, no author, no status in chunk
4. Thinks: 'is this still how we do it?' 4. Treats chunk as ground truth
5. Asks in #backend                      5. Generates repository classes per ADR-014
6. Learns it was reversed in ADR-031     6. Opens PR: 'Follows ADR-014'
                                         7. Reviewer rejects; 20 min lost, trust lost

Outcome: 5 min detour, correct work      Outcome: confident, cited, wrong

The human's doubt is acting as a validity check, and it comes for free. An agent has no doubt unless you give it some, and the only way to do that is through data. You can attach metadata to the chunk, write the reversal down explicitly, or make sure the stale chunk is never retrieved.

The four ways knowledge goes stale

There are several kinds of staleness. Each leaves a different trace, so you detect each one differently.

• ▹Reversed decisions. A later decision overturns an ADR or design doc. Signal: a newer decision exists on the same topic. Fix: an explicit superseded_by link and a graveyard entry. This kind is the most dangerous because the old doc is usually well written and reads as authoritative.
• ▹Drifted runbooks. The doc doesn't change, but the system underneath it does. Commands pick up new flags, dashboards move, a step stops being needed. Signal: the code, infra or config the doc references has changed since the doc was last reviewed. Fix: review_by dates, and a check of the doc's review date against recent commits to the paths it mentions.
• ▹Renamed systems. 'billing-svc' became 'payments-core' and half your docs still use the old name. Signal: terms that show up in docs but no longer appear in the codebase or service catalog. Fix: an alias map applied at query time, plus bulk edits. Agents handle this badly and will happily create a new module under the old name.
• ▹'Temporary' workarounds that stayed. 'For now, restart the worker manually if the queue stalls.' Two years later the workaround is either the real process or long gone, and nobody knows which. Signal: phrases like 'temporary', 'for now', 'workaround', 'until we', and TODOs pointing at closed tickets. Fix: require a short review_by on any doc that describes a workaround.

Pattern 1: Freshness metadata on every chunk

Start with the boring step, because it does most of the work. Every document carries machine-readable validity metadata. Chunks inherit it from their parent doc at indexing time, which makes it available as a filter field in your vector store. Keep the schema small, since every field you add is one more field people will leave blank.

---
# Minimal freshness frontmatter. Put it at the top of every doc the agent can retrieve.
id: adr-014
title: Repository layer over the ORM
status: superseded        # active | superseded | deprecated
owner: team-backend       # a team or a person; a team survives reorgs better
review_by: 2026-12-01     # when someone must re-confirm this is still true
superseded_by: adr-031    # required when status is superseded
last_reviewed: 2026-06-10 # optional but useful for ranking and audits
---

• ▹status: active means 'we believe this'. superseded means 'replaced by something specific'. deprecated means 'we no longer trust this and nothing has replaced it yet'.
• ▹owner is who gets pinged when review_by passes. Use a team alias over a single person where you can.
• ▹review_by is the date by which someone has to confirm the doc is still true. Runbooks and workarounds get short windows (about 3 months). Architectural decisions can go 6 to 12 months.
• ▹superseded_by is a pointer, not a description. The retrieval layer follows it to fetch the replacement.
• ▹Confluence and Notion pages have no frontmatter, so store the same fields as page properties or labels and map them in your ingestion job. What matters is the schema, not where it's stored.
• ▹If you write ADRs, Nygard's original format already has a Status field, and tools like adr-tools will mark the old ADR superseded when you create its replacement. Use them.

Pattern 2: The graveyard file

Metadata only helps if someone updates the old doc, and often nobody does because the reversal happened in a meeting or a chat thread. The graveyard file handles that case. It's a single document listing what the team no longer believes, why, and what replaced it. It is not retrieved by similarity. It gets loaded into the agent's context ahead of retrieval results, either referenced from CLAUDE.md or AGENTS.md or injected by your RAG layer as a fixed preamble.

# GRAVEYARD.md: things we no longer believe

> Agents: read this before acting on any retrieved document.
> If a retrieved doc conflicts with an entry below, the entry below wins.
> If you are about to produce code matching a 'Tell-tale signs' line, stop and flag it.

## Repository layer over the ORM (reversed 2026-08)
- Was: ADR-014. Wrap every model in a repository class.
- Now: Call the ORM from the service layer. See ADR-031.
- Why: Double abstraction; every new table produced boilerplate with no behaviour.
- Tell-tale signs: new files under src/repositories/, classes named *Repository.

## Manual worker restart on queue stall (retired 2026-05)
- Was: Runbook step 4 in 'Queue incidents'.
- Now: Auto-restart via health check. Manual restart masks the real alert.
- Why: The workaround outlived the bug it covered.
- Tell-tale signs: instructions telling someone to run a restart command by hand.

## billing-svc (renamed 2026-03)
- Was: billing-svc
- Now: payments-core. Never create new modules, configs or dashboards with the old name.

Teams keep this file current because it costs almost nothing to maintain. An entry takes two minutes. You don't need the original author's permission and you don't have to rewrite the old doc first. The engineer who rejected the agent's PR can add the entry before they've closed the tab. The file is meant to stay short. Once an entry is reflected in the metadata and the old doc has been cleaned up or deleted, remove it from the graveyard. If you let the graveyard grow forever, it becomes one more stale corpus.

• ▹Keep it to roughly one or two pages so it fits in every context without crowding out the task.
• ▹For coding agents, the 'Tell-tale signs' line is the most useful part. It turns an abstract reversal into a pattern the model can spot in its own output.
• ▹Put the newest entries first, because recent reversals are the ones indexed docs are most likely to contradict.

Pattern 3: Supersession-aware retrieval

Metadata is useless if the retrieval layer ignores it. The rule is that superseded content never reaches the context window looking like current truth. When a superseded chunk matches a query you have two choices: drop it entirely, or replace it with a short stub saying it was reversed and pointing to the replacement. The stub is often better. It tells the agent the topic has history, and since the superseded chunk matched, the replacement almost certainly belongs in context as well.

Use review_by as a ranking penalty, not a hard cutoff. An overdue doc is probably still true; nobody has re-confirmed it yet. If you cut overdue docs outright, a team that falls a month behind on reviews loses half its corpus overnight.

from datetime import date

def retrieve(query, k=8):
    # Over-fetch: filtering will remove some candidates.
    candidates = search(query, k=k * 4)
    results, stubs, seen = [], [], set()

    for c in candidates:
        m = c.metadata

        if m.status == 'superseded':
            # Never pass the old content through. Leave a warning instead.
            stubs.append(f"[{m.id} was reversed. Current guidance: {m.superseded_by}]")
            # The topic matched, so the replacement is relevant too.
            if m.superseded_by and m.superseded_by not in seen:
                for r in fetch_doc_chunks(m.superseded_by, query, top=2):
                    results.append((r, r.score))
                seen.add(m.superseded_by)
            continue

        score = c.score
        if m.status == 'deprecated':
            score *= 0.5                      # still visible, rarely wins
        if not m.owner:
            score *= 0.7                      # unowned = unverified
        if m.review_by and m.review_by < date.today():
            overdue_days = (date.today() - m.review_by).days
            score *= max(0.6, 1 - overdue_days / 365)   # soft decay with a floor

        results.append((c, score))

    results.sort(key=lambda x: x[1], reverse=True)
    top = [c for c, _ in results[:k]]
    return dedupe(stubs)[:3] + top

# The multipliers are starting knobs, not benchmarks.
# Tune them against your own eval set from earlier in this series.

• ▹Filter in the vector store where you can, since most stores support metadata filters like status != superseded. The stub-and-fetch step belongs in your application layer.
• ▹Apply freshness after the re-ranker, or pass it in as a re-ranker feature. If you apply it before, the re-ranker will undo it.
• ▹Put status and date into the chunk text the model sees, with a header line like '[ADR-031 | active | reviewed 2026-09]'. That gives the agent back some of the cues a human gets from page chrome.
• ▹Log every stub you emit. If one topic produces a lot of stubs, the old doc should be rewritten or deleted (see Pattern 4).

Pattern 4: Deleting is a retrieval feature

The change that helped us most was deleting documents. We pruned about half our context files: superseded ADRs whose reversal was already in the graveyard, runbooks for retired systems, duplicated onboarding notes, and old CLAUDE.md fragments that contradicted each other. Contradicting chunks showed up in the agent's context less often, and fewer agent PRs were rejected in review. The agent did better once it had less to read.

That goes against the instinct that more context is always better. The instinct holds when the extra context is consistent. Once a corpus contains contradictions, every extra document is another chance for the wrong version to rank high. If both versions land in context, the model has to pick one without any reliable signal for which is current. Published research supports this, at least in general terms. Liu et al.'s 'Lost in the Middle' showed that models use information unevenly across long contexts, and Chroma's 'Context Rot' report documented performance falling as input length grows. Contradictions make this worse because the extra text actively argues for the wrong answer.

• ▹Delete a doc when it's superseded and the replacement fully covers it. If the old pattern is still likely to creep back, keep a one-line graveyard entry.
• ▹Archive it outside the index, instead of destroying it, if you need it for audit or history. The rule is 'not retrievable'; it can still be stored.
• ▹Merge two active docs that cover the same topic. With two near-duplicates, one of them will drift.
• ▹Keep a doc that's old but still true. Age on its own is no reason to delete. Being unowned and unreviewed is.

Ownership and the expiry loop

Metadata goes stale too if nothing enforces it. A small scheduled job does the enforcing. Once a week it finds entries past review_by and pings each owner with four options: re-confirm, update, supersede or delete. Entries with no owner are set to status: deprecated automatically.

# weekly_freshness.py, run from CI on a schedule
from datetime import date, timedelta

for doc in load_all_docs():
    m = doc.metadata

    if not m.owner:
        if m.status == 'active':
            set_status(doc, 'deprecated', reason='no owner')
            notify('#kb-hygiene', f'{m.id} deprecated: no owner. Claim it to restore.')
        continue

    if m.review_by and m.review_by < date.today():
        notify(m.owner, f'{m.id} is past review_by ({m.review_by}). '
                        'Re-confirm, update, supersede, or delete.')
        # Escalate if ignored for a long time
        if m.review_by < date.today() - timedelta(days=60) and m.status == 'active':
            set_status(doc, 'deprecated', reason='review ignored 60d')

# Re-confirming = bump review_by + last_reviewed in one small PR.
# Make that a one-click action or nobody will do it.

'No owner means deprecated' works because the two possible mistakes cost very different amounts. If you deprecate a doc that's still true, it gets down-ranked until someone notices and claims it, and then it's back. That costs a few minutes. If you keep an unowned stale doc active, it can ship a reversed pattern to production with an ADR citation attached. The default also makes ownership visible and cheap: claiming a doc keeps it alive. And because deprecation down-ranks a doc without deleting it, the default is safe to automate.

Audit your own KB tomorrow: five checks

You can start without the full pipeline. Run these checks against your corpus to find out how big the problem is.

# 1. Docs with no owner
grep -rL '^owner:' docs/ --include='*.md' | wc -l

# 2. ADRs without a status field
grep -L '^status:' docs/adr/*.md

# 3. 'Temporary' language that may have quietly become permanent
grep -rniE 'temporar|for now|workaround|hack|until we|stopgap' docs/ --include='*.md'

# 4. Contradicting pairs: see the Python sketch below

# 5. Age of what the agent actually sees (needs retrieval logs)
#    For each logged query, take the top-3 chunks and record their last_reviewed
#    or last-modified date. Look at the median and the oldest.

# Check 4: find likely contradictions.
# Near-duplicate topics + different dates or decisions = review candidates.
pairs = []
for a, b in high_similarity_pairs(all_chunks, threshold=0.85):
    if a.doc_id == b.doc_id:
        continue
    pairs.append((a, b))

for a, b in pairs[:200]:
    verdict = llm_judge(
        'Do these two passages give conflicting guidance on the same topic? '
        'Answer CONFLICT, CONSISTENT, or UNRELATED, then one sentence why.',
        a.text, b.text,
    )
    if verdict.startswith('CONFLICT'):
        print(a.doc_id, '<->', b.doc_id, verdict)

• ▹Check 1 shows how much of your corpus the expiry loop will deprecate on day one. Expect a sobering number.
• ▹Check 2 finds the ADRs most likely to come back the way ours did.
• ▹Check 3 is noisy, but every real hit is either something to fix or a doc to delete.
• ▹Check 4 finds the contradictions that hurt agents most. Each confirmed pair should end up as a supersession link or a deletion.
• ▹Check 5 shows whether retrieval actually favours old documents. If the median top result is years old on a codebase that changes every week, start with Pattern 3.

Bridge to Day 23

Today covered recognising what's no longer true and keeping it away from the agent. The other half of the problem is where new truth comes from. The reversal behind our bad PR was never written down anywhere retrievable, and the best moment to capture it was when the reviewer corrected the agent. Day 23 is about turning those corrections into knowledge as they happen, so the graveyard and the active docs grow out of everyday work rather than documentation sprints.

Extend your knowledge

• ▹Read Michael Nygard's original 'Documenting Architecture Decisions' post, then try the supersede option in adr-tools. ADRs already have the status model this lesson builds on.
• ▹Anthropic's engineering post 'Effective context engineering for AI agents' makes the broader case for curating context instead of maximising it.
• ▹Liu et al., 'Lost in the Middle: How Language Models Use Long Contexts', and Chroma's 'Context Rot' report explain why adding documents can make answers worse.
• ▹Hands-on: add status/owner/review_by to your ten most-retrieved docs, write a five-entry GRAVEYARD.md, reference it from your CLAUDE.md or AGENTS.md, and re-run your Day 21 eval set to compare.