Your Backfill Job Said 100% Success. Your Index Disagrees.

Day 14: Backfilling Without Breaking Recency

Your backfill job just finished. Green across the board — no errors, no nulls, row count matches the source exactly. Ship it, right? Except a week later someone asks your RAG agent 'what's our current refund policy?' and it answers with total confidence, citing a policy that was superseded four months ago. That's not a hallucination. The agent read it straight off a chunk your own index had labeled 'current.' The job succeeded. The index lied.

The backfill that lied

Walk through it slowly. You've got six months of historical policy docs to load into an index that's been live since Day 1. You run the backfill. It finishes clean. Every check you'd normally trust — exception count, null count, source-vs-ingested row count — comes back perfect. And still, somewhere in that index, an old document is wearing the 'current' badge that belongs to a newer one. Nothing in the job output tells you that. You find out when a user does.

Name the mechanism: it's not what got ingested, it's the order

Here's the part that trips people up: most backfill jobs walk historical data in whatever order the source happens to hand it over — file listing order, S3 key order, database pagination, alphabetical filenames. None of that is chronological. But your index's 'latest version wins' logic is quietly assuming it always is. Feed it out of sequence and a March document can steal the 'current' pointer away from a July document, purely because March's filename sorted after July's in the listing. The index isn't broken. It's doing exactly what you told it to do — trust arrival order — and arrival order just stopped meaning anything.

Why Days 1-13 never caught this

The incremental pipeline you built earlier in this series only ever sees new documents as they show up, a few at a time, and new documents always arrive after everything already in the index — that's what 'incremental' means. Document N+1 being older than document N was never a scenario your system had to survive, so it never did. Backfill is the first time you hand it a large, unordered, non-monotonic batch. It's the first real stress test of an ordering assumption you baked in on day one and never once exercised.

The fix: backfill is a replay, not a separate code path

• ▹Pin down an authoritative timestamp per document — last_modified, effective_date, a version field from the system of record. Not file mtime — that tells you when a copy was exported, not when it was true.
• ▹Sort or partition the entire backfill input by that timestamp before a single ingest call fires.
• ▹Route it through the exact same versioning/supersession logic live ingestion uses. Do not write a 'backfill mode' shortcut. That branch is where this bug goes to hide.
• ▹Verify with a query, not a row count. Pick 3-5 documents you know were superseded, ask the index about that topic, and confirm the current version — not the old one — comes back on top.

# wrong: discovery order
for doc in list_files(bucket_prefix):
    ingest(doc)  # index sees docs in S3-key order, not time order

# right: explicit chronological replay through live ingest logic
docs = list_files(bucket_prefix)
docs.sort(key=lambda d: d.authoritative_timestamp)
for doc in docs:
    ingest(doc)  # same function live ingestion calls -- no backfill-only branch

# verification: query-based, not count-based
for known_superseded_id, known_current_id in spot_check_pairs:
    result = index.query(topic_of(known_current_id))
    assert result.top_hit.doc_id == known_current_id, \
        f"stale doc {known_superseded_id} outranking current {known_current_id}"

The agent trap

Hand a coding agent the ticket 'backfill the missing six months of docs' and it will optimize for precisely what you wrote down: job completes, no exceptions, no nulls, source count equals ingested count. Ordering isn't in that list, so it isn't in the agent's model of 'done.' It has no way to infer that your index carries implicit temporal semantics just by reading the ingestion function's signature — that context lives in your head, not the code. Fixing this isn't about a smarter agent. It's about what you specify.

• ▹Write the ordering invariant into the task itself: 'sort input by authoritative timestamp before ingest; reuse the live supersession path; no separate backfill branch.'
• ▹Make it prove correctness before it reports done: 'pick 5 known-superseded/current doc pairs, query the index, confirm the current one wins.' Done means passed a check, not exited zero.

Checklist for tomorrow

• ▹Order source: which timestamp field is driving replay order — and is it authoritative (business time), or just incidental (file mtime, export time)?
• ▹Supersession reuse: does backfill call the exact same versioning code live ingestion uses, with zero backfill-only branches?
• ▹Recency spot-check: have you run at least one query against a known superseded/current pair and watched the right one win?
• ▹Rollback plan: if the spot-check fails after the job already ran, can you replay idempotently, or are you now looking at a point-in-time restore of the index?

Tomorrow (Day 15) we pick up the rollback question left hanging above: what an idempotent, re-runnable ingestion pipeline actually requires, and why 'just re-run the job' is a trap the moment your ingest wasn't built for it.

Extend your knowledge

• ▹Look at how Flink distinguishes event-time from processing-time using watermarks, and how Kafka Streams handles the same problem via stream-time tracking and grace periods — the same event-time-ordering discipline applies directly to index backfills, just without the streaming infrastructure.
• ▹Read up on Kafka log compaction as a mental model for 'latest key wins' semantics, and notice how compaction explicitly depends on offset order, which is the same invariant your index's supersession logic depends on.
• ▹If your index is backed by a vector DB with metadata filtering, check whether it exposes a way to filter/boost by an effective_date field at query time as a second line of defense against ordering bugs, independent of ingest-time correctness.
• ▹Practice writing the 'ordering invariant + verification query' pattern into your next agent-delegated pipeline task — treat it as a template you reuse anytime you hand an agent a data-processing job with implicit temporal assumptions.