Lineage for agentic pipelines

Day 30: Lineage for Agentic Pipelines: Walking a Wrong Answer Back to Its Source

All 400 spans are green and the answer is still wrong. Nothing in your logs can tell you which chunk produced the bad sentence. A RAG or multi-agent system is only fixable when you can name the input that caused the failure: the chunk, the index and the prompt behind that sentence. Without that, you're guessing. On-call work needs evidence.

The wrong answer, the perfect trace, and the question the trace can't answer

You'll hit this one sooner or later. I have. A user reports a wrong answer, and you open the trace in Langfuse, Phoenix, LangSmith or whatever your team uses. It's all there: 400 spans, token counts, latencies, the retriever call, the reranker call, three agent turns and the final completion. Every span is green and nothing errored. You scroll to the bottom and find that not one of those spans says which chunk the bad sentence came from.

The trace tells you retrieval returned 20 chunks and the reranker kept 8. It tells you Agent A passed something to Agent B and Agent B wrote the answer. It doesn't tell you which of those 8 chunks produced the third sentence. It doesn't tell you whether that chunk came from the current version of the document, or whether Agent B's summary changed the meaning on the way through. This lesson closes that gap.

Tracing records events. Lineage records causes.

A trace is a timeline: this happened, then this, and it took this long. Lineage is a dependency graph: this output was derived from these inputs by this transform. Classic data lineage (OpenLineage, the dbt DAG, column-level lineage in your catalog) builds that graph for tables and answers "which upstream table fed this column". For agents, that's the wrong level of detail. A multi-agent system rarely fails because a column was wrong. It usually fails at a context handoff. The wrong chunk got retrieved, or a stale index served it, or a prompt version framed it badly. Or an upstream agent's summary distorted it before the next agent built on it. So the unit you track is the unit of context: a chunk, a summary, a tool result, a claim. Tables are too coarse.

The last 29 days, redrawn as a lineage graph

Every part of this series added a node that can change the final answer without anyone noticing. Here they are on one line:

• ▹Source document (Days 1-5): the same URI can hold different content after a revision, so a URI alone doesn't identify the content. Pair it with a content hash.
• ▹Chunker version (chunking days): a new splitter moves chunk boundaries. A fact that used to sit next to its heading can end up in a separate chunk.
• ▹Embedding model (embedding days): switch models and every vector moves, even though nobody edited a word.
• ▹Index snapshot (vector-store days): the index is a build artifact, and it can be weeks older than the documents it claims to represent.
• ▹Retriever and reranker config: change top-k, a hybrid weight or the reranker model, and a different set of chunks reaches the prompt.
• ▹Prompt template version: the same chunks under a different instruction can produce a different claim.
• ▹Agent handoff: whatever an upstream agent outputs becomes ground truth for the next one. Any summarisation error it introduced gets passed along as if it were fact.

Any of these nodes can be the root cause. Lineage means recording, for every piece of context, which version of each node it passed through.

The core design: a provenance ID on every unit of context

Keep the schema small on purpose. Each record describes one unit of context and points at the context it was derived from. Documents are the roots. Chunks are their children, summaries are children of chunks, and claims in the final answer are children of whatever they cite. That gives you a DAG.

• ▹context_id: a stable unique ID, such as ctx_ followed by a UUID.
• ▹parent_ids: the context_ids this unit was derived from. Empty for source documents.
• ▹source_uri + content_hash: where the root came from and exactly which bytes. Every record gets a hash, not just the roots.
• ▹transform + transform_version: what produced it (chunker, embedder, reranker, summariser, llm_call) and which version, e.g. chunker@2.3.1.
• ▹produced_by: for anything an LLM generated, which agent and which prompt version.
• ▹index_snapshot: for retrieved chunks, the snapshot ID that served them.
• ▹trace_id: the join key back to the traces you already have.

Two rules hold the design together. Teams usually skip the second one. Rule 1: no context enters a prompt without an ID. Rule 2: no agent output leaves without listing the IDs it consumed. If you break either rule, the graph has a hole, and that's where you'll want to look during an incident.

import hashlib, uuid
from dataclasses import dataclass

@dataclass(frozen=True)
class Context:
    context_id: str
    text: str
    parent_ids: tuple[str, ...]

def sha256(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

def derive(store, parents, text, kind, transform, version,
           produced_by=None, index_snapshot=None, source_uri=None,
           trace_id=None) -> Context:
    """The only way new context is created. Writes an append-only provenance record."""
    ctx = Context(
        context_id=f"ctx_{uuid.uuid4().hex}",
        text=text,
        parent_ids=tuple(p.context_id for p in parents),
    )
    store.append({
        "context_id": ctx.context_id,
        "parent_ids": list(ctx.parent_ids),
        "kind": kind,                        # document | chunk | retrieval | summary | claim | answer
        "content_hash": sha256(text),      # hash, not the text
        "transform": transform,              # chunker | embedder | reranker | summariser | llm_call
        "transform_version": version,
        "produced_by": produced_by,          # e.g. "agent_b/summarise@v7"
        "index_snapshot": index_snapshot,
        "source_uri": source_uri,
        "trace_id": trace_id,
    })
    return ctx

def render_for_prompt(contexts: list[Context]) -> str:
    # Rule 1: every block the model sees carries its ID.
    return "\n".join(
        f'<context id="{c.context_id}">\n{c.text}\n</context>' for c in contexts
    )

Make derive() the only constructor for context in your pipeline. Then the chunker, the summariser and every agent have to declare their parents, because they can't create context any other way. Retrieved chunks already carry IDs from indexing time. At query time, you add one record per retrieval that captures the index snapshot and the reranker config that served them.

Make the LLM cite its inputs, then don't fully trust it

The last hop goes from a sentence in the answer to the context behind it, and only the model can supply it, because only the model knows what it relied on. Ask for structured output where every claim lists the context_ids that support it. Inline markers like [ctx_17] also work, but they're harder to parse reliably. Some providers have built-in citation features that return spans of the documents you supplied. Anthropic's Citations is one. Use them where they fit.

{
  "claims": [
    {
      "text": "Contractors are entitled to 30 days' notice.",
      "supported_by": ["ctx_b_summary_9"]
    }
  ]
}

Models misattribute. They cite the chunk that sounds most related instead of the one they actually used. They make up IDs that look plausible. They leave claims uncited. So treat a self-citation as a hint and check it. Two checks are cheap. First, were the cited IDs actually in the prompt? That's a set membership test and costs nothing. Second, does the cited context entail the claim? That's the entailment or LLM-grader setup from the evals day, run one claim at a time.

def verify_claims(answer, prompt_context_ids: set[str], resolve_text, judge):
    """resolve_text(id) fetches text from the chunk store / trace, not from provenance."""
    problems = []
    for claim in answer["claims"]:
        cited = claim["supported_by"]
        if not cited:
            problems.append(("uncited", claim["text"], []))
            continue
        unknown = [c for c in cited if c not in prompt_context_ids]
        if unknown:
            problems.append(("hallucinated_id", claim["text"], unknown))
            continue
        premise = "\n\n".join(resolve_text(c) for c in cited)
        if judge(premise=premise, hypothesis=claim["text"]) != "entailed":
            problems.append(("unsupported", claim["text"], cited))
    return problems

Run it on a sample of production traffic and on every case in your eval set. When the rate of unsupported claims starts climbing, that's drift. Without this check, you'd usually hear about it first from user complaints.

Worked example: walking an incident backwards

A typical incident. The question is "What notice period do contractors get?" The answer says 30 days, but the current policy says 14. Here's how you walk it back:

• ▹Hop 1, claim to cited context. The claim "Contractors are entitled to 30 days' notice" cites ctx_b_summary_9, and the verifier passed it, because the summary really does say 30 days. First lesson: entailment against cited context proves the claim matches its inputs. It doesn't prove the inputs are right. Keep walking.
• ▹Hop 2, summary to its parents. ctx_b_summary_9 was produced by agent_b/summarise@v7, with parent_ids [chunk_17, chunk_42]. chunk_17 comes from the employee leave policy ("Employees: 30 days' notice"). chunk_42 comes from the contractor policy ("Contractor notice follows the employee policy").
• ▹Hop 3, chunk to its build. chunk_42 has transform_version chunker@1.4 and index_snapshot idx_2026_08_01. The current chunker is 2.x, so this snapshot was built before the chunker upgrade.
• ▹Hop 4, chunk to source. The content_hash on chunk_42's root document doesn't match the current contractor policy. Last month the document was revised to set an explicit 14-day period and drop the cross-reference. Nobody rebuilt the index.

Root cause: two faults, and either one alone might not have produced the wrong answer. The stale index served a cross-reference that no longer exists. The summariser then combined "follows the employee policy" with "employees get 30 days" and stated the result as a fact about contractors. That's four hops and about ten minutes of queries, compared with a day of re-running the pipeline and guessing. You also come out with two concrete fixes. One is a freshness check that compares source hashes against the index. The other is a change to the summariser prompt, plus an eval case that penalises merging claims across documents.

Where to store it without buying a platform

You don't need a lineage product. You need append-only records stored alongside the traces you already collect. There are two practical options, and plenty of teams use both:

• ▹OpenTelemetry span attributes: on each LLM or retrieval span, set custom attributes like ctx.consumed_ids (array) and ctx.produced_ids (array). OTel supports array-valued attributes. The GenAI semantic conventions cover model, agent and tool calls but say nothing about context provenance, so use your own namespace. It's cheap, and it lets you jump from a trace straight into the graph.
• ▹One Postgres table: holds the provenance records, with parent_ids as an array. This is where you run your queries. Recursive CTEs handle the graph walks, and at this scale you don't need a graph database.

CREATE TABLE provenance (
  context_id        TEXT PRIMARY KEY,
  parent_ids        TEXT[] NOT NULL DEFAULT '{}',
  kind              TEXT NOT NULL,        -- document | chunk | retrieval | summary | claim | answer
  source_uri        TEXT,                 -- roots only
  content_hash      TEXT NOT NULL,
  transform         TEXT NOT NULL,
  transform_version TEXT NOT NULL,
  produced_by       TEXT,
  index_snapshot    TEXT,
  trace_id          TEXT,
  created_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX provenance_parents_gin ON provenance USING GIN (parent_ids);
-- Append-only: grant INSERT and SELECT to the app role, never UPDATE.

-- Walk back: from a claim to every ancestor
WITH RECURSIVE lineage AS (
  SELECT p.*, 0 AS depth FROM provenance p WHERE p.context_id = $1
  UNION ALL
  SELECT parent.*, l.depth + 1
  FROM lineage l
  JOIN provenance parent ON parent.context_id = ANY(l.parent_ids)
  WHERE l.depth < 20
)
SELECT depth, context_id, kind, transform, transform_version,
       produced_by, index_snapshot, source_uri, content_hash
FROM lineage ORDER BY depth;

-- Blast radius: which answers cited anything derived from this document?
WITH RECURSIVE downstream AS (
  SELECT context_id, kind FROM provenance WHERE source_uri = $1
  UNION
  SELECT child.context_id, child.kind
  FROM downstream d
  JOIN provenance child ON child.parent_ids @> ARRAY[d.context_id]
)
SELECT context_id FROM downstream WHERE kind = 'answer';

One detail that's easy to miss: the forward query uses @> ARRAY[...] instead of = ANY(...). A GIN index on parent_ids only helps array operators like containment (@>) and overlap (&&). It doesn't help = ANY(). Use = ANY() and the blast-radius query scans the whole table at every recursion step.

The forward direction is the bonus you didn't plan for

You build lineage to debug incidents backwards. Walk the same graph forwards and it answers questions that come up every week:

• ▹Right-to-be-forgotten and deletes: a document has to go. Which chunks, summaries, cached answers and agent memories came from it? The blast-radius query gives you the full list, so you don't have to hope you found everything.
• ▹Re-embedding migrations: you're moving to a new embedding model. Which cached answers depended on chunks retrieved under the old model and snapshot? Filter on transform_version and index_snapshot and invalidate exactly those.
• ▹Eval regressions: faithfulness dropped this week. Group the failing claims by produced_by. More often than not, one prompt version or one agent accounts for most of the failures.

That's why lineage earns its keep outside incidents too. Compliance, migrations and evals all need the same graph.

What not to do

• ▹Don't log full context blobs on every record. Storage costs add up, and worse, you copy PII into a second system that has its own retention and access rules. Store hashes and pointers. The text already lives in your document store and vector store, and you can fetch it when you need it.
• ▹Don't treat model self-citations as ground truth. A citation is the model's account of what it relied on, and that account is sometimes wrong. Verify with ID membership plus entailment, and store the verifier's verdict in its own field.
• ▹Don't skip IDs on intermediate agent outputs. Teams often tag chunks and final answers and leave the agent-to-agent handoff bare. In the worked example, that handoff is where the bug was.
• ▹Don't wait for a vendor to support agent handoffs. This is one table, one helper and two queries. You can ship it this sprint and migrate later if a standard emerges.

Course close: 30 days in one sentence

For 30 days you learned how to get data into a model's context. Lineage lets you prove which data got there and why, and you need that before you can be on call for the system.

Your checklist for this week:

• ▹At index time, add context_id, parent_ids, content_hash and transform_version to your chunks. At retrieval time, record index_snapshot.
• ▹Make every agent's output declare the context_ids it consumed, and ask the final model for a supported_by list on each claim.
• ▹Create the provenance table and save the walk-back query as a runbook snippet. Then take one known-bad answer from your eval set and walk it all the way back to its source.

Extend your knowledge

• ▹Read the W3C PROV data model (PROV-DM / PROV-O). Entity, Activity and Agent map directly onto context, transform and produced_by, and they give you shared vocabulary if you ever need to export lineage.
• ▹Read the OpenLineage spec and its facets. Note what table lineage covers well and where it stops for agent handoffs.
• ▹Read the OpenTelemetry GenAI semantic conventions, then design your own ctx.* attribute namespace so it sits cleanly beside the gen_ai.* attributes your tracer already emits.
• ▹Read the PostgreSQL docs on WITH RECURSIVE queries and on GIN indexes over arrays. Then load a week of real provenance records and time the walk-back and blast-radius queries.