Agent memory as a feature store

Day 27: Agent Memory Is a Feature Store You're Rebuilding Badly

Your agent scores 94% in eval and 71% in production. You swap the model, rewrite the prompt, and the gap stays put. Often neither of those is the problem. The problem is memory that doesn't know what time it is. Classic ML ran into this about ten years ago, and feature stores were the fix. Today we borrow their vocabulary so you can name this bug and get rid of it.

The hook: 94% in eval, 71% in prod

Here's a typical case. The numbers are illustrative, but I've debugged this exact pattern more than once. A support-triage agent reads a new ticket. It pulls memory about the customer (past tickets, preferences, earlier tool calls), then proposes a root cause and a fix. The offline eval replays a few hundred historical tickets and scores each proposed fix against the fix that actually resolved the ticket. Eval says 94%. After launch, with the same prompt and the same model, the agent is right about 71% of the time.

To find the gap, take one eval case that passed and replay it by hand. Log exactly which memory rows retrieval returned. For ticket T-4812, opened 2026-03-03 09:14, you get this:

ticket_id  : T-4812
opened_at  : 2026-03-03 09:14

memory rows returned for user_id = 'acme-corp':
  id   created_at          content
  m-17 2026-02-11 16:02    "Prefers responses with CLI commands, not UI steps"
  m-23 2026-02-27 10:40    "Runs plugin v2.4 on self-hosted SSO"
  m-31 2026-03-05 18:22    "T-4812 resolved: SSO metadata URL misconfigured;
                            workaround was downgrading plugin to v2.3"   <-- !!

Look at m-31. It was written two days after the ticket was opened, by the async summariser when the ticket closed. So in the eval the agent didn't diagnose anything. It read the answer key out of its own memory. In production that row doesn't exist yet at the moment the agent has to decide, and the score drops.

ML people call this leakage: during training or evaluation, the model sees information it couldn't have had at decision time. Your eval wasn't measuring the agent. It was measuring the agent plus a time machine.

Where this sits in the series

Days 1–26 were about getting the right data into context: embeddings, chunking, hybrid retrieval, reranking, evals. All of that was about relevance. Today we add a second axis, time. The question stops being only "which rows are relevant?" and becomes "which rows were known at the moment this decision was made?". Retrieval without a time axis is half a system. Your retrieval tests pass in eval, and the leakage goes straight into your eval numbers.

What a feature store actually is

A feature store keeps facts about entities. An entity is anything you make decisions about: a user, a ticket, a merchant, a repo. Each fact (a "feature") carries a timestamp for when it was true. "User 42 made 3 purchases in the last 7 days, as of Tuesday 10:00." "Repo X has failing CI, as of 14:32."

It serves those facts two ways. The offline path handles bulk historical questions for training and evaluation: "for each of these 50,000 past events, what did we know about the entity at that moment?" The online path answers one question fast for a live request: "what do we know about user 42 right now?", in milliseconds.

The part that matters: both paths use the same feature definitions, and the offline path is time-correct. It never hands back a fact from after the event. That's really all a feature store is. Everything else is plumbing. Agent memory needs the same three properties.

The translation table

This is the core of the lesson. Each line pairs a feature-store term with its agent-memory equivalent and the bug you get when it's missing.

• ▹Entity ↔ user / session / repo / ticket. The key that memory hangs off. Skip it and one user's or repo's memory bleeds into another's ("why does the agent think I use Java?").
• ▹Feature ↔ memory item. One fact: a preference, a summary, a tool-call result. If items have no clear boundaries, you can't version, expire or audit anything, and memory turns into one long blob.
• ▹event_time vs created_time ↔ "when it happened" vs "when the agent learned it". The user switched language on Tuesday at 09:30, and the summariser recorded it at 11:00. Store only one timestamp and the lag disappears, so replay either leaks the future or drops facts the agent really had.
• ▹Point-in-time join ↔ eval replay. Rebuild the memory state exactly as it stood at each historical decision. Without it you get leakage, and inflated scores like that 94%.
• ▹Online store / offline store ↔ live memory / replay log. Live memory serves the agent now. The append-only log lets you rebuild any past moment. No log means no replay, just guessing.
• ▹Training/serving skew ↔ the eval/prod behaviour gap. One "feature", two code paths, two different values. Without a single definition you get "the agent behaves differently in prod" and nobody can tell you why.
• ▹TTL / freshness SLA ↔ stale preferences. Every fact has a shelf life. Without one, the agent hangs on to a preference from eight months ago ("prefers verbose answers") that the user has contradicted three times since.

The mechanism: one timeline, two queries

Take user u_42, three memory writes, and one agent decision on Tuesday at 10:00. Each write has an event_time (when it happened) and an ingested_at (when the memory system recorded it).

• ▹The naive query `SELECT * FROM memory WHERE user_id = 'u_42'` returns W1, W2 and W3. W3 is plain leakage, since it's the outcome. W2 is sneakier: it happened before the decision, but the agent didn't learn about it until an hour after.
• ▹The point-in-time query `... AND ingested_at <= '2026-03-03 10:00'` returns only W1. That's exactly what the agent saw in production, so a replay using this query reproduces the real decision, mistakes included.
• ▹Rule of thumb: filter on ingested_at to decide what the agent knew. Use event_time for ordering, recency weighting and validity windows. Mix them up and your replays go wrong in ways that are hard to spot.
• ▹W2 is a finding on its own. A 90-minute summariser lag is a freshness problem. With both timestamps stored you can measure ingested_at − event_time and put an SLA on it.

The minimal fix: five columns and one parameter

You don't need a platform. You need two changes. One: every memory row carries five fields: entity_id, event_time, ingested_at, source and valid_until. Two: every retrieval call takes an as_of parameter. In production it defaults to now(). In replay you set it to the decision time.

CREATE TABLE agent_memory (
  id           uuid PRIMARY KEY,
  entity_id    text        NOT NULL,   -- 'user:42', 'repo:acme/api', 'ticket:T-4812'
  content      text        NOT NULL,
  embedding    vector(1536),
  event_time   timestamptz NOT NULL,   -- when it happened in the world
  ingested_at  timestamptz NOT NULL DEFAULT now(),  -- when memory learned it
  source       text        NOT NULL,   -- 'summariser@v3', 'tool:jira', 'user_explicit'
  valid_until  timestamptz             -- NULL = until superseded
);
-- Append-only: corrections INSERT a new row and set valid_until on the old one.
-- Never UPDATE content in place, or you lose the ability to replay.

def retrieve_memory(entity_id, query, as_of=None, k=8):
    as_of = as_of or now()                     # prod: now; replay: decision time
    candidates = store.vector_search(
        query=query,
        filter={
            "entity_id": entity_id,
            "ingested_at": {"lte": as_of},     # only what was KNOWN then
        },
        k=k * 3,                               # oversample; validity filter drops some
    )
    live = [r for r in candidates
            if r.event_time <= as_of
            and (r.valid_until is None or r.valid_until > as_of)]
    return live[:k]                            # keep the vector score order

# prod
ctx = retrieve_memory(f"user:{user.id}", msg.text)
# eval replay: same function, different clock
ctx = retrieve_memory(f"user:{t.user_id}", t.text, as_of=t.opened_at)

Put the time filter inside the vector search, not after it. Qdrant and Weaviate do filtered search natively. In pgvector, a WHERE clause on an HNSW or IVFFlat index applies to whatever candidates the index scan returns, so turn on iterative index scans (pgvector 0.8.0+) or raise ef_search. Filter after taking the top k and the future rows eat your top-k slots, so you get back fewer results than you asked for.

Skew: the subtler bug

Say you fix the timestamps and the eval/prod gap shrinks but doesn't close. Next suspect: skew. The same "feature" is being produced by two different code paths.

It usually goes like this. In production, an async summariser writes memory after each session. It runs prompt v3 on a small, fast model and sees the last 20 turns. When the eval set was built, historical sessions had no memory, so someone wrote a batch backfill script. That script uses a different prompt, a bigger model, and sees the whole transcript at once. Both outputs are labelled "session summary". They're different features. The backfill summaries are cleaner and more complete, so in eval the agent does better on memory it will never get in prod.

• ▹Fix: one write path. Have a single write_memory(event) function and use it for live writes and backfills alike. A backfill should push historical events through that same function, not reimplement it.
• ▹Put the version in source (summariser@v3). When the summariser changes, you can see which rows came from which definition and re-backfill deliberately.
• ▹Feature stores learned this the hard way. Define a feature once, materialise it to both online and offline stores from that one definition, and never let a notebook compute its own variant.

Where the analogy breaks

To be honest, agent memory isn't a classic feature store, and treating it like one will cause its own problems.

• ▹It's unstructured text, not typed columns. You can't range-check "Prefers concise answers unless debugging" the way you'd validate a float.
• ▹The agent rewrites it. In classic ML a pipeline writes features and the model reads them. In agent systems the model writes its own memory: it summarises, merges and "corrects" it. If those writes are in-place UPDATEs, replay is gone. That's why append-only plus valid_until is the part of the fix you don't negotiate on.
• ▹There's no clean label. A fraud model eventually finds out whether a transaction was fraud. "Was this memory helpful?" is fuzzy and arrives late. Judging memory quality takes the eval techniques from earlier in this series, not a join against a labels table.
• ▹So installing Feast won't finish the job. Borrow the semantics (entities, two timestamps, as-of reads, one write path, TTLs) and build them into the Postgres or vector store you already run.

Run-this-week checklist

• ▹Replay: pick one past decision. Can you rebuild exactly what the agent knew at that moment, and nothing more? If replay returns any row with ingested_at after the decision time, you have leakage.
• ▹Two timestamps: do your memory rows store both event_time and ingested_at? If all you have is created_at, work out which of the two it actually means.
• ▹One write path: grep for every place that inserts into memory. Two or more code paths producing the same kind of item means skew.
• ▹Expiry: does anything ever expire or get superseded? Count the preference rows older than 90 days that are still being retrieved.
• ▹Same retrieval: does your eval harness call the same retrieve function as production, with as_of set? Or did someone give it its own "simpler" query?

Bridge to Day 28

Make memory time-aware and a new question turns up. When several agents share memory (a planner, a coder and a reviewer all reading and writing about the same repo), who's allowed to write which items? Whose summary wins? How do you trace a bad decision back to the write that caused it? The source column you added today is the start of an answer. Day 28 is about lineage and ownership: who wrote what, from which inputs, and who gets to change it.

Extend your knowledge

• ▹The Feast documentation on point-in-time joins and entity/feature view concepts. Read it for the semantics. You don't need to install it.
• ▹Martin Zinkevich's "Rules of Machine Learning" (Google), especially the rules on training-serving skew. Most of them carry straight over to agent memory.
• ▹"Hidden Technical Debt in Machine Learning Systems" (Sculley et al., NeurIPS 2015). Read "agent" wherever it says "model" and it describes the memory systems a lot of teams are building right now.
• ▹Exercise: add ingested_at and an as_of parameter to your own agent's memory retrieval, replay 20 past decisions, and count how many pulled in at least one future row. That count is your leakage rate.