Your Analysts Know Which Table Lies — Your Agent Doesn't, and It Won't Ask

Day 4: Schema-on-Read Was a Deal With Humans — Agents Never Signed It

Point an agent at a data lake whose only real documentation lives in your senior analyst's head, and watch what it doesn't do: it doesn't hesitate, it doesn't flag uncertainty, it doesn't ask around. It just answers — fluent, confident, and wrong. That's the failure mode this lesson is about.

The wrong customer_id

An agent I was testing got asked for a customer's lifetime value. Two customer_id columns existed in the lake: one in the canonical dim_customer table, one left over in a legacy staging table nobody had gotten around to dropping, because some downstream job still quietly depended on it. The agent picked the legacy one. The join was clean, the number came back fast, and the number was wrong. No analyst on that team would have made this mistake — not because they're sharper than the agent, but because every single one of them had, at some point, been told in a Slack thread: 'don't use that table, it's stale.' The agent had no such thread to read. It had a schema. And the schema never said 'don't use me.'

Quick recap: Days 1-3 covered ingestion, storage, and retrieval — how data gets into the lake and how you get it back out. Today's about why a lake that's worked fine for humans for years can quietly fall apart the moment you swap the human for an agent.

Tribal knowledge: the undocumented API your lake actually runs on

Schema-on-read means you store data first and figure out its structure and meaning at query time. That's exactly why data lakes scaled — no rigid schema negotiated up front, just dump it in and interpret later. But 'interpret it later' always had a hidden dependency: a human who already knew what the columns meant, which tables were deprecated, which joins were safe, and which numbers were 'directionally right, don't put this in a board deck.' None of that lived in the metadata store. It lived in onboarding docs nobody kept current, Slack threads that scrolled out of search, and the four-word answer 'ask Dave.' It didn't need to be written down, because humans have a side channel — hallway conversations, code review comments, the instinct that builds up after six months on a team. The lake looked well-documented. It wasn't. The humans using it were just doing constant, invisible patching.

Why agents can't inherit it the way a new hire does

A new hire spends their first month absorbing tribal knowledge by osmosis — overhearing the wrong-table warning in standup, getting corrected in a PR comment, clocking the senior engineer's flinch when someone mentions a specific dataset. None of that is written down anywhere an agent can retrieve it. An agent's entire understanding of your data is whatever it can pull at query time: table names, column names, whatever comments happen to exist in the catalog, maybe a data dictionary if one exists and isn't three years stale. There's no flinch to notice, no hallway to walk down, no Dave to ask. If the meaning isn't machine-readable, it doesn't exist for the agent — and unlike a human, the agent won't pause and think 'let me check with someone first.' It'll just produce a fluent, confident, wrong answer. That's the actual shift here: undocumented meaning used to be a minor inconvenience, a Slack search away. For agents, it's the single largest source of silent errors, because there's no self-doubt built into the failure mode.

The diagnostic: 3 questions to ask your lake today

• ▹Can a stranger with zero tribal context find the canonical table for a concept like 'customer' — or do they first need to know who to ask?
• ▹Is 'deprecated' or 'do not use' encoded anywhere machine-readable — a catalog tag, a table property — or does it only exist as a Slack message and a shared memory?
• ▹Does every business-critical column have a real description in the metadata store, or does its meaning live in someone's head and get handed over verbally when someone happens to ask?

If you answered 'in someone's head' to any of these, you don't have a documentation gap. You have an agent-reliability gap — because the only thing standing between a correct answer and a confidently wrong one is a person you haven't scaled.

What actually closes the gap

It's not more governance process, a stricter review checklist, or a wiki page someone promises to keep updated. Wikis don't get read at query time, and agents don't sit in on design reviews. What closes the gap is turning tribal knowledge into a first-class, queryable artifact: a semantic layer or catalog where table and column descriptions, deprecation status, and canonical-vs-legacy markers are things the agent's retrieval step actually fetches before it answers — not documentation the agent has to already know exists. Metadata catalogs (DataHub, Unity Catalog, OpenMetadata) and semantic layers sitting in front of the warehouse aren't new. What's new is the job they're being asked to do: they've gone from 'nice for discoverability' to 'the actual interface the agent reads instead of tribal knowledge.' If it's not retrievable in the same call the agent uses to find the table, it might as well not exist.

Close

Here's the reframe worth sitting with: you're not building documentation for agents instead of humans. You're finally paying down debt that humans have been quietly covering for you for years — every time someone remembered not to touch that stale table, they were patching a gap your metadata should have closed. Fix it for the agent, and your next new hire ramps up in a week instead of a quarter.

Extend your knowledge

• ▹Pick your highest-traffic table and check: does its catalog entry have real column descriptions, or is it blank with the meaning living in your head or a teammate's?
• ▹Look into a metadata catalog if you don't have one — DataHub, OpenMetadata, or your warehouse's native catalog (e.g. Unity Catalog on Databricks) — specifically for whether it supports machine-readable deprecation tags.
• ▹If you already run agents against your warehouse, check what context actually gets passed into the retrieval step — is it table/column names only, or does it include descriptions and canonical/deprecated status?
• ▹Read up on the 'semantic layer' pattern (tools like dbt's semantic layer or Cube) as the mechanism for turning tribal metric definitions into something queryable rather than argued about in Slack.