Semantic data contracts

Day 26: Data Contracts That Encode Meaning, Not Just Shape

Your orders table can pass every check for 19 days straight while Q3 revenue quietly drops 18%. Most data contracts in production only check that a column exists and has the right type. That catches the bugs that crash a pipeline. It misses the ones that keep it running and quietly corrupt the numbers. Coding agents and LLM steps now edit pipelines too, so the contract file is often the only stakeholder that gets a say before a change ships.

The hook: 19 days of green checks

This is a composite of incidents I've seen at more than one company. The details are changed and the shape is real. Finance sends a dashboard screenshot with one line: 'why is Q3 down 18%?' Every data check on the orders table is green, and has been for weeks. The cause is a one-line diff in the upstream orders service:

- amount = gross - refunds
+ amount = gross - refunds - fees

The PR description said 'align amount with net revenue'. That made sense to the payments team, who wanted net-of-fees for payouts. It was wrong for every downstream consumer that read `amount` as customer-facing revenue. Refunds land late, so the orders model reprocessed the whole open quarter on every run. All of Q3 was quietly restated under the new definition, while closed Q2 stayed frozen under the old one. In that business, fees included marketplace commission, which is why the gap was so big. Nobody downstream was told, because as far as the tooling knew, nothing had changed. It came to light 19 days later, when finance did a mid-quarter review by hand.

The autopsy: what the contract checked, and what it didn't

The team had a data contract, and they enforced it. Here's everything it verified on each of those 19 days:

• ▹Column `amount` exists. Passed.
• ▹Type is DECIMAL(12,2). Passed. Subtracting fees doesn't change the type.
• ▹`amount` is non-null. Passed.
• ▹Table landed by 06:00 UTC. Passed.
• ▹Row count within the usual band. Passed too: the number of orders didn't change, only the value inside each one.

It never checked what `amount` means. A type tells you the value is a number with two decimals. It can't tell you which number. I call this failure mode **structurally valid, semantically drifted**: every byte matches the schema, but the business meaning has shifted underneath it. A contract that only encodes structure is a type system pretending to be an agreement.

Definition, kept short

A **data contract** is a producer-owned, versioned agreement about a dataset's shape, meaning, and guarantees, enforced at the boundary where the data changes hands.

• ▹**Producer-owned**: the team that writes the data signs it. The team that suffers when it breaks doesn't.
• ▹**Versioned**: every change gets a number, and some numbers mean 'consumers must act'.
• ▹**Shape, meaning, guarantees**: three layers, covered below. Most teams ship only the first.
• ▹**Enforced at the boundary**: checks run where data passes from one owner to another, both in CI before merge and in production on every load.

You've already met this boundary several times in this course: between ingestion and your vector index, and between your chunker and your embedder. Wherever one component's output becomes another's input, a contract exists, whether or not anyone wrote it down.

Layer 1: Shape (the part everyone already has)

Shape covers schema, types, nullability, uniqueness, and freshness. Here's a minimal contract, close to the one from the incident:

dataset: analytics.orders
version: 1.4.0
freshness: { landed_by: '06:00 UTC' }
schema:
  order_id:   { type: STRING, nullable: false, unique: true }
  order_date: { type: DATE, nullable: false }
  amount:     { type: 'DECIMAL(12,2)', nullable: false }

Add a CI check that compares the contract on `main` with the one on the PR branch, and blocks any breaking change that comes without a major version bump:

# ci/contract_diff.py: fail the PR if a breaking change lacks a major bump
import sys, yaml
from packaging.version import Version

old = yaml.safe_load(open(sys.argv[1]))   # contract on main
new = yaml.safe_load(open(sys.argv[2]))   # contract on this branch

def breaking(old, new):
    reasons = []
    for name, col in old['schema'].items():
        if name not in new['schema']:
            reasons.append(f'removed or renamed: {name}')
            continue
        n = new['schema'][name]
        if n['type'] != col['type']:
            reasons.append(f'type change: {name}')
        if col.get('nullable') is False and n.get('nullable') is not False:
            reasons.append(f'nullability loosened: {name}')
    return reasons

reasons = breaking(old, new)
if reasons and Version(new['version']).major <= Version(old['version']).major:
    sys.exit('Breaking change without major bump:\n  ' + '\n  '.join(reasons))

You need this check. It also would have passed. The PR never touched the contract file, and even if it had, the shape didn't change. Layer 1 protects you from pipelines that crash. It does nothing about pipelines that keep running and return the wrong answer.

Layer 2: Meaning, written as invariants

Writing 'amount = customer-facing revenue' in a description field is a fine start, but no check will ever fail on it. My test is simple: if you can't write a query that returns rows when the meaning breaks, you haven't really written the meaning down. These are the three patterns I use, strongest first.

**(a) Reconciliation query.** Compare your aggregate against an independent system of record. In this case, daily SUM(amount) has to match the billing ledger within 0.5%.

-- checks/recon_billing.sql: any row returned is a violation
WITH wh AS (
  SELECT order_date AS d, SUM(amount) AS wh_amount
  FROM analytics.orders
  WHERE order_date >= CURRENT_DATE - 7
  GROUP BY 1
), bill AS (
  SELECT settlement_date AS d, SUM(gross_amount - refund_amount) AS bill_amount
  FROM billing.ledger_daily
  WHERE settlement_date >= CURRENT_DATE - 7
  GROUP BY 1
)
SELECT d, wh_amount, bill_amount,
       ABS(wh_amount - bill_amount) / NULLIF(bill_amount, 0) AS rel_diff
FROM wh JOIN bill USING (d)
WHERE ABS(wh_amount - bill_amount) / NULLIF(bill_amount, 0) > 0.005;

**(b) Distribution check.** Compare yesterday's median and p95 with the trailing 28 days. You don't need a second system for this. The catch is that it can only tell you something changed. It can't tell you what's correct.

-- checks/amount_dist.sql: yesterday vs trailing 28-day baseline, 10% band
WITH daily AS (
  SELECT order_date AS d,
         PERCENTILE_CONT(0.5)  WITHIN GROUP (ORDER BY amount) AS p50,
         PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY amount) AS p95
  FROM analytics.orders
  WHERE order_date >= CURRENT_DATE - 29
  GROUP BY 1
), baseline AS (
  SELECT AVG(p50) AS b50, AVG(p95) AS b95
  FROM daily WHERE d < CURRENT_DATE - 1
)
SELECT t.d, t.p50, b.b50, t.p95, b.b95
FROM daily t CROSS JOIN baseline b
WHERE t.d = CURRENT_DATE - 1
  AND (ABS(t.p50 / b.b50 - 1) > 0.10 OR ABS(t.p95 / b.b95 - 1) > 0.10);

**(c) Golden query.** Keep a small, fixed set of known orders whose correct `amount` finance has signed off on, and require the model to return exactly those values. The inputs are fixtures, so this check runs in CI against the PR's build, before anything merges.

-- contracts/golden_orders.csv (signed off by finance)
-- order_id  gross   refunds  fees  expected_amount
-- ord_g001  120.00  20.00    3.80  100.00
-- ord_g002   59.90   0.00    1.97   59.90
-- ord_g003   80.00  80.00    2.62    0.00   -- full refund: the edge case that matters

-- checks/golden_orders.sql: run against the CI build on fixture inputs
SELECT g.order_id, g.expected_amount, o.amount
FROM contracts.golden_orders g
LEFT JOIN analytics.orders o USING (order_id)
WHERE o.amount IS DISTINCT FROM g.expected_amount;

Now replay the incident. Here's which checks would have fired, and when:

• ▹**Golden query: day 0, on the PR.** ord_g001 comes back as 96.20 instead of 100.00, and ord_g003 as -2.62. The build goes red before merge, and the author has to explain the change to the owner.
• ▹**Reconciliation: day 1, first production load.** The warehouse lands well below a ledger that doesn't net out fees, far outside the 0.5% tolerance. It catches the drift no matter who changed what, or where.
• ▹**Distribution check: probably day 1, then it goes quiet.** If fees move the median past the band, it fires. If someone snoozes it, the trailing 28-day baseline absorbs the new normal and the check goes green again within about four weeks. An unacknowledged drift check eventually accepts the drift. Treat it as a smoke alarm. Don't make it your only check on meaning.

Layer 3: Ownership

An invariant that fails with nobody to call is just noise. Ownership turns a failure into a specific person's problem and makes the change process explicit.

• ▹**Producer**: the service and team that write the dataset. Name the actual team. 'Data platform' doesn't count.
• ▹**Accountable human**: one named person who approves major changes. With only a team alias, nobody feels responsible.
• ▹**On-call**: the pager rotation that gets woken up when an invariant fails in production.
• ▹**Semver policy**: renaming or *redefining* a field is a major bump, even if the type stays the same. Adding a nullable field is minor. A doc fix is a patch.
• ▹**Deprecation window**: how long the old meaning stays available after a major bump, say 90 days. During that window you publish `amount` (old meaning) and `net_amount` (new meaning) side by side. You never swap them silently.

To enforce the redefinition rule mechanically, extend the Layer 1 CI diff so that a changed meaning counts as breaking:

        # Layer 2/3 rule: same type, new meaning is still a breaking change
        for key in ('definition', 'derivation'):
            if col.get(key) != n.get(key):
                reasons.append(f'redefined: {name}.{key}')

That gives you two gates. If someone changes the meaning honestly and edits the definition in the contract, they have to bump the major version. If someone changes the meaning in code and never touches the contract, the golden query catches it. Contracts are socio-technical. The YAML records the commitment, and a named person stands behind it. YAML without a person means nobody owns the commitment. A person without YAML means the commitment is tribal knowledge, and tribal knowledge rarely survives the next reorg, let alone the next agent PR.

The retrieval twist: the same drift ruins RAG quietly

All of this applies to your retrieval pipeline, and the drift is harder to spot there because nobody reconciles retrieval quality against a ledger. Two cases I've seen:

• ▹**`doc_date` switches from published-at to ingested-at.** It's still a non-null TIMESTAMP, so the schema check passes. After a full re-ingestion, every document looks like it was published last Tuesday. Your recency filter (`doc_date > now - 365d`) now lets a 2023 refund policy compete with the 2026 one, and the agent cites whichever chunk embeds closer to the question.
• ▹**The chunker changes what `section` means.** It used to hold the nearest H2 heading ('Refunds'). Now it holds a breadcrumb ('Billing > Refunds > EU'), still a non-null STRING. Every metadata filter on `section = 'Refunds'` matches nothing. Depending on how your retriever is wired, you get empty results or a silent fallback to pure vector similarity.

The fix is the same pattern. Write the meaning down, then add a golden retrieval query to the index's contract: a fixed question whose correct top-k results you already know.

dataset: rag.doc_chunks
version: 3.1.0
schema:
  doc_date:
    type: TIMESTAMP
    nullable: false
    definition: >
      When the source document was published in the CMS.
      Never the ingestion time.
  section:
    type: STRING
    nullable: false
    definition: >
      Text of the nearest H2 heading only, no breadcrumb path.
invariants:
  - name: doc_date_is_publish_date
    sql: checks/doc_date_vs_cms.sql      # golden docs: doc_date = cms.published_at
    run: [ci, daily]
  - name: golden_retrieval_refunds_eu
    query: 'What is the current refund window for EU customers?'
    filters: { doc_date_after: '-365d', section: 'Refunds' }
    expect_in_top_k:     { k: 5, doc_ids: [policy-refunds-eu-2026] }
    expect_not_in_top_k: { k: 5, doc_ids: [policy-refunds-eu-2023] }
    run: [ci, daily]

One golden retrieval query doesn't measure quality. The eval sets from earlier in the course do that. This check has a narrower job: fail loudly when a field's meaning shifts under the filters you rely on.

Why this matters more now: agents read the contract, not the Slack thread

In the human version of this incident, a few things might have saved you. A senior engineer who remembers that `amount` excludes fees for a reason. A Slack thread from 2024. A reviewer who knows finance reads this table. Now give a coding agent the task 'clean up the revenue calculation and align naming with the payments domain.' It will open a tidy PR with passing tests and a sensible description. It hasn't seen the Slack thread or met the finance team. The only stakeholder it reads is the contract file sitting next to the code.

In my experience, agents follow written invariants well and ignore unwritten ones completely. Say the contract has `derivation: gross - refunds` and a golden test fails on 96.20. The agent sees the failure, reads the definition, and usually either reverts or proposes a new field under a major bump. If the meaning only lives in people's heads, the agent has no way of knowing it's there. Shape was always the easy part. Meaning now does most of the work, and only if a machine can fail on it.

The same goes upstream. If an LLM extraction step fills `amount` from invoice PDFs, a prompt tweak can flip it from tax-inclusive to tax-exclusive with zero schema change. In multi-agent systems, the messages between agents are datasets with a contract. When a research agent starts returning `price` in cents while the planner still reasons in dollars, the result looks like a hallucination. It's actually a structurally valid, semantically drifted handoff. Put definitions and at least one golden example into your tool and message schemas. Then point your agent instructions (CLAUDE.md, AGENTS.md, or equivalent) at the contract files so the agent reads them before it edits anything.

Before and after: the full contract

Before. Shape only, the contract that stayed green for 19 days:

dataset: analytics.orders
version: 1.4.0
freshness: { landed_by: '06:00 UTC' }
schema:
  order_id:   { type: STRING, nullable: false, unique: true }
  order_date: { type: DATE, nullable: false }
  amount:     { type: 'DECIMAL(12,2)', nullable: false }   # passes before and after the bug

After. All three layers in about 40 lines, with comments marking which line catches the bug and when:

# ---- Layer 1: shape ----
dataset: analytics.orders
version: 2.0.0
freshness: { landed_by: '06:00 UTC' }
schema:
  order_id:   { type: STRING, nullable: false, unique: true }
  order_date: { type: DATE, nullable: false }
  amount:
    type: 'DECIMAL(12,2)'
    nullable: false
    # ---- Layer 2: meaning ----
    definition: >
      Customer-facing revenue for the order: gross charged minus refunds.
      Excludes processor and marketplace fees.
    derivation: gross - refunds            # editing this = major bump (CI diff)
invariants:
  - name: golden_orders                    # <- fails on the PR: day 0
    sql: checks/golden_orders.sql
    expect: zero_rows
    run: [ci, daily]
  - name: billing_reconciliation           # <- fails on first prod load: day 1
    sql: checks/recon_billing.sql
    tolerance: 0.005
    run: [daily]
  - name: amount_distribution              # <- likely day 1; self-heals by ~day 28 if ignored
    sql: checks/amount_dist.sql
    band: 0.10
    run: [daily]
# ---- Layer 3: ownership ----
ownership:
  producer: orders-service
  team: payments-data
  accountable: <named engineer>            # approves every major bump
  on_call: pagerduty:payments-data-primary # who gets paged on day 1
versioning:
  policy: semver
  major_if: [remove_field, rename_field, type_change, redefine_field]
  deprecation_window_days: 90
consumers: [finance-dashboards, revenue-forecast, support-rag-index]

When to go all three layers, when not, and common pitfalls

• ▹**Do it for** fields that feed money, compliance, customer-facing answers, or agent decisions. Usually a handful of fields carries most of the risk.
• ▹**Don't do it for** every column in every staging table. A contract nobody maintains is worse than no contract, because it gives you false confidence. Start at cross-team boundaries, not inside one team's pipeline.
• ▹**Pitfall: description-only meaning.** Prose with no failing check behind it is documentation. It isn't a contract.
• ▹**Pitfall: distribution checks as the main defence.** Rolling baselines absorb drift. Pair them with a reconciliation or golden query that has a fixed reference.
• ▹**Pitfall: consumer-written contracts.** If the consumer wrote it and the producer has never seen it, what you've built is an alerting system, not an agreement.
• ▹**Pitfall: golden sets that drift.** Golden values need an owner and a sign-off. Once an engineer can edit expected values just to turn CI green, the check means nothing.

Do this today

• ▹Pick your single most important metric field: the one that finance, a customer, or an agent would act on.
• ▹Write its definition in one sentence, then write its derivation.
• ▹Add one reconciliation query against an independent source, or one golden query with three fixtures. Include the nasty edge case, such as a full refund.
• ▹Put one named person and an on-call rotation in the contract.
• ▹Add the redefinition rule to your CI contract diff, so the build fails when a definition changes without a major bump.
• ▹Tomorrow, on Day 27, we push these boundary checks into continuous data quality monitoring and observability: what to watch between deploys, and how to get an alert to the right person.

Extend your knowledge

• ▹Andrew Jones, 'Driving Data Quality with Data Contracts' (Packt). A practical book on the producer-ownership model and how to roll it out across teams.
• ▹The Open Data Contract Standard (ODCS) from the Bitol project under the Linux Foundation, plus the open-source datacontract-cli. Try mapping this lesson's three-layer YAML onto their fields.
• ▹On dbt? Combine model contracts (enforced shape) with dbt unit tests on fixture inputs. You get Layer 1 and a golden query in the same CI run.
• ▹Chad Sanderson's writing on data contracts as an organisational agreement between producers and consumers, not just a tooling feature. Useful framing if you're the EM who's been asked to 'add data contracts'.