Provenance in Spec Recovery

Day 5 — Provenance: why a correct spec can still be wrong

Point an agent at legacy code and it hands you a spec that's accurate, internally consistent, and quietly wrong. Accurate because it read every line. Wrong because a spec is supposed to say what someone meant — and the code never recorded that. This lesson is about the gap between what the code does and what anyone intended, and the one habit that stops a recovered spec from writing your oldest bugs down as requirements.

The PR that lost an argument to a document

I fix a rounding bug in a billing engine. Invoice totals were rounding each line item to 2 decimals and then summing, which drifts by a cent on big invoices. Standard fix: sum in full precision, round once at the end. Small diff, tests green. I open the PR expecting a rubber stamp.

The reviewer rejects it — not with an argument, with a quote. They paste a line from our recovered spec: "Line items are rounded to 2 decimal places before aggregation." Then: "Your change violates the spec. Was this intended?" And here's the trap: the spec is right. That IS what the code does. The spec is now sitting on the bug's side of the table, and it outranks me, because three weeks ago everyone signed off on it.

Rewind three weeks: how a wrong spec passed review

We ran an agent over the billing engine to recover a spec before a rewrite. Nobody could explain half the behavior anymore — the people who wrote it had left. The agent did a genuinely good job: read every branch, traced the money path, produced a clean, internally consistent document where every clause mapped to real code. We reviewed it the obvious way — read each clause, find the matching code, confirm they agree. They all agreed. Approved.

That review felt rigorous and was worthless, for a reason that took me a while to see. We were checking the spec against the exact thing the spec was extracted from. Of course they matched. The rounding bug matched too. It sailed through precisely because it was faithful.

The invisible property: agreement IS the failure

Name it clearly, because it runs against instinct. A spec that both agrees with the code AND is wrong is undetectable by any review that checks the spec against the code. Under that test, the two failure modes look identical:

• ▹Clause is a real requirement, matches code → passes review (correct).
• ▹Clause is a calcified bug, matches code → passes review (wrong, invisible).
• ▹The review only has one signal — spec vs. code — and both cases score the same on it.

So the agreement you were treating as evidence of correctness is exactly what hides the error. You can't catch this by staring harder at the code. The information that would separate the two cases was never in the code to begin with.

The load-bearing gap: behavior vs. intent

Here's the mechanism underneath all of it. Code records behavior. A spec is supposed to record intent. A bug is behavior with no intent behind it — nobody chose it, the machine just does it. When an agent reads a codebase it recovers behavior perfectly and intent not at all, because intent was never written into the source. It lives in someone's head, in a Slack thread, in a decision made in a meeting in 2019.

Now watch what an LLM optimizing for fidelity-to-code does when it hits an ambiguous line. It has to answer "was this intended or accidental?" and it has exactly one input: the line itself. The line can't tell it. So the model resolves the ambiguity the only way its objective allows — toward "intended," because describing the behavior as a rule is what maximizes agreement with the code. Every accident comes back written in the confident grammar of a decision. "Line items are rounded before aggregation" reads like policy. It was a bug.

Why the oldest bugs are the most dangerous

Give a wrong behavior enough time and it stops being harmless. This is Hyrum's Law: with enough consumers, every observable behavior of your system — intended or not — becomes something someone depends on. The rounding quirk has been in production for six years. Somewhere downstream, a reconciliation job, a tax report, and a partner's ledger have all been built to expect the drift. Fix the code and you don't fix a bug — you break three teams who were correct to rely on what the system actually did.

So now the behavior is a real contract: accurate as a description of reality, false as a statement of intent. Nobody ever decided it should work this way, but everyone acts as if someone did. This is the exact seam where "just fix the bug" turns into a cross-team negotiation. And the recovered spec, by canonizing the behavior, quietly sided with the people who don't want it fixed.

The move: provenance over description

The fix is not a better extractor. A better extractor gives you a more faithful mirror, and fidelity is the problem, not the solution. The fix is to stop asking one question and start asking two.

• ▹Description (what the agent already does well): what does this line do?
• ▹Provenance (the missing column): why does this line exist — did someone decide it, or does the code just happen to do it?

Provenance is recoverable, just not from the code alone. It lives in the history around the code: git blame on the line, the PR that introduced it, whether a human ever wrote down a reason, whether anyone defended it in review. "Someone chose this" and "the code happens to do this" are two different columns, and the whole point is to never let them collapse into one. A commit message that says "round per-line per finance requirement, ticket FIN-241" is intent. A commit that says "fix build" and happens to touch the rounding is not — it's behavior that leaked in.

How to make an agent actually do it

You change the agent's job from "describe the code" to "describe the code AND attach provenance to every clause." Give it the history, not just the source, and make it grade its own confidence about intent. The prompt shape looks like this:

For each rule you extract, output:
  - clause: the behavior, in plain language
  - code_ref: file:line it came from
  - provenance:
      origin_commit: <sha from git blame>
      pr: <originating PR, if any>
      human_rationale: <quote from commit/PR/review, or NONE>
  - intent_status: one of
      DECIDED         (a human stated a reason for this behavior)
      BEHAVIOR_ONLY   (no decision found; code just does this)
Mark every clause with no human decision behind it as
BEHAVIOR_ONLY, UNVERIFIED INTENT. Do not phrase it as a requirement.

Now the spec has two kinds of lines. The DECIDED ones are requirements — real intent, defend them. The BEHAVIOR_ONLY ones are your two most valuable lists at once: your fix-it-safely worklist (change behavior here without violating anyone's decision, because there was no decision) and your negotiation list with downstream teams (the Hyrum's-Law contracts nobody chose, that you'll need to renegotiate before you touch them).

Go back to my rounding PR. Under this scheme the reviewer doesn't get to quote a bare requirement at me. They get: "Line items rounded before aggregation — BEHAVIOR_ONLY, origin commit 'wip', no rationale, never defended in review." That tag flips the whole conversation. The spec is no longer on the bug's side; it's telling us this was never decided. So the real question isn't "does your fix violate the spec," it's "who downstream is quietly depending on the old behavior, and have we told them."

Close and bridge

A living spec isn't a mirror of the code. A mirror hands you back the bugs with a straight face. A living spec is the code plus the reason each thing is there — and where the reason runs out, it says so out loud instead of inventing one. That honesty about its own gaps is the entire value. A spec that can't tell you what it doesn't know is worse than no spec, because it launders behavior into requirements under review-shaped cover.

Tomorrow we pick up exactly where the reason runs out: the invariants that don't live in any single file. The rules that hold in the seams between modules — module A assumes B never hands it a null, B assumes A already validated — that no line of code states and no single-file read can recover. Provenance tells you which rules were decided; next we go after the rules nobody wrote down at all.

Extend your knowledge

• ▹Read Hyrum Wright's one-page statement of Hyrum's Law (hyrumslaw.com) and the 'Deprecation' / 'Dependencies' chapters of Google's 'Software Engineering at Google' — the canonical treatment of implicit contracts and how observable behavior becomes a dependency.
• ▹Run the two-question pass on one real module you own: pick 10 non-obvious behaviors, git blame each, and tag DECIDED vs BEHAVIOR_ONLY. Count how many 'requirements' turn out to have no decision behind them — that ratio is your risk surface.
• ▹Feed an agent the git log/blame for a file alongside the source and have it produce a provenance-tagged spec (using the intent_status schema above). Compare it to a plain 'summarize this code' spec and note exactly which clauses change status.
• ▹Look at how characterization tests (Michael Feathers, 'Working Effectively with Legacy Code') pin behavior without claiming intent — the same discipline, applied to tests instead of specs: they lock what the code does while staying explicitly agnostic about whether it's correct.