Measuring Your Team's Spec-Driven Development Level

Day 2 — Your spec-driven level is a behavior, not a folder

One question sorts every "we're spec-driven" team into the truth: when the code and the spec contradict each other, which file do you open first? Not your tooling, not how many docs live in specs/ — that one reflex. In the AI era you can generate a gorgeous specs/ folder before lunch. What you can't generate is the nerve to trust it at 11pm with prod on fire.

The hook: two teams, same folder, opposite reflex

Two teams. Both have a specs/ directory. Both give the same standup answer: "yeah, we're spec-driven, it's all in specs/." Same tooling, same Cursor/Copilot setup, same nightly agent runs. On paper, twins.

A bug hits prod. Team A opens the code, patches the handler, ships the hotfix, moves on — the spec is now a lie and nobody notices for three weeks. Team B opens the spec, fixes the requirement that was wrong, regenerates the module, and lets the fix flow down into the code. Same folder. Opposite reflex. Which file gets touched first when the pager's going off — that's the whole measurement. Everything else is decoration.

Why "we have a specs/ folder" is the most common self-lie in ASE

Most maturity self-checks count artifacts: do we have specs, do we have ADRs, do we have a PRD template. Artifacts are easy to count and easy to fake — point an agent at your repo and it'll scaffold a beautiful specs/ tree by the afternoon. So artifact-counting flatters everyone into claiming a rung or two above where they actually behave. The documents exist, so you feel spec-driven.

But a spec that describes the code isn't driving anything. It's trailing it, the way documentation always has. The honest variable isn't "do the documents exist" — it's "is the spec authoritative enough that when it fights the code, you fix the spec first and regenerate." Authority is a behavior, and you can only see it under disagreement. It never shows up in a file listing. That's why the folder lies and the reflex doesn't.

• ▹Counting artifacts asks: what documents do we own? Easy, fakeable, flattering.
• ▹Measuring behavior asks: which one wins when they diverge under pressure? Hard, honest, unfakeable.
• ▹And the artifact is nearly free to produce now, so counting artifacts measures almost nothing about your real level.

The actual ladder: 5 rungs by what's authoritative

The rungs aren't sorted by how many docs you have. They're sorted by which artifact is the source of truth and what it costs to regenerate from it. Read them as behaviors, not document counts.

• ▹Rung 1 — Spec describes. The spec is written after or alongside the code, as documentation. Code is truth; the spec trails. When they disagree you fix the code and maybe update the doc later — usually never.
• ▹Rung 2 — Spec reviews. The spec is a gate: PRs get checked against it in review, by a human or an LLM judge. Code is still truth, but at least divergence gets flagged. When they disagree you argue about which is right; the spec sometimes wins on paper, rarely in the commit.
• ▹Rung 3 — Spec generates once. The spec seeds the first cut — an agent scaffolds the module from it — then humans take over and the spec is abandoned. Authority lasts exactly one generation. After that, code is truth again.
• ▹Rung 4 — Spec regenerates on demand. The spec is authoritative: to change behavior you edit the spec and re-run the agent, and hand-editing generated code is a smell you push back on. When they disagree you fix the spec first. This is the first rung that's genuinely spec-driven.
• ▹Rung 5 — Code is disposable. The spec, plus tests and constraints, is the artifact you version, review, and defend. The code is a build output you'd happily throw away and regenerate. Nobody hand-edits output because there's no point — the next regeneration overwrites it. Very few teams live here for real, and that's fine.

Watch the gap between rung 3 and rung 4. Rungs 1 through 3 all quietly keep code as the real source of truth — the spec is scenery. Rung 4 is where authority actually flips. Most teams who say "we're spec-driven" are sitting at rung 2 or 3 and calling it rung 4.

The mechanism nobody names: leveling up moves drift, it doesn't kill it

Here's where people get it wrong. They think climbing the ladder kills drift. It doesn't. Each rung just relocates drift somewhere new, louder, and cheaper to catch. Drift is conserved — all you're choosing is where it lives.

• ▹Rung 1: drift lives in someone's head — the gap between what the code does and what the one senior who wrote it thinks it does. Invisible until they leave.
• ▹Rung 2–3: drift moves to the doc-vs-code gap. The spec says X, the code does Y, and you find out in review or never.
• ▹Rung 4: drift moves to coverage gaps. The spec is authoritative, so divergence now shows up as "the spec said nothing about this case and the agent guessed." Louder, because regeneration surfaces it.
• ▹Rung 5: drift moves to spec-vs-intent. The code faithfully implements the spec, and the spec faithfully captures the wrong intent. The cheapest place to catch a mistake, because it's one edit to the source artifact.

So the real KPI isn't "do we have drift." Everyone has drift, at every rung, forever. The KPI is time-to-detect-divergence. A rung-1 team can carry a lie in the code for months. A rung-4 team catches it on the next regeneration. Climbing the ladder buys you a shorter detection window, not a drift-free life. If your maturity pitch promises "no more drift," you're selling something that doesn't exist.

This matters more with agents writing the code, not less. When an LLM spits out a thousand lines from a vague spec, the gap between what you meant and what shipped is bigger and faster than any human could produce. A short detection window is your only defense — and that's exactly what the upper rungs buy you.

The diagnostic you can run this week

Don't self-assess by asking "are we spec-driven?" You'll get the flattering answer every time. Force the disagreement and watch the reflex. Two ways to run it:

• ▹Force a divergence. Take your gnarliest module, change the spec so it contradicts the current code (or plant a real bug), and watch what your team fixes first. Spec first → rung 4+. Code first, spec updated "later" → rung 1–2. Code first, spec never touched → be honest, that's rung 1.
• ▹Pull the owner. Take the one person who holds that module in their head and put them on leave for a week. If the team can regenerate or safely change it from the spec, the authority is real. If everything stalls until they're back, the spec was scenery and the truth was in their head the whole time.
• ▹Time it. From the moment the divergence exists to the moment someone notices — that's your time-to-detect-divergence. That number, not your folder, is your level.

How badly this fails is the point. A team that yelps "wait, which one's right?" and reaches for the code is showing you rung 2 no matter what the standup said. The failure mode is the measurement.

This-week diagnostic
--------------------
1. Pick the module you trust least.
2. Introduce a spec/code disagreement (or pull its owner).
3. Observe — do NOT coach — which artifact gets edited first.
4. Record time-to-detect-divergence.

Reflex observed          -> Real rung
----------------------------------------
code patched, spec ignored -> 1
code patched, spec "later"  -> 2
spec seeded then hand-edited -> 3
spec fixed, regenerated     -> 4
nobody hand-edits output    -> 5

The trap that caps your number: maturity is a min, not an average

This is the one that quietly sandbags most teams. You've got one senior who genuinely works at rung 4 — edits the spec, regenerates, never touches output by hand. Three other engineers hand-edit the generated code because it's faster for their ticket. Your instinct is to average it: "we're roughly rung 3." Wrong.

Maturity is a min-function. The moment anyone hand-edits generated output, the spec stops being authoritative for that module — because the next person who regenerates from the spec silently wipes out their teammate's edit. So the safe assumption for the whole team becomes "you can't trust regeneration," which means nobody regenerates freely, which means you're all operating at hand-edit level. One defector caps the entire team at their rung. The senior's rung-4 discipline is worth nothing if regenerating might blow away someone's undocumented fix.

The practical consequence: you level up a team by raising the floor, not the ceiling. Adding more spec discipline to your best engineer changes nothing. Stopping the three hand-editors — or making their edits flow back into the spec — is the only move that raises the real number.

Close: what to do with your honest rung

Once you've run the diagnostic and got your unflattering number, don't march into the next all-hands announcing "we're rung 4." Name the real rung out loud, then pick the one behavior that raises the floor. If you're at rung 2, the goal isn't rung 5 — it's making one module survive a spec-first fix under real pressure. Rungs get climbed one module and one reflex at a time, not by rewriting the maturity slide.

• ▹Write down your honest rung and the evidence — the reflex you actually observed, not the aspirational one.
• ▹Find your min. The lowest-behaving person or module is what you fix, not your best case.
• ▹Pick one module to move from its current rung to the next, and make the spec-first reflex the default there before you spread it anywhere.
• ▹Track time-to-detect-divergence as the metric, so "we improved" means something measurable instead of vibes.

Day 3: the prerequisite for climbing even one rung is a spec that's actually authoritative enough to regenerate from — and most specs aren't, they're wishlists. We'll write one that can carry the authority the ladder demands: precise on behavior, explicit on constraints, testable — so the agent has no room to guess and you have every reason to fix it first.

Extend your knowledge

• ▹Run the week-one diagnostic on exactly one module and write down the rung plus the reflex you observed — bring the evidence, not the vibe, to your next retro.
• ▹Read GitHub's Spec Kit and the 'spec-driven development' framing behind it. Study how it treats the spec as the regenerable source artifact (rung 4 behavior), then compare it honestly to how your team behaves.
• ▹Look at how Kiro (AWS) and tools like Cursor's rules/specs structure the spec-as-authority workflow — note where the tool assumes regeneration and where your team quietly hand-edits instead.
• ▹Instrument one metric: for your chosen module, start tracking time-to-detect-divergence over a month. It'll tell you whether you're actually climbing or just redecorating the folder.