Two Green PRs Killed Our Pricing Service at 2AM — and Git Never Saw It Coming

Why This Matters

Run enough agents in parallel on the same codebase and you learn this the hard way: the bug that actually hurts you doesn't show up as a merge conflict. It shows up as two PRs, each green, each merged, that quietly kill production the moment they meet. If the only thing you're watching for is diff overlap, you're staring at the wrong layer.

The 2am Incident: Two Green PRs, One Broken Production

Here's what actually happened. Two agents, same sprint, both working the caching layer for a pricing service. Agent A's ticket: reduce stale price reads. It built a read-through cache with stale-while-revalidate — on a miss or near-expiry, serve the stale value and kick off a background refresh. Agent B's ticket: fix cache invalidation on price updates. It built TTL-based invalidation plus a write-through refresh that fires whenever a price changes upstream. Each agent owned its own file, its own tests, its own PR description. Each suite passed. Each PR was correct on its own terms.

Nobody told either agent the invariant that mattered: only one code path gets to refresh a given cache key inside a TTL window, or you get a refresh stampede that serves inconsistent prices to requests landing milliseconds apart. Agent A's background refresh and Agent B's write-through refresh could both fire for the same key within milliseconds of each other. Harmless alone. Together, a race where a stale price and a fresh price both won depending on timing — and at 2am, that meant a handful of orders got priced off a value that should've died 400ms earlier.

Why Git's Conflict Detection Is Structurally Blind to This

Git works on lines and files. Agent A touched cache_reader.py. Agent B touched cache_writer.py. Zero line overlap, zero file overlap — git merges this without a whisper of complaint. CI is just as blind: each suite mocks the other agent's code path, because as far as that agent knew, the other path didn't exist yet. Code review is blind too, unless the reviewer happens to be holding the entire invariant in their head while reading two diffs that each look small and harmless on their own — which is precisely the judgment call that's easy to skip.

The bug isn't sitting in either diff. It lives in the interaction between two correct implementations of two reasonable-sounding tickets. No static analyzer finds this, because the code isn't arguing with itself — the specs are.

Naming the Real Failure Mode

It's tempting to file this under scheduling: whoever's PR landed first should've won, we just needed to serialize them. Wrong diagnosis. Flip the order and nothing changes — if Agent B ships first and Agent A builds on top, Agent A still has no way to know write-through refresh exists, because "don't cause a refresh stampede" was never written down anywhere it could read.

This is a specification problem. Both tickets described a desired outcome — fresher prices, correct invalidation — without naming who owns the shared invariant: exactly one refresh path per key per TTL window. Two agents independently satisfied their own tickets while jointly violating a rule that belonged to neither one. That's the pattern worth burning into memory: not a race condition, not a bad merge, but an unowned invariant sitting between two changes that are each individually correct.

Why the Obvious Fixes Don't Work

The reflex fix is a lock — only one agent touches the caching layer at a time, or agents merge one at a time with a human gate in between. Both work. Both also defeat the entire reason you're running more than one agent. The economic pitch for parallel agents is N agents finishing in roughly the time of one, as long as their work doesn't collide. The instant you force sequential merges on anything touching shared state, you've paid for N agents and bought the throughput of one, with coordination overhead stacked on top.

• ▹File locks: stop two agents editing the same file — but our incident had zero file overlap, so the lock never even engages.
• ▹Sequential merges: eventually surface the conflict (whoever merges second gets ambushed by CI or prod), but serialize every agent touching anything shared — the exact parallelism you're trying to buy back.
• ▹More code review: only catches this if the reviewer already knows the invariant exists and holds both diffs in their head simultaneously — a trick that stops scaling past a couple of agents or a handful of invariants.
• ▹Bigger test suites: each suite is written against its own agent's understanding of the system, so it will never test an invariant nobody told it to test.

The Actual Mechanism: Intent Declarations, Diffed Before Code Exists

What we landed on at PhoenixDX: before an agent writes a line of implementation, it publishes a short, structured intent declaration — what shared state it's about to touch, which invariant it believes it owns or depends on, and what it currently assumes to be true. The orchestrator's job is to diff intents, not code. Intent declarations are cheap to write, cheap to compare — small, structured, no AST required — and, most importantly, they exist before either agent has sunk hours into a build. A collision becomes a five-minute conversation instead of a 2am rollback.

• ▹Shared resource: the specific piece of state, service, or invariant being touched — e.g. "cache refresh path for pricing keys."
• ▹Claimed invariant: what the agent believes must hold true after its change — e.g. "at most one refresh per key per TTL window."
• ▹Assumed current state: what the agent believes is true right now, before its change — this is the field that actually catches collisions, because two agents holding contradictory beliefs about the same resource is the tell.
• ▹Blast radius: what else reads or writes this resource, as far as the agent can see from the code in front of it.

{
  "agent": "agent-a",
  "ticket": "reduce stale price reads",
  "touches": "pricing.cache",
  "claims_invariant": "cache never serves a value older than 2x TTL",
  "assumes_current_state": "only read path refreshes cache; no write-through refresh exists",
  "plan": "add stale-while-revalidate: serve stale, trigger background refresh on near-expiry"
}

{
  "agent": "agent-b",
  "ticket": "fix cache invalidation on price updates",
  "touches": "pricing.cache",
  "claims_invariant": "cache reflects latest price within 500ms of upstream change",
  "assumes_current_state": "only invalidation-on-write exists; no background refresh path",
  "plan": "add write-through refresh on upstream price change event"
}

Neither declaration is wrong on its own. But diff them and the collision surfaces in seconds: both target pricing.cache, both add a refresh path, and both assumed_current_state fields flatly deny that the other agent's plan exists. That contradiction is the whole bug, caught before either agent has written a test, let alone opened a PR.

Before/After: Where the Caching-Layer Collision Would've Been Caught

Two days before the incident, both tickets were already assigned and both agents were already planning. Had intent declarations been mandatory at that point, the orchestrator would've flagged the contradiction the moment both declarations named pricing.cache with clashing assumed_current_state fields — before either agent wrote a line of code, let alone opened a PR. The fix at that stage is nearly free: hand ownership of the refresh invariant to one agent, update the other's declaration to depend on it instead of quietly re-solving it. A Slack-message-sized conversation instead of a rollback plus a postmortem.

Tomorrow: Declarations Only Matter If Something Reads Them

An intent declaration nobody diffs is just a comment nobody reads. The mechanism only earns its keep if there's an actual protocol for agents — or an orchestrator — to negotiate when two declarations collide. That's exactly what Day 13 covers: negotiation protocols between agents.

Extend Your Knowledge

• ▹Pull up your last two agent-driven PRs that touched the same subsystem from different tickets. Write the intent declaration each agent should have published, and check whether they'd have collided.
• ▹Read up on invariant-ownership patterns from distributed systems — how event-sourcing and aggregate-root designs assign single-writer ownership per key, or how CRDTs solve the same problem differently by making concurrent writes commutative instead of exclusive. Same discipline, applied to agents.
• ▹If you're running an orchestrator — LangGraph, a custom framework, or a plain queue — find where a pre-execution 'plan' or 'intent' hook could live before code generation starts. That's where the diff has to happen.
• ▹Day 13 — negotiation protocols — is the natural next read once intent declarations are actually flowing. Without a negotiation step, declarations just pile up unread.