The Blackboard Bug That Sails Through Code Review Every Time

Day 9 — Blackboard Corruption: The Bug That Never Throws

Here's the failure mode that should worry you more than a stack trace: your agents can corrupt shared state and nothing will ever tell you. No exception. No red dashboard. No retry in the logs. Just a wrong answer that reads exactly like a right one. If nothing threw, you should assume you still have a bug — you just haven't found it yet.

The incident

A PR passed review at PhoenixDX last month. Two agents in a planning pipeline had both written to the same blackboard slot — call it plan.next_steps — close enough in time that neither had seen the other's write. No exception. No merge conflict. No retry logged. What came out the other end was fully coherent English: a clean, well-formed plan step. It just called a function that one of the two agents had deleted three turns earlier. The planner had read the slot before the delete landed, then written its update after — silently clobbering it. A human had to diff the plan against the actual repo to catch the ghost reference. Nothing flagged it, because nothing was watching for it.

What actually happened underneath

Call this a race condition and you'll miss the point. A classic race condition announces itself — a crash, a dropped write, a retry storm, something your dashboard lights up red for. This was quieter than that: a semantic overwrite. Two writes, each individually valid, each syntactically clean, landed in the wrong order and produced a third thing that looked valid and was false. That's worse than a crash. A crash tells you where to look. A semantic overwrite tells you everything is fine.

30-second anchor: why shared state exists at all

Days 1-8 covered agents talking to each other directly — roles, messages. A blackboard (or any shared-state store: a plan doc, a task graph, a shared memory object) exists so agents stop having to message each other for every fact. Agent A posts a belief, Agent B reads it whenever it needs to, and no router has to shuttle every update through itself. That's the whole appeal. It's also the whole risk — 'anyone can read, anyone can write' is precisely the condition under which two agents quietly disagree and nobody notices.

The reframe

Every framework you'll evaluate — LangGraph state, AutoGen's shared context, a homegrown Redis blackboard — will happily tell you the shape of the data. Almost none of them answer the question that actually matters: which agent is allowed to overwrite which entry, and does the agent whose belief just got overwritten ever find out. That second half is the part people skip. Define ownership all you want — if the overwritten agent has no way to learn its belief was invalidated, it keeps acting on stale state until something downstream breaks.

The one-sentence test — run it today

For every entry your agents write to shared state: can you say, in one sentence, who owns it and who is allowed to invalidate it? Try it on your own blackboard right now:

• ▹"plan.next_steps is owned by the Planner agent; the Executor may only append status, never rewrite the step list."
• ▹"code_map.symbols is owned by whichever agent last ran static analysis; any write must carry the commit SHA it was computed against."
• ▹If your honest answer is 'whoever gets there first' or 'last write wins' — that's not an answer. That's the bug, waiting.

Two guardrails that would have caught the PhoenixDX incident

• ▹Versioned claims: every write carries the version (or hash) of the state it was based on. If the current slot version doesn't match what the writer last read, the write gets rejected or forced to re-derive. Optimistic concurrency control, just applied to beliefs instead of database rows.
• ▹Explicit ownership tags: each entry declares which agent (or role) is the writer of record. Everyone else's write to that entry is either forbidden or downgraded to a 'proposed update' the owner has to ack.
• ▹Neither one needs a new framework. They're a few extra fields on the entry and a check before you accept a write — the kind of thing you'd bolt onto any shared cache in a normal distributed system, just pointed at agent state instead of application state.

// blackboard entry — before (the PhoenixDX bug shape)
{
  "key": "plan.next_steps",
  "value": ["call parse_invoice()", "..."]
}

// after — versioned + owned
{
  "key": "plan.next_steps",
  "value": ["call parse_invoice_v2()", "..."],
  "owner": "planner-agent",
  "based_on_version": 14,
  "version": 15
}

// write rule any agent must pass:
function acceptWrite(entry, incoming) {
  if (incoming.based_on_version !== entry.version) {
    return reject("stale write — reread and reconcile");
  }
  if (incoming.owner !== entry.owner && !entry.allowExternalPropose) {
    return reject("not the owner — submit as proposal");
  }
  return commit(incoming);
}

Close: the uncomfortable corollary

Once you've assigned ownership per entry and decided who can invalidate whom, look at what you've actually built: a set of rules for whose write wins and who gets told. That's an arbiter. Shared state didn't remove the need for orchestration — it just let you put off designing it, until a silent corruption bug forced the question anyway. Day 10 is about building that arbiter on purpose, before it builds itself by accident.

Extend your knowledge

• ▹Read up on optimistic concurrency control (the compare-and-swap / version-check pattern used in databases like DynamoDB and Git's own ref updates) — it's the same mechanism this lesson applies to blackboard writes.
• ▹Look at how LangGraph and AutoGen expose shared state today, and check specifically whether they give you any built-in ownership or version-check hook, or whether you'd have to add it yourself.
• ▹Try the one-sentence ownership test on a real pipeline you run: pick one shared-state key and write down, in one sentence, its owner and invalidation rule. If you can't, that's your first guardrail to add.
• ▹Preview for Day 10: read up on the 'arbiter' or 'coordinator' pattern in classic blackboard-architecture AI systems (e.g., the Hearsay-II speech system) — the ownership rules you add today are the seed of that role.