Your Agents Aren't Flaky — They're Breaking an API Contract Nobody Wrote

Day 10: The Inter-Agent Contract Problem

One agent hands another a JSON blob. That blob is an API contract — undeclared, unversioned, and unvalidated in most pipelines shipping right now. Skip that discipline and every 'flaky multi-agent bug' sitting in your issue tracker is actually a silent breaking change nobody caught.

The postmortem that wasn't about the model

A team I know burned three weeks chasing a 'flaky' extraction-to-routing pipeline. Agent A pulled order data out of a support ticket and handed Agent B a JSON object; Agent B routed it to a fulfillment queue. About 1 order in 10 vanished — no error, no crash, just gone. They tuned the prompt. They swapped models. They bolted on retries. The actual cause: someone had edited Agent A's prompt to handle a gift-order edge case, and on that one path it now emitted 'customer id' with a space instead of 'customer_id'. Agent B's router did order.get('customer_id'), got None back, and kept going without complaint. No exception, no log line — just an order with no owner. Three weeks of model debugging, and the bug was a renamed field.

Name the pattern: nobody owns the seam

In a single-agent app, this bug barely exists. One prompt, one parser — if the shape drifts, the same code that emits the JSON also consumes it, so it breaks immediately and loudly. Multi-agent systems split that into N independently-evolving authors of the same message, often different prompts, sometimes different models, sometimes different teams, and nobody explicitly owns the contract between them. The message shape ends up as tribal knowledge baked into prompt wording, not something anyone versioned or reviewed.

Why it hides as 'flakiness'

The failure looks probabilistic because the LLM usually still emits the old shape out of habit — trained behavior, few-shot anchoring, whatever you want to call it. It only drifts on certain inputs, certain temperature settings, certain prompt-edit ripple effects. So it shows up as a 9-out-of-10 success rate, and a 9/10 success rate reads like a model reliability problem. You reach for retries and self-consistency checks instead of the actual bug: there's no enforced shape at the boundary, so 'usually correct' is quietly doing the job your validation layer should be doing.

This is not a new problem — it's a rediscovered one

Distributed systems solved this over a decade ago: schema registries, IDL contracts, API versioning. Avro and Protobuf schemas in Kafka, gRPC's .proto contracts, REST's OpenAPI specs. The lesson never changed — once two independently-deployed processes exchange structured data, someone has to own the shape, evolve it on purpose, and reject anything that doesn't match. Multi-agent LLM pipelines are rediscovering that lesson one production incident at a time — except now the 'independent process' emitting the message is a prompt, which is far easier to edit by accident than a compiled service ever was. That's the part of this I actually study: in multi-agent research, coordination failures between autonomous components are treated as a first-class problem, not an afterthought. Production teams are still catching up to that framing.

The fix: treat the handoff like an API

• ▹Define the message as a typed schema — pydantic, zod, or plain JSON Schema — not a free-floating dict.
• ▹Give it an explicit version field (order.extracted.v2), not one you're inferring from prompt wording.
• ▹Validate at the boundary, the instant the message crosses from Agent A to Agent B — not three hops downstream where it's anyone's guess which agent caused the drift.
• ▹Fail loud on mismatch: raise, alert, dead-letter the message. A silently-defaulted field is worse than a crash, because a crash actually gets fixed.
• ▹Version bumps are a two-sided, deliberate change — the producing prompt and the consuming parser move together, reviewed together.

Here's the same 2-agent handoff, undeclared vs. contracted:

# BEFORE: the blob has no owner
def extract_order(llm_output: str) -> dict:
    return json.loads(llm_output)

def route_order(order: dict):
    customer_id = order.get("customer_id")  # None if the key drifted — no error
    queue.push(customer_id, order.get("line_items", []))

# The prompt for extract_order gets edited for a gift-order edge case.
# It now sometimes emits "customer id" (space) instead of "customer_id".
# Nothing breaks. Orders just quietly lose their customer_id 1 time in 10.

# AFTER: the handoff is a versioned, typed contract
from typing import Literal
from pydantic import BaseModel, ValidationError

class LineItem(BaseModel):
    sku: str
    qty: int

class OrderExtractedV2(BaseModel):
    schema_version: Literal["order.extracted.v2"] = "order.extracted.v2"
    customer_id: str
    line_items: list[LineItem]

def extract_order(llm_output: str) -> OrderExtractedV2:
    return OrderExtractedV2.model_validate_json(llm_output)  # raises on mismatch

def route_order(order: OrderExtractedV2):
    queue.push(order.customer_id, order.line_items)

# The moment the prompt drifts to "customer id", model_validate_json raises
# immediately, at Agent A's output — not silently at Agent B, three steps later.

Notice what changes when the next incident hits. With the undeclared dict, you're grepping logs across three services trying to find where a None first showed up. With the versioned contract, the validator tells you exactly which agent, which version, which field broke — the moment it happens, at the source. That's the entire payoff of the discipline: it turns a distributed guessing game into a single stack trace.

Where this sits in the 30-day arc

Today's lesson is the load-bearing layer under tomorrow's topic. Everything you're about to learn about orchestration, retries, and multi-agent failure recovery assumes the messages agents exchange are trustworthy in shape. Skip contract discipline today, and tomorrow's retry logic will faithfully retry a malformed request, your fallback agent will faithfully consume a blob with the wrong fields, and your error handling will catch exceptions that only exist because nobody validated the handoff in the first place. Contracts aren't a nice-to-have bolted on later — they're the foundation the rest of the coordination layer stands on.

Extend your knowledge

• ▹Read the pydantic docs on model_validate_json and strict mode — fastest way to bolt a hard validation boundary onto an existing Python agent pipeline.
• ▹On TypeScript, look at zod's .parse() vs .safeParse() — same fail-loud vs. silent-default distinction, same tradeoff.
• ▹Read the schema evolution chapter in Kleppmann's 'Designing Data-Intensive Applications' — the clearest existing treatment of exactly this problem, just not written with agents in mind.
• ▹Look at how Confluent's Schema Registry enforces backward/forward compatibility rules on Avro schemas — the same compatibility-mode thinking (backward, forward, full) maps directly onto versioning your agent message contracts.