We Debugged the Wrong Agent for Two Days — The Real Bug Was a 'Harmless' Prompt Tweak

Day 4 — The Planner's Output Is an API Contract (Even Without a Schema)

Here's the version of this bug that'll ruin your week: the executor agent is failing in production, so you audit the executor. Its code is clean. Its tests pass. You read every failed run line by line. Nothing. Two days in, you finally find it — and it's not in the executor at all. It's three days upstream, in a planner prompt tweak nobody flagged as risky, because on the org chart, planners 'just produce a to-do list.' If you don't version your planner the way you version a service, this is the debugging session you're signing up for.

The two-day chase

Production failure rate on the executor agent started climbing. Obvious first move: the executor regressed. Git blame on its code — clean, nothing merged in two weeks. Test suite run in isolation — every case green. So we went manual: logging turned up, staring at individual failed runs one by one, hunting for a pattern in tool calls, retries, the prompt itself. Nothing held. Two days of this, and the executor kept passing every check we knew how to run.

The reveal

Turned out the executor was never the problem. Three days before the failure rate started climbing, someone had tweaked the planner's system prompt to make plans 'more efficient' — fewer, chunkier steps instead of many small ones. On paper it looked harmless: a smaller diff than most PRs, no new dependencies, nothing that screamed review-me-carefully. But the executor had been built and tested against one specific shape of input — small steps, each one a single clear action carrying enough context to run standalone. The new planner started handing it fewer, denser steps and assuming the executor would infer whatever context got left out. The executor's code hadn't moved an inch. The distribution of inputs it was written for had shifted out from under it.

Why nobody suspected the planner

The planner prompt change went through review — technically. But nowhere near the scrutiny a change to the executor's logic would've triggered. The unspoken mental model on the team: the planner 'just produces a to-do list.' It doesn't call tools, doesn't touch data, doesn't have side effects. The executor is where the real work happens, so that's where review rigor and test coverage piled up. Call it the invisible-dependency problem: the component that looks the most harmless is usually the one everything else is quietly standing on.

• ▹No obvious blast radius. It emits text, not actions, so it never triggers the review reflexes you'd apply to code that calls APIs or mutates state.
• ▹The output passed every schema check — valid JSON, right field names — while quietly violating a contract nobody wrote down: step granularity, ordering assumptions, how much context each step is supposed to carry.
• ▹Nobody had a test pinning the planner's output shape, because 'correct' meant 'this plan looks sensible to a human reading it,' not 'this plan matches what the executor was built to consume.'

Naming the pattern

Here's the pattern, named plainly: a planner's decomposition strategy — step count, granularity, what context is assumed present or absent, what ordering guarantees hold — is an implicit API contract. Every downstream agent consuming those steps is coupled to it, even though nothing enforces that coupling at the type level. A JSON schema tells you the plan is well-formed. It says nothing about whether the plan is shaped the way your executor expects. We learned this lesson decades ago in distributed systems: a service's response schema is a contract, you version it, you deprecate it, you announce breaking changes ahead of time. Multi-agent systems have identical coupling — but because the 'interface' is natural language instead of a typed schema, most teams don't reach for that discipline until it bites them.

The fix: version the planner like a service

• ▹Pin prompt text and model ID together as one versioned artifact — planner-v3, tied to a specific system prompt hash and a specific model (claude-opus-5 vs. claude-sonnet-5 are not interchangeable). Never let 'the planner' mean a moving target.
• ▹Keep a fixed eval set of representative planning tasks. Before any planner change ships, diff its outputs against the previous version on that set — not to judge 'is this plan good' but to catch 'did the shape change': step count distribution, average step length, context packed into each step.
• ▹A shape diff is a breaking-change signal, not a style note. If the diff is real, the executor's test suite needs new cases built against the new shape before rollout — not discovered after.
• ▹Make planner rollback independent of executor code. If production failures spike and the executor hasn't touched, reverting the planner to its last pinned version should be a one-line, low-drama operation — not a forensic investigation.

# planner.versions.yaml — the planner as a pinned, versionable artifact
planner:
  active_version: v3
  versions:
    v2:
      prompt_hash: sha256:8f2a1c...
      model: claude-sonnet-5
      eval_baseline: evals/planner_v2_outputs.json
    v3:
      prompt_hash: sha256:4b7e90...
      model: claude-sonnet-5
      eval_baseline: evals/planner_v3_outputs.json
      shape_diff_vs: v2
      shape_diff_report: evals/v2_vs_v3_shape_diff.md
      rollout_status: canary   # canary -> full -> or rollback to v2

The generalizable rule for Day 4

So here's the rule before you even open the debugger: before asking 'why did my agent fail,' check whether the plan it actually received matches the plan it was built and tested against. Any change to a planner — a prompt edit, a model swap, a temperature tweak, even an instruction change that looks 'minor' — is a breaking-change candidate for every downstream agent until you've diffed its output shape and proven otherwise. The planner isn't a harmless to-do list generator. It's upstream API surface, and it earns the same versioning discipline you'd give any service that other services depend on.

Bridge to Day 5

Everything above assumes a clean handoff: the planner produces a plan once, the executor runs it. Tomorrow that assumption breaks. We'll look at what happens — and what new failure modes show up — when the planner re-plans mid-execution based on the executor's intermediate results, instead of handing over a one-time contract up front.

Extend your knowledge

• ▹Pull up your own planner's prompt history — git log on the prompt file, or whatever your prompt-management tool tracks — and check: has it changed in the last month without a matching update to the executor's test fixtures?
• ▹Build a minimal shape-diff script: run your planner on 10-20 fixed tasks, log step count and average step length per version, diff it before your next prompt or model change ships.
• ▹Go read up on API versioning and contract testing — consumer-driven contract testing, Pact-style. The discipline maps almost one-to-one onto planner/executor coupling.
• ▹Running different models for planner and executor? Treat any model upgrade on either side — swapping in a newer Claude model, say — as a planned experiment with a rollback path, not a drop-in replacement.