Your Agent Picked the Right Tool Every Time — and Production Broke Anyway

Your agent calls the right tool. Every time. Tool-selection accuracy sits near 100% on your dashboard. And production still breaks — because nobody's checking whether "In Transit" and "in_transit" mean the same thing to your parser until the string is already sitting downstream, failing silently or loudly, several services past the point where the mistake was made.

The agent that never picked the wrong tool

At PhoenixDX we ran an agent that, by every metric we tracked, picked the correct tool for the job almost every single time. Tool selection was never the bottleneck. It still took down production. Every incident retro opened with the same line: "but it called the correct function." Correct function, wrong shape. That's the failure this lesson is about — and once you've shipped more than a toy agent, you'll see it far more often than a genuine tool-selection miss.

A schema attached to a function is a type signature — enforced at the wrong stage

Write a JSON schema for a tool and you've written a type signature. Required fields, enums, nesting, formats — that's the exact same job a function signature does in a statically typed language. What's different is when it gets enforced. A compiler stops you before your code runs; it catches the mismatch the second you try to compile. A JSON schema attached to a tool call gets checked after the model has already finished generating tokens — at parse time, once the string is fully formed and out the door. There's no compiler sitting inside the generation loop. The model produces a token sequence that looks like valid JSON matching your schema, and nothing stops it from being subtly wrong until something downstream actually tries to parse and use it.

A trenchcoat bug, straight from production

Here's one that slipped through code review because it read like a prompt problem, not a schema problem. We had an update_shipment_status tool. The schema required status as a lowercase enum ("in_transit", "delivered", "delayed") and eta as an ISO 8601 date. The model called the right tool, with the right intent, and handed back this:

// what the schema declared
{
  "name": "update_shipment_status",
  "parameters": {
    "status": { "enum": ["in_transit", "delivered", "delayed"] },
    "eta": { "type": "string", "format": "date" } // ISO 8601 expected
  }
}

// what the model actually emitted
{
  "status": "In Transit",   // wrong casing — enum mismatch
  "eta": "09/08/2026"        // looks like a date, wrong format
}

Both fields are, to any human skimming a log, obviously what was meant. Read that tool call out loud and you'd say "yeah, that's in transit, that's a date." A strict parser downstream didn't agree on either count — and it only threw on a fraction of calls, because most of the time the model happened to land on the exact casing and format we expected. That's what made it look like the model's reasoning was flaky. It wasn't. It was a validation gap: nothing checked the shape before the string left the model's context, and our downstream parsing was loose enough to sometimes coerce it and sometimes choke. That inconsistency is exactly what made it read as "maybe the instructions aren't clear enough" instead of what it actually was — a missing type check at the one point where a type check would have caught it, deterministically, every time.

Why this looks like flaky reasoning but is structurally a type error

Next-token generation has no side channel to a type checker. The model is predicting tokens conditioned on your schema description — it isn't evaluating that schema the way a constraint solver would. Even function-calling modes that bias sampling toward valid JSON structure mostly guarantee structural validity, not semantic conformance: enum values, date formats, nested-object-vs-string, field presence under conditional requirements — none of that is covered by "the JSON parses." Structurally valid JSON is not the same thing as schema-valid arguments. The model is guessing at compile time with no compiler in the loop, and it guesses right often enough that the failures feel random instead of what they are: systematic.

The tell: type error vs. real reasoning failure

You can tell these two failure modes apart in your logs without guessing — as long as you bucket failures correctly instead of dumping everything into "agent got confused."

• ▹Same tool name across every failing call, not a different one — rules out a selection or routing problem
• ▹Param shape drifts call to call for the same intent — casing, nesting, format — while the meaning stays right
• ▹Fields flicker in and out — optional vs. required fields showing up inconsistently
• ▹Retry the identical prompt and it sometimes succeeds with the exact same reasoning trace — the intent never changed, only the serialization did
• ▹A real reasoning failure looks nothing like this: wrong tool chosen, wrong entity referenced, wrong plan entirely — the content is wrong, not the container

The fix is runtime validation as a contract, not better prompting

Prompting can nudge probability mass toward the right shape. It can't guarantee it — you're still generating without a compiler. The real fix is putting an actual type checker at the boundary and treating it as part of the calling contract, not an afterthought: validate every tool call against its schema in strict mode (Pydantic, Zod, Ajv, whatever fits your stack) before execution. When it fails, don't surface that to the user, and don't just quietly drop the call. Feed the validator's error back to the model on the next turn — the same way a compiler error gets fed back to a developer — and let it regenerate. Cap the retries (2-3 is usually plenty) and log every single rejection. That log is your best signal on schema-authoring quality, not model quality.

try:
    validated = ToolSchema.model_validate_json(raw_args)  # strict
except ValidationError as e:
    # feed it back like a compiler diagnostic, not a user-facing error
    messages.append({
        "role": "tool",
        "content": f"Invalid arguments: {e}. Regenerate matching the schema exactly."
    })
    retry_count += 1
    if retry_count <= MAX_RETRIES:
        continue  # let the model retry with the error in context
    else:
        escalate_to_human_or_fallback()
else:
    result = call_tool(name, validated)

One more lever worth pulling: tighten the schema itself. Enums instead of free-text strings. Explicit ISO formats instead of "a date." Nested objects instead of stringly-typed composites. Every ambiguity left in the schema is a shape the model is free to guess wrong. Stack a stricter schema on top of a reject-and-retry loop and an intermittent production bug turns into something bounded, observable, and self-correcting.

Where this fits in the course

Day 3 covered how a model calls a function at all — the mechanics of tool definitions and invocation. Today is about why a call that mechanically works can still be wrong: the schema is a type system enforced one step too late, and the fix is validation with feedback, not prompt tuning. Day 5 picks up right after — what happens when a correctly-shaped call still isn't enough, and a validated call gets retried into idempotency and side-effect problems instead of shape problems.

Extend your knowledge

• ▹Run your existing tool schemas through a strict validator (Pydantic, Zod, or Ajv in strict mode) against last week's logged tool calls, and count how many past incidents it would have caught before execution.
• ▹Read Anthropic's tool use documentation on schema definitions and forced tool choice — see what structural guarantees you already get for free, versus what you still have to validate yourself.
• ▹Compare it against OpenAI's structured outputs / strict JSON schema mode — a different point on the same spectrum, pushing validation earlier into generation instead of after it.
• ▹Add a log dimension that buckets tool-call failures into "same tool, different shape" versus "different tool entirely," so you can quantify how much of your failure budget is type errors versus real reasoning errors.