The Ticket-Routing Bug That Never Threw an Error

Day 1: A Prompt Is a Spec, Not a Suggestion

Here's the uncomfortable part: your agent can run flawlessly, every single call, zero exceptions, and still be wrong for a week before anyone notices. No stack trace. No red build. Just a slow, silent drift that only a human catches, downstream, by accident. If you can't write a pass/fail test against your prompt, that's not a spec — it's a well-worded guess, and it will break exactly this way.

The hook: silent drift

At PhoenixDX we ran an agent that triaged incoming support tickets and routed them to the right team. Same prompt. Same model version. Same code path, untouched. And one week, routing accuracy just... dropped. Nothing paged us. Nothing failed a build. Someone on infra finally asked why billing tickets kept showing up in their queue. That's it — that's how we found out. The prompt looked identical on every single run; it just wasn't right anymore, and we had no instrument in the system capable of telling us that.

The tell

There's one question that separates a real spec from well-phrased hope: what would make this output wrong? If your team can't answer that in a single sentence, you don't have a spec — you have vibes with good grammar. It's the exact same discipline as a unit test: no failing condition, no way to detect the failure.

Traditional spec vs prompt-spec

A compiler fails loud. Wrong type, red squiggly, build breaks, you fix it before it ever ships. A prompt failure is the opposite of loud — it's quiet, plausible, and dressed up in perfect grammar. The model still hands you a confident, well-formed answer. It's just maybe not a correct one. That's the actual danger with LLM systems: the failure mode isn't a crash, it's a convincing wrong answer that sails through code review because there's nothing there to review it against.

• ▹A traditional spec throws or refuses to compile the moment a precondition is violated — you see the failure right at the boundary.
• ▹A prompt-spec done wrong just produces fluent, structured, wrong output — nobody notices until someone downstream does.
• ▹With a traditional spec, the compiler and runtime decide correctness for you.
• ▹With a prompt-spec done right, correctness is whatever assertion you bothered to write — the model won't hand you that for free.

The fix: preconditions, postconditions, failure modes

Take any vague instruction and split it into three honest parts: what the input has to satisfy before you even try (preconditions), what the output has to guarantee if it succeeds (postconditions), and what the model should do when neither can be met cleanly — an explicit failure mode, not a shrug and a guess. Here's the actual before/after from our ticket-routing agent.

BEFORE (a suggestion, not a spec):
"Classify this support ticket and tell us which team should handle it."

AFTER (a testable spec):

Preconditions:
- Input is a non-empty ticket body in English or Vietnamese.
- If the ticket body is empty or unintelligible, do not guess a team.

Postconditions:
- Output is valid JSON: {"team": <enum>, "justification": <string>}
- team is exactly one of: "billing", "infra", "product", "security", "needs_human_review"
- justification is <= 20 words and references specific text from the ticket.

Explicit failure mode:
- If the ticket plausibly belongs to two or more teams, output "needs_human_review" — never pick the more likely one silently.

Look at what actually changed. The vague version asked the model to be smart. The rewritten version tells it precisely what a correct answer looks like — and, this is the important bit, gives it a legal way to say 'I'm not sure' instead of quietly picking its best guess. That third piece, the explicit failure mode, is almost always the one teams skip. It's also the exact piece that was breaking our routing without a sound.

Turn it into a test

A prompt-spec isn't real until you can run it as an assertion. Here's the bar: can you diff two runs of the same prompt and get a deterministic pass or fail — not a feeling, an actual boolean? Below is the one-line check we now run against every routing response before it ever reaches a team's queue.

VALID_TEAMS = {"billing", "infra", "product", "security", "needs_human_review"}

assert (
    result["team"] in VALID_TEAMS
    and len(result["justification"].split()) <= 20
    and result["justification"] != ""
), f"prompt-spec violated: {result}"

Today's exercise

• ▹Pick the riskiest prompt you've got running in production — the one you'd hate most to explain drifting to your team.
• ▹Write the assertion first. What specific, checkable thing makes this output wrong? Not 'it should be good' — an enum check, a length bound, a required field, a phrase that should never appear.
• ▹Run your current prompt against that assertion on 5-10 real inputs and watch it fail somewhere you didn't expect.
• ▹Rewrite the prompt with explicit preconditions, postconditions, and at least one named failure mode — then don't stop until it passes your own assertion, consistently, not just once.

What's coming Day 2

Running your assertion by hand against 5-10 examples is fine for today. It's not fine forever — you need it firing automatically, every time the prompt changes or the model gets swapped out from under you. Day 2 is about building that harness: the thing that runs your assertions on every prompt edit, the same way CI runs your unit tests on every commit.

Extend your knowledge

• ▹Try promptfoo (open-source) and turn the assertion you wrote today into an actual test file you can run across prompt versions.
• ▹Read Anthropic's prompt engineering docs on being explicit and specifying output format — same preconditions/postconditions idea, just from the model provider's side of the fence.
• ▹If your team's ready to track pass rates over time instead of a single pass/fail, look at DeepEval or Braintrust.
• ▹Worth knowing: this maps almost one-to-one onto Design by Contract (Bertrand Meyer) — preconditions, postconditions, invariants for code, just aimed at a prompt instead of a function.