Agent dead letter queues

Day 28: Designing refusal back into LLM consumers (the agent dead letter queue)

An empty dead letter queue used to be good news. With an LLM as the consumer, it can mean the opposite. Queue systems have always relied on one safety net: bad input fails loudly. An LLM doesn't fail loudly. If you don't deliberately give it a way to say no, poison messages stop landing in your DLQ. They land in your warehouse instead, as clean-looking rows that are wrong.

The hook: six weeks of an empty DLQ

We had an LLM extractor that turned supplier invoices (PDFs and emails) into structured rows. Like every other consumer we run, it sat behind a queue with retries and a dead letter queue (DLQ). The DLQ stayed empty for six weeks, and we read that as good health.

Our monitoring didn't catch the problem. A finance analyst did. Some supplier totals were labelled AUD when the invoices were actually in EUR. We traced those rows back to the extractor and found it had not refused a single invoice in six weeks. Every truncated PDF, missing currency code and odd layout had come out as a confident, schema-valid row. Nothing had gone to the DLQ because nothing could get there. Plenty had gone wrong.

The rest of this lesson explains how that happens and walks through three patterns that fix it.

Dead letters in 90 seconds

If you skipped ops class, here's the whole idea:

• ▹Poison message: a message the consumer will never process successfully, however many times it tries. Malformed JSON, a missing required field, or a reference to a record that doesn't exist.
• ▹Retries: the consumer fails and the broker redelivers. That handles transient problems like timeouts or a locked row. It never fixes poison, because the problem is the message itself.
• ▹Parking: after N failed attempts the message moves to a separate queue, where it stops blocking work and stops burning compute. That might be a max receive count in SQS, a delivery limit plus a dead-letter exchange on RabbitMQ quorum queues, or a DLQ topic that your consumer or Kafka Connect writes to. Kafka itself has no built-in DLQ.
• ▹Then a human looks. Someone inspects the parked message, fixes the data or the code, and replays it.

This has worked for decades because of an assumption that rarely gets written down: deterministic consumers throw on bad input. Hand a parser 'amount': 'twelve' and it raises an error. The exception is the signal, and the whole retry-and-park machinery runs off it.

What changes when the consumer is an LLM

An LLM consumer breaks that assumption. Models are trained to be helpful, so a malformed message looks to them like an incomplete one, and incomplete things get completed. The model infers a currency from the bill-to address, guesses a date format from the locale, and picks one of two conflicting totals. Each guess is reasonable on its own. None of them raises an exception, so retries never fire and nothing reaches the DLQ.

Here's what that looked like for us. Below is the text the extractor received after an upstream PDF splitter had dropped page 2:

RECHNUNG Nr. 2024-0917
Lieferant: Brandt Industrietechnik GmbH
Rechnung an: Harbour Fit-Out Pty Ltd, Sydney NSW
Datum: 03.09.2024

Pos  Beschreibung              Menge   Betrag
1    Hydraulikventil HV-220    4       3.900,00
2    Versand                   1         920,00

Gesamtbetrag:                          4.820,00
-- Seite 1 von 2 --

The model returned this. It's valid against our schema and fully confident:

{
  "invoice_number": "2024-0917",
  "supplier": "Brandt Industrietechnik GmbH",
  "invoice_date": "2024-09-03",
  "currency": "AUD",
  "total": 4820.00
}

The currency was actually EUR. It appeared only on page 2, next to the IBAN and the 'Gesamtbetrag inkl. MwSt. EUR' line, and the model never saw that page. What it did see was a Sydney bill-to address and no currency code, so it filled the gap with the most plausible value. A deterministic parser would have thrown 'currency not found'. The model gave us AUD.

It also got a lot right. It read the German number format correctly (4.820,00 became 4820.00) and parsed the date as September 3, not March 9. All that competence on the other fields is what makes the one wrong field so hard to spot.

The inversion: worry that agents fail too rarely

The usual worry with deterministic consumers is that they fail too often: noisy DLQs, flaky parsers, alert fatigue. With agents it's the other way round, and the thing to worry about is failing too rarely. Real input always contains some unprocessable messages, such as truncated files, the wrong document type or contradictory fields. If your consumer reports none of them, it hasn't handled them. It has hidden them.

The validation work from earlier in the series stops helping at this point. Schema validation catches the wrong shape: a missing field, a string where a number belongs, an enum value that doesn't exist. It can't catch a plausible wrong value. 'AUD' is a valid ISO 4217 code and 4820.00 is a valid decimal, so the row passes every structural check you have. Keep validating, but remember it only covers shape. Covering meaning takes two more things: a producer that is allowed to say 'I don't know', and a check on the evidence it claims to have.

Pattern 1: refusal as a first-class tool

Refusal that only exists as free text ('I'm not sure about the currency...') gets lost, because downstream code parses the JSON and skips the hedge. Make refusal a tool instead, as explicit as the success path. It has three parts:

• ▹A closed set of reason codes. Alerts, dashboards and a weekly review can't run on free text. An enum keeps refusals countable.
• ▹A field and evidence. Each refusal names the field it couldn't fill and quotes the input that caused the problem, which lets a human triage it in seconds.
• ▹Forced tool choice. The model has to call either extract_invoice or cannot_process and can never answer in prose. That leaves no third path for a hedge to slip through.

REASON_CODES = [
    "MISSING_REQUIRED_FIELD",   # value not present anywhere in the input
    "AMBIGUOUS_VALUE",          # present, but more than one reading is plausible
    "CONFLICTING_EVIDENCE",     # two parts of the input disagree
    "TRUNCATED_OR_UNREADABLE",  # input is visibly cut off or garbled
    "UNSUPPORTED_DOCUMENT",     # not an invoice (credit note, statement, quote)
]

TOOLS = [
    {
        "name": "extract_invoice",
        "description": "Use ONLY when every required field is stated in the input. "
                       "For each field, quote the exact source text in `evidence`.",
        "input_schema": {
            "type": "object",
            "properties": {
                "invoice_number": {"type": "string"},
                "invoice_date": {"type": "string", "format": "date"},
                "currency": {"type": "string", "pattern": "^[A-Z]{3}$"},
                "total": {"type": "number"},
                "evidence": {
                    "type": "object",
                    "description": "field name -> verbatim quote from the input",
                    "additionalProperties": {"type": "string"},
                },
            },
            "required": ["invoice_number", "invoice_date", "currency", "total", "evidence"],
        },
    },
    {
        "name": "cannot_process",
        "description": "Use when any required field is missing, ambiguous or contradicted. "
                       "This is a correct, expected outcome, not a failure.",
        "input_schema": {
            "type": "object",
            "properties": {
                "reason_code": {"type": "string", "enum": REASON_CODES},
                "field": {"type": "string"},
                "evidence": {"type": "string", "description": "verbatim quote, or 'NOT FOUND'"},
            },
            "required": ["reason_code", "field", "evidence"],
        },
    },
]

SYSTEM = """You extract invoices into structured records.
Rules:
- Never infer a value from context. Currency is NOT implied by the buyer's address.
- If a required field is not stated verbatim, call cannot_process.
- Refusing is a correct answer. A wrong value is worse than no value.
- You must call exactly one tool. Do not reply in prose."""

# In the API call: tool_choice={"type": "any"}  -> model must pick a tool

def handle(tool_call, source_text, msg):
    if tool_call.name == "cannot_process":
        return route_to_dlq(msg, refusal=tool_call.input)
    inp = tool_call.input
    # Evidence check: catches values the model claimed but the input doesn't contain
    for field, quote in inp["evidence"].items():
        if quote not in source_text:
            return route_to_dlq(msg, refusal={
                "reason_code": "CONFLICTING_EVIDENCE",
                "field": field,
                "evidence": f"quoted text not in source: {quote!r}",
            })
    # Value check: the currency code itself must appear in its quote
    if inp["currency"] not in inp["evidence"].get("currency", ""):
        return route_to_dlq(msg, refusal={
            "reason_code": "MISSING_REQUIRED_FIELD",
            "field": "currency",
            "evidence": f"{inp['currency']!r} not in its quoted evidence",
        })
    return write_row(inp)

Two details in this code matter. First, the prompt says outright that refusing is a correct outcome. In our experience, models without that line lean towards guessing. Second, the evidence check is a cheap deterministic backstop, but checking that the quote exists in the source isn't enough by itself. The model could quote 'Sydney NSW' as its currency evidence, and that text really is in the source. So the code also requires the currency code itself to appear inside its quote. 'AUD' appears nowhere in our invoice, so the row goes to the DLQ even if the model ignores the prompt.

When to skip it: for genuinely free-form enrichment like summaries or tags, where close enough is fine, a refusal tool adds friction and no real safety. Use it anywhere a wrong value costs more than a missing one, which covers money, dates, identifiers, units and anything that joins to another table.

Pattern 2: the agent dead letter envelope

With a deterministic consumer, the parked payload is all you need to replay: same code plus same input gives the same result. That doesn't hold for an agent. Its output depends on the model and its version, the prompt, the tools it was offered, the context it retrieved and whatever upstream agents handed it. Suppose you replay next month on a newer model with an edited prompt and it succeeds. You've learned nothing about why it failed, and you may have hidden a bug that's still running in production.

So don't park only the message. Park the whole decision context:

{
  "envelope_version": 1,
  "message_id": "inv-7f3a...",
  "source": "supplier-email-ingest",
  "received_at": "2024-09-05T02:14:09Z",
  "payload_ref": "s3://dlq-raw/inv-7f3a.../original.pdf",
  "payload_sha256": "9c1e...",
  "model": { "provider": "anthropic", "id": "<exact model id>", "params": { "temperature": 0 } },
  "prompt": { "template_id": "invoice-extract", "sha256": "a41b..." },
  "tools_offered": { "sha256": "e02d..." },
  "retrieved_context": [ { "doc_id": "supplier-master/brandt", "version": "2024-08-30", "sha256": "..." } ],
  "upstream_handoffs": [ { "agent": "pdf-splitter", "version": "1.4.2", "output_sha256": "...", "note": "pages_emitted=1 of 2" } ],
  "tool_calls": [ { "name": "cannot_process", "input": { "reason_code": "MISSING_REQUIRED_FIELD", "field": "currency", "evidence": "NOT FOUND" } } ],
  "trace_id": "otel-5b9e...",
  "route": "refused_by_agent"
}

• ▹Hash, don't copy. Store hashes of the prompt and tool definitions and keep the full versions in a registry. The envelope stays small, and 'what changed?' becomes a diff.
• ▹Record upstream handoffs. In our incident the extractor wasn't the root cause. The splitter was. Without 'pages_emitted=1 of 2' in the envelope, you'd spend days tuning the wrong prompt.
• ▹Replay in two modes, and be honest about which is which. A pinned replay uses the same model, prompt and context, and tells you whether the failure reproduces. A current replay uses today's stack and tells you whether it's fixed. Report both.
• ▹Route records where the message came from. 'refused_by_agent', 'failed_evidence_check' and 'exception' mean different things, so keep them separate.

If you already emit OpenTelemetry traces for LLM calls, put the trace_id in the envelope and leave the spans where they are. The envelope is the index and the trace holds the detail.

Pattern 3: measure abstention

From the outside, a refusal tool the model never calls looks exactly like a pipeline with perfect input. Measuring is the only way to tell the two apart. Track the refusal rate per source and per reason code, and alert when it moves in either direction:

-- daily refusal rate per source and reason
SELECT
  source,
  date_trunc('day', received_at)                        AS day,
  count(*)                                              AS processed,
  count(*) FILTER (WHERE route <> 'written')            AS parked,
  round(100.0 * count(*) FILTER (WHERE route <> 'written') / count(*), 2) AS park_rate_pct
FROM extraction_outcomes
GROUP BY 1, 2;

-- alert rules (per source, rolling 7 days, minimum volume threshold):
--   park_rate_pct = 0       -> refusal path may be broken (prompt/model/tool change?)
--   park_rate_pct > baseline band -> upstream format change or a broken producer

Most teams leave out the zero-rate alert, and that's the alert that would have caught our six weeks. Give it a minimum volume so a source sending three invoices a week doesn't page anyone. You won't know the right rate on day one. Set a baseline from the first few weeks of real data, then tighten it.

Monitoring can only tell you something looks off. To prove the refusal path still works, use canaries: a small set of known-bad messages, one per reason code. Run them after every prompt edit, model upgrade or tool-schema change, and on a schedule in production:

CANARIES = [
    ("truncated_no_currency.txt", "MISSING_REQUIRED_FIELD", "currency"),
    ("two_totals_disagree.txt",   "CONFLICTING_EVIDENCE",   "total"),
    ("credit_note.txt",           "UNSUPPORTED_DOCUMENT",   None),
    ("date_01_02_no_locale.txt",  "AMBIGUOUS_VALUE",        "invoice_date"),
]

@pytest.mark.parametrize("fixture, code, field", CANARIES)
def test_agent_still_refuses(fixture, code, field):
    result = run_extractor(load(fixture))
    assert result.route != "written", f"{fixture} was accepted, refusal path is broken"
    assert result.refusal["reason_code"] == code
    if field:
        assert result.refusal["field"] == field

In production, tag canary messages so they stay out of business tables but still count in the refusal metrics. If a canary ever gets written as a row, treat it as a sev-2, because it means the system has stopped saying no.

Watch out for over-correction. After an incident it's tempting to tighten the prompt until the agent refuses anything slightly unusual, and then the DLQ floods. A high rate is a signal too, so look at which reason code is spiking. Usually it points to one fixable upstream issue or one rule that's too strict.

The weekly read: the DLQ as a requirements backlog

An agent DLQ is really a list of requirements nobody wrote down. A reason code that keeps coming back is the pipeline telling you about a case your design missed. Spend 30 minutes a week grouping parked messages by source and reason code, and turn each recurring cluster into one of three things:

• ▹A test. The message becomes a regression fixture or a new canary, so the behaviour is pinned down.
• ▹A validation rule, such as 'currency must appear verbatim in the source' or 'invoice date cannot be after received_at'. That moves a whole class of meaning checks into deterministic code.
• ▹A tool contract change. A new document_type field means credit notes get routed instead of refused. A currency_source field forces the model to say where the value came from.
• ▹Sometimes the fix belongs upstream, as it did with our splitter. The envelope's handoff fields show you when the extractor is the wrong place to fix things.

Over time, the DLQ should shrink for known reasons and only fill up with new ones. Tomorrow (Day 29) builds on this: these refusals and canaries become the seed of your evaluation set, and a shift in refusal rate is one of the earliest signs of drift in model behaviour, prompts or upstream data.

Monday checklist

• ▹Add the refusal tool: cannot_process with a closed set of reason codes, forced tool choice, and a prompt line saying refusal is a correct outcome. Require verbatim evidence on the success path and check it in code.
• ▹Add the envelope fields: model id and params, prompt hash, tool-definition hash, retrieved context versions, upstream handoffs, tool calls and trace id. Store them on every parked message, and on written ones too if you can afford the storage.
• ▹Add canary messages, one known-bad fixture per reason code. Run them in CI on every prompt or model change and on a schedule in production.
• ▹Add a zero-rate alert per source, over a rolling window, with a minimum volume. It should fire on zero as well as on a spike.
• ▹Book a 30-minute weekly DLQ review. Group by reason code and turn each recurring cluster into a test, a validation rule or a tool contract change.
• ▹Then ask your team: 'When did our agent last say no?' If nobody knows, that's where you start.

Extend your knowledge

• ▹In Enterprise Integration Patterns (Hohpe & Woolf), read 'Dead Letter Channel' and 'Invalid Message Channel'. Their split between undeliverable and invalid messages maps straight onto 'exception' versus 'refused_by_agent'.
• ▹Read your broker's DLQ docs (Amazon SQS redrive policy, RabbitMQ dead-letter exchanges, Kafka Connect error tolerance and DLQ topics) and decide where the agent envelope should live next to them.
• ▹Go through the OpenTelemetry GenAI semantic conventions, so the trace_id in your envelope points at spans that already record the model, its parameters and the tool calls.
• ▹For the research side, 'Know Your Limits: A Survey of Abstention in Large Language Models' (Wen et al., 2024) gives an overview of how abstention is elicited and evaluated in LLMs.