Dead-letter queues

Sooner or later, every queue-based system receives a message it will never be able to process. What happens next depends on where that message goes and whether anyone notices. Handle it well and you have a contained failure. Handle it badly and you lose data slowly, without anyone knowing. Now that LLM agents consume from queues, the stakes are higher, because retrying a message no longer replays the same work.

The second incident

We had an outage on an event pipeline. A downstream service changed the shape of a payload and our consumer started throwing errors. We fixed it in under an hour, wrote a short postmortem, closed the ticket and moved on.

Weeks later, someone chasing an unrelated data mismatch opened the dead-letter queue and found a backlog of parked events. Some were from the outage window. The rest had been trickling in ever since from a second, quieter edge case that our fix never covered. No alert fired and nobody had looked at the queue, because nobody owned it.

The uncomfortable truth is that the system behaved correctly. The consumer retried, gave up and parked each message exactly as designed. The queue did its job and the team didn't do theirs. That was the second incident, and it was worse than the first because we only found it by accident.

Where this sits in the series

On Day 21 we made consumers retry with backoff. On Day 22 we made handlers idempotent so those retries are safe. Today's question: what do you do with a message that will never succeed, however many times you retry it?

Poison messages: the core concept

A poison message fails deterministically. Retrying won't help, because the problem isn't the network or a busy dependency. It's the message itself, or the way your code reads it.

• ▹Schema drift: a producer renamed or removed a field your consumer needs.
• ▹Unexpected enum: a new status such as 'PARTIALLY_REFUNDED' turns up, and your switch statement has never seen it.
• ▹Missing foreign key: the event points to a customer or order that doesn't exist, either not yet or not ever.
• ▹Payload over a size limit: the downstream API or database column rejects it on every attempt.
• ▹For agent consumers: an input that reliably makes the model emit an invalid tool call or break your output schema.

Without a DLQ, a poison message ends up in one of two places. In ordered systems such as a Kafka partition or a FIFO queue, it blocks everything behind it (head-of-line blocking). In unordered queues, it keeps cycling and eats compute and retry budget until it expires. On SQS, that means it's silently deleted when its retention period runs out. If nothing expires it, it cycles forever. A dead-letter queue (DLQ) gives the message somewhere to go: after N failed attempts it's moved aside, and the rest of the stream keeps flowing.

Look at where the diagram ends. It doesn't stop at the DLQ. If your mental model does stop there, all you've built is a slower way to lose data.

The mechanics in one screen

• ▹Amazon SQS: attach a redrive policy to the source queue with a deadLetterTargetArn and a maxReceiveCount. When a message has been received more than maxReceiveCount times without being deleted, SQS moves it to the DLQ. Built-in redrive (moving messages from the DLQ back to the source) is available in the console and the API.
• ▹RabbitMQ: set x-dead-letter-exchange, and optionally x-dead-letter-routing-key, on the queue. A message is dead-lettered when it's rejected without requeue, when it expires, or when the queue goes over its length limit. Quorum queues also support a delivery-limit. RabbitMQ adds an x-death header recording the reason and the count.
• ▹Kafka: the broker has no DLQ, so you build one yourself. Create a separate topic (for example orders.dlq), have your consumer publish failed records to it, then commit the offset. Kafka Connect gives sink connectors errors.deadletterqueue.topic.name, and Spring Kafka gives you DeadLetterPublishingRecoverer. Many teams put intermediate retry topics with delays in front of the final DLQ topic.

// SQS source queue: redrive policy
{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:ap-southeast-2:123456789012:orders-dlq",
    "maxReceiveCount": "5"
  }
}

// What your consumer should attach when it parks a message
// (SQS message attributes / Kafka headers / AMQP headers)
{
  "original_payload": "<unchanged bytes or pointer to blob storage>",
  "source_queue": "orders",
  "error_class": "SchemaValidationError",
  "error_message": "field 'currency' missing",
  "stack_trace": "<truncated, full trace in logs via trace_id>",
  "attempt_count": 5,
  "consumer_service": "order-projector",
  "consumer_version": "git:4f2a91c",
  "first_failed_at": "2026-10-13T02:14:07Z",
  "parked_at": "2026-10-13T02:19:55Z",
  "trace_id": "…"
}

A DLQ is only useful if the messages carry metadata. A bare payload with no error, no attempt count and no consumer version means whoever triages it has to reproduce the failure from scratch, and usually nobody does. Of all these fields, consumer_version matters most. It tells you whether the code that failed is still the code that's running.

One SQS trap has cost people real data. On standard queues, a message's retention clock starts from when it was first enqueued, not from when it moved to the DLQ. If the DLQ's retention period isn't longer than the source queue's, parked messages can expire before anyone looks at them. Set DLQ retention to the maximum of 14 days, and treat that as a deadline. It isn't storage.

The turn: DLQs fail organisationally, not technically

Back to the story. The redrive policy worked. The metadata was mostly there. The messages were intact. Three things were missing, and none of them was code.

• ▹No owner. The DLQ came from a Terraform module that every service used. It belonged to everyone, which in practice meant it belonged to nobody.
• ▹The wrong alert. We alerted on DLQ depth, with the threshold set for a burst. A few messages a day never came close to it. Depth measures volume, and our problem was neglect.
• ▹No runbook. Even once we'd found the backlog, nobody knew which messages were safe to replay, in what order, or into which version of the consumer.

Dead-lettering a message doesn't resolve the failure. It moves the failure out of your consumer and into your organisation. The three rules below are how the organisation catches it.

Rule 1: Alert on age, not just depth

One poison message that's 10 days old is worse than 500 that are 5 minutes old. The 500 are a live incident, and someone is probably already looking. The 10-day-old message is a customer order, a payment event or a compliance record that the business believes was processed. Age tells you how long you've been wrong without knowing it.

• ▹Set an SLO on the age of the oldest DLQ message, for example 'No DLQ message older than 24 hours on a business day.' Pick the number by asking how long that data can safely be wrong.
• ▹SQS publishes ApproximateAgeOfOldestMessage to CloudWatch. For Kafka, work it out from the parked_at header on the oldest unprocessed DLQ record, or from consumer lag in time if a triage consumer reads the topic. For RabbitMQ, add a timestamp header and a small exporter.
• ▹Keep a depth alert as well, tuned to catch the bursts that mean a live incident. Depth tells you something broke just now. Age tells you nobody is dealing with it.

# CloudWatch alarm: page the DLQ owner when the oldest message passes 24h
AlarmName: orders-dlq-oldest-message-age
Namespace: AWS/SQS
MetricName: ApproximateAgeOfOldestMessage
Dimensions:
  - Name: QueueName
    Value: orders-dlq
Statistic: Maximum
Period: 300
EvaluationPeriods: 1
Threshold: 86400        # seconds
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: [arn:aws:sns:...:orders-team-oncall]

Rule 2: The DLQ gets a named owner and a weekly triage slot

Engineering managers, this one's for you. A DLQ is a work queue for humans, and nobody works an unowned queue. 'Owned by the platform team' isn't ownership, and neither is 'alerts go to #pipeline-alerts'. Shared channels are where alerts go to get muted.

• ▹Each DLQ has one owning team, recorded in the service catalogue next to the consumer. Whoever owns the consumer owns its DLQ. The team that runs the broker doesn't.
• ▹The age alert pages that team's on-call like any other alert. If it doesn't seem worth paging for, change the SLO threshold. Don't send the alert somewhere quieter.
• ▹Book a recurring 30-minute triage slot each week. The on-call engineer goes through the DLQ, classifies each message, and either deals with it or files a ticket. If the DLQ is empty, the meeting takes 2 minutes. That's fine.
• ▹Track 'DLQ messages resolved vs. parked' as a team health signal. A DLQ that only ever grows tells you more about team capacity than any velocity chart will.

Rule 3: Replay only what's idempotent, and replay deliberately

Redrive is the most dangerous button in your messaging stack. Every message in a DLQ has already been at least partly processed, possibly more than once, and may have left side effects behind: a row written before the exception, an email sent, a downstream call made. This is where Day 22 earns its keep. Only replay a message type into a handler you've confirmed is idempotent for it.

Before you touch a parked message, put it in one of three buckets:

• ▹Fix the code, then redrive. The consumer was wrong, for example it didn't handle a new enum. Deploy the fix, check that the new consumer_version handles a sample, then redrive.
• ▹Fix the data, then redrive. The message was wrong or arrived too early, for example the foreign key didn't exist yet. Repair the data or create the missing entity first, then redrive.
• ▹Discard with a record. The message is obsolete, a duplicate, or invalid by design. Delete it, but write down what it was and why you dropped it, in a ticket, an audit log or the triage doc. 'Purged the DLQ' doesn't count as a record.

Never bulk-redrive into a consumer that hasn't changed. If the code and the data are the same as when the message failed, it'll fail again. You'll use up the attempt budget, maybe repeat side effects, and the messages will land back in the DLQ with fresh timestamps that hide how old the original problem is. Redrive a small sample, watch it go through, then do the rest.

The agent twist: a retry is a new decision

Everything so far assumes a deterministic consumer: same input, same code, same result. That assumption is what makes retries and redrives reasonable. If the consumer is an LLM agent, the assumption no longer holds.

• ▹Same context, same failure, more cost. If the model produced a malformed tool call because the prompt or the input was ambiguous, retrying with the same context usually fails the same way, and every attempt costs tokens and latency. Five retries on a large-context agent step can cost real money and achieve nothing.
• ▹A different outcome that nobody reviewed. Sampling isn't deterministic, and the agent's world has moved on since the first attempt: tool results, retrieved documents, even the date. A retry that 'succeeds' may do something the first attempt wouldn't have, like refund a different amount, classify the ticket differently or make a different file edit. Nobody approved that decision. It just went through.
• ▹Partial side effects stack up. Say an agent made two of its four tool calls before failing. On retry it plans again from scratch and may make those calls a second time. Idempotency keys on your tools matter even more here.

In practice, agent workers should retry less and dead-letter sooner:

• ▹Separate errors by class. Transient infrastructure errors, such as 429 rate limits, 5xx responses from the model API and tool timeouts, get normal backoff retries. Semantic failures, such as schema validation errors, invalid tool arguments, guardrail refusals and the agent giving up, get one retry at most, with changed context like the validation error fed back in. After that they go to the DLQ.
• ▹Park the full decision context, not only the input event. That means the rendered prompt or a pointer to it, the model ID and parameters, the version of the tool definitions, the tool calls made so far and their results, the raw model output that failed validation, and the trace ID from your agent observability tool (LangSmith, Langfuse, OpenTelemetry spans, or whatever you run).
• ▹Triage can be done by a human or a repair agent, but redrive stays deliberate. A repair agent can suggest a fix, such as patching the prompt, correcting extracted fields or flagging the input as out of scope. The redrive decision still goes through the three buckets from Rule 3, and a human approves anything with side effects.

Bonus habit: snapshot it, don't just drain it

Every message in a DLQ is a labelled edge case: an input your system has proven it can't handle, with the error attached. It's the most valuable test data you have, and most teams throw it away.

• ▹Before you purge, or after you redrive, copy the cleared items into a regression fixture set if the consumer is deterministic, or an eval set if it's an agent.
• ▹For agents, a parked message plus its triaged correct outcome is a ready-made eval case. Run it against every prompt or model change.
• ▹Scrub or tokenise personal data before anything goes into a test or eval store. DLQ payloads are production data, and they often carry exactly the fields your data-handling policy is written to protect.

Monday checklist

• ▹[ ] Every consumer has a DLQ (or DLQ topic), and its retention outlives the source queue's.
• ▹[ ] Consumers attach metadata on park: original payload, error class and message, stack trace or trace ID, attempt count, consumer version, timestamps. Agent workers also attach prompt, model, tool calls and raw output.
• ▹[ ] There's an alert on the age of the oldest DLQ message, with an explicit SLO, routed to on-call.
• ▹[ ] There's a separate depth or rate alert tuned for bursts.
• ▹[ ] Each DLQ has a named owning team in the service catalogue and a weekly triage slot.
• ▹[ ] A redrive runbook is documented: classify (fix code / fix data / discard with record), sample first, then batch.
• ▹[ ] Idempotency is confirmed for the handler before any replay.
• ▹[ ] Agent workers separate transient from semantic errors and cap semantic retries at 0–1.
• ▹[ ] Cleared items are snapshotted (PII scrubbed) into a regression or eval set before purge.

Bridge to Day 24

A DLQ deals with messages that fail. Tomorrow's problem is quieter: messages that succeed, but in the wrong order. Before then, go and open your DLQs. The queue is doing its job. Make sure someone is doing theirs.

Extend your knowledge

• ▹AWS documentation: 'Using dead-letter queues in Amazon SQS', plus the section on DLQ redrive. Read the retention and FIFO caveats carefully.
• ▹RabbitMQ documentation: 'Dead Letter Exchanges', including the x-death header and how delivery-limit works on quorum queues.
• ▹Uber Engineering blog: 'Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka'. This is the classic write-up on retry topics combined with DLQ topics.
• ▹Exercise: choose one consumer you own and find its DLQ, or confirm it doesn't have one. Check what's in it today, how old the oldest message is, and who would get paged. Then go through the Monday checklist for that consumer.