The Message That Vanished at 2:14pm — and Why Your 'Healthy' Queue Never Noticed

Message Queues Decouple Services. They Don't Decouple Debugging.

You added a queue to decouple two services. It worked great — right up until the day a message vanished and nobody could say which service ate it. That's the moment 'who broke this' stops having an answer without distributed tracing. And you're the one holding the pager when it happens.

The 2:14pm message that vanished

Here's the pipeline: an orchestrator published a 'process order' event to a queue, a worker agent picked it up, called an LLM to draft a fulfillment plan, and pushed the result to a notification service. At 2:14pm, one order just stopped. Not an error in the orchestrator. Not an error in the worker. Not an error in the notification service. The order sat in limbo, the customer never got a confirmation, and support escalated it four hours later. We spent the rest of that day grepping three separate log streams, trying to stitch together a sequence of events no single system had ever recorded end to end. The culprit, when we finally found it: the worker's LLM call had returned an empty plan — not an error, just an empty JSON object — and the code acked the message and moved on like nothing happened. Nothing crashed. Nothing alerted. The message wasn't lost. It was quietly murdered by a success path that never checked its own output.

Why this is Day 7, not Day 1

On Day 6 you learned decoupling buys you independent scaling, deploying, and failing — one service can crawl or crash without taking the other down with it. That's the payoff the queue gets sold on. What nobody puts on the slide is the flip side: you traded one connected system for two systems that no longer share a call stack, and every debugging session from here on out starts with re-establishing a link that used to come for free.

The mechanism: you traded a stack trace for a gap

A synchronous call — a function call, an HTTP request, an agent calling a tool in-process — hands you a stack trace for free. Cause and effect share one execution context: if B throws, you can see exactly which call from A triggered it, arguments and all. A queue snaps that context in two. Producer A writes to a topic and walks away; it has no idea what happens downstream. Consumer B reads on its own schedule, in its own process, into its own log stream. There's no shared stack anymore — just two independent logs and a gap where the causal link used to live. In an agentic pipeline that gap is worse, because the thing sitting in the middle is often a non-deterministic LLM call: the same message can come back as a valid plan, an empty plan, or a malformed one on different runs — and none of those necessarily throws.

'Is the queue healthy' is not 'where did this message go'

When we finally sat down to fix this properly, we realized our dashboards had been answering a question nobody was asking. Consumer lag, queue depth, DLQ counts, error rate per consumer — all green, the entire time. Those metrics tell you the queue as a system is healthy: throughput's fine, nothing's backing up, nothing's getting rejected. They tell you nothing about the fate of one specific message. That's a per-message lineage question, and it needs a completely different kind of instrumentation than aggregate queue health. Most teams — us included, that day — build the first kind because the queue vendor hands it to you out of the box, and skip the second because it has to be designed, not switched on.

• ▹Queue health metrics — depth, consumer lag, throughput, DLQ count, error rate per consumer — tell you if the pipeline is moving.
• ▹Message lineage — where one message went, what each hop did with it, why it stopped — tells you what happened to one customer's order.
• ▹A fleet dashboard reading '0 errors, lag near zero' can be true at the exact moment a specific LLM tool call silently returned garbage that a downstream consumer swallowed without complaint.

What actually closed the gap: correlation IDs, designed in up front

The fix wasn't a smarter dashboard. It was threading a correlation ID — a trace ID — through every hop of the message's life, and logging it at every boundary: publish, receive, ack, publish-downstream. Once every log line for a message carries the same ID, you can reconstruct the causal chain across services after the fact, the same way a stack trace would've handed it to you for free in a synchronous call. This has to be baked into the message schema and the logging convention before the queue ships. Retrofit correlation IDs mid-incident and you're adding logging to a system that's already on fire, for a message that's already gone — you get the fix in time for the next incident, not this one. In agentic systems it matters even more: the trace ID has to survive the LLM call itself, so if you're logging the prompt/response pair, tag it with the same ID as the message envelope. OpenTelemetry's messaging semantic conventions and the W3C Trace Context spec exist for exactly this — a `traceparent` header rides in the message attributes, not the body, so every consumer can pick it up without parsing payloads.

// message envelope — trace context travels with the message, not buried in it
{
  "headers": {
    "trace_id": "abc123",
    "span_id": "span-worker-1",
    "parent_span_id": "span-orchestrator"
  },
  "body": {
    "order_id": "ord-9981",
    "payload": { "...": "..." }
  }
}

// every hop logs the same trace_id, on receive AND on the outcome
log.info("received", { trace_id: msg.headers.trace_id, hop: "worker" })
const plan = await callLLM(msg.body)
if (!plan || plan.steps.length === 0) {
  log.error("empty_plan_from_llm", { trace_id: msg.headers.trace_id })
  // nack or route to DLQ instead of silently acking
}

The reframe: eventual delivery is not traceable delivery

A queue promises eventual delivery — durability, retries, at-least-once semantics. It does not promise you'll be able to explain what happened to any given message. Those are two separate engineering problems with two separate mechanisms: durability comes from the broker — replication, acks, DLQs. Traceability comes from correlation IDs and structured logging that you design and own. The lost day happened because we'd conflated the two — we assumed a healthy, durable queue was automatically a debuggable one. It isn't. A message can be delivered exactly as promised and still be functionally lost, because 'delivered' and 'delivered in a way you can trace' are different guarantees.

The gut check before you ship a queue

Before you put a queue between two services — or between an orchestrator and a fleet of agents — ask one question: if a message disappears here, can I find out why in under an hour, or am I signing up for a full day of grepping logs across three services? If it's the second one, you don't have a queue problem yet. You have a tracing gap. And it will find you at 2:14pm on some perfectly ordinary Tuesday.

Extend your knowledge

• ▹Read the OpenTelemetry Semantic Conventions for Messaging Systems — the standard for how trace context (trace_id, span_id) should ride in message attributes across Kafka, SQS, RabbitMQ, and similar brokers.
• ▹Read the W3C Trace Context specification (`traceparent` header) — the interoperable format correlation IDs are increasingly built on, so tracing survives across services written in different languages or frameworks.
• ▹Martin Kleppmann's 'Designing Data-Intensive Applications', the chapter on stream processing and message queues, for the delivery-guarantee semantics (at-least-once vs. exactly-once) that correlation IDs sit on top of.
• ▹Running agent fleets on queues? Check whether your worker's LLM-call logging carries the same trace ID as the queue message envelope — that's the specific gap most teams miss first.