The Queue Dashboard Was Green for Six Hours While Your Backlog Rotted

Day 15: A Queue Decouples Deploy Timing, Not Failure

Add a message queue and you buy exactly one thing: the producer and consumer stop having to run on the same clock. That's it. A queue does nothing to protect you from a consumer that quietly dies — and worse, it will hide that death from you, cheerfully, for as long as it takes the backlog to become unrecoverable.

The dashboard said everything was fine

Broker dashboard: green. Alerts: zero. Meanwhile six hours of messages piled up behind a consumer group that had died without telling anyone. Nobody got paged, because the only thing anyone was watching was whether the broker itself was up — and it was, the whole time. The broker did its one job perfectly: accept messages, hold onto them. It just had nobody left to hand them to.

Replay the timeline

• ▹02:14 — a downstream deploy triggers an OOM in the consumer process; the orchestrator restarts the container, but it rejoins the consumer group in a crash loop that never stabilizes
• ▹02:14–08:00 — the producer has no idea and no reason to check; it keeps publishing at completely normal rate
• ▹Queue depth climbs hour over hour, disk usage on the broker climbs with it — nothing pages, because uptime and liveness checks on the broker itself are all green
• ▹08:03 — a customer support ticket asks why an order confirmation never arrived; on-call greps logs and finds the consumer group has zero active members
• ▹08:20 — consumer is restarted and immediately has to process six hours of backlog, which is where the second incident starts

The misconception nobody said out loud

In the design pitch, "the queue decouples the services" meant one thing: producer and consumer can deploy on different schedules, run different stacks, scale independently. Nobody meant "if the consumer dies, the producer's world is unaffected." Those are two separate claims, and the pitch only ever made the first one. A queue buys you deploy independence. It does not buy you failure independence — it just gives failure somewhere to hide.

What actually breaks, physically

• ▹Unbounded queue growth eats memory and disk on the broker — most brokers aren't built to hold infinite backlog forever without page-cache pressure or retention limits kicking in
• ▹Recovery becomes its own outage: a "replay storm," where six hours of backlog drains into downstream systems — databases, APIs, other services — far faster than they'd normally see traffic
• ▹At-least-once delivery means some of that replay reprocesses messages that had partially succeeded before the crash — duplicate emails, duplicate charges, duplicate side effects
• ▹Scale the consumer pool out to drain the backlog faster and you get a thundering herd instead: a pile of cold workers hitting the same downstream dependency — DB connections, an LLM API, a cache — all at once

The fix: alert on lag, not liveness

Broker-up is necessary. It is not sufficient. The metric that actually tells you the truth is consumer lag — the gap between the latest offset the producer has written and the last offset a consumer group has committed — or better still, backlog age: how old is the oldest message still waiting. Either one pages you in minutes. This one showed up as a support ticket six hours later instead.

# Prometheus alerting rule sketch — exact metric names vary by exporter
# kafka_consumergroup_lag is exposed by the common kafka_exporter (message-count lag)
- alert: ConsumerGroupLagTooHigh
  expr: kafka_consumergroup_lag > 5000
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "Consumer group {{ $labels.consumergroup }} lag > 5000 messages"

# Time-based lag needs a time-aware exporter (e.g. kafka-lag-exporter, Burrow) —
# confirm the exact metric name against your exporter's docs before shipping this
- alert: OldestUnprocessedMessageTooOld
  expr: kafka_consumergroup_group_lag_seconds > 600
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "Oldest unconsumed message is over 10 minutes old"

Ten minutes of lag paging you beats six hours of backlog paging your customers.

The bigger lesson — and why it matters more with AI in the stack

This exact failure is now everywhere AI agents show up. Run an agent fleet where a task queue feeds a pool of LLM workers, and the "consumer" is that worker pool making inference calls. One crashes, or a rate limit quietly stalls it, and the queue just keeps absorbing whatever comes in — user requests, background jobs, other agents' tool calls — while nobody notices, because everyone's watching whether the queue service is reachable, not whether tasks are actually draining. Recovery is worse than the outage here: a replay storm means a burst of duplicate LLM calls slamming your rate limits and your bill at the same moment, and a thundering herd means dozens of cold agent workers re-establishing context and hitting the same model endpoint simultaneously. Carry this forward: a queue is a buffer with its own failure mode, not a failure boundary. You'll need that assumption when you design consumer autoscaling and dead-letter queues later in this series.

Extend your knowledge

• ▹Read your broker's documentation on consumer group rebalancing and how lag is actually computed (e.g. Kafka's records-lag-max, or SQS's ApproximateAgeOfOldestMessage)
• ▹Check whether your current alerting covers consumer lag / backlog age at all, or only broker liveness — most default dashboards only ship the latter
• ▹Look into dead-letter queue patterns for your broker so a poison message can't block an entire partition's backlog
• ▹If you run LLM or agent worker pools behind a queue, review backpressure and autoscaling patterns so backlog drains gradually instead of as a thundering herd against your model endpoint