We Built a Rate Limiter. It Forgot How to Say No.

Day 14: The Rate Limiter That Queued Instead of Rejecting

Every rate limiter has one job on its résumé: protect the API. Nobody ever writes down the second job — what happens to the requests it doesn't reject. That's the part nobody interviews for, and it's exactly where this outage came from. It matters more now than it used to, because a single user request can fan out into five or six agent-to-agent calls, each one spending its own slice of a shared timeout budget before anyone notices the clock is running out.

Mid-incident: everything's green, then everything's slow

Picture the pager going off. Service A takes a traffic spike, and its p99 climbs — fine, expected, that's the service that got hit. Ninety seconds later, three services downstream of A start climbing too. All three, at the same moment, with no spike of their own. That simultaneity is the tell — three unrelated services don't degrade in lockstep by coincidence — but at 2am with a pager buzzing, it just reads as everything catching fire at once.

The false lead

On-call's first instinct is almost always the database, or the cache behind it — when several services get sick at once, a shared resource is the obvious suspect. So ten minutes go to connection pool saturation, replica lag, cache hit rate. All green. Not just fine — boringly fine, no unusual load at all. Which means whatever's compounding this latency lives upstream of the database, not inside it.

The reveal: a queue wearing a rate limiter's name

Turns out the rate limiter sitting in front of Service A was a leaky bucket — not the metering-algorithm version people mean when they say that in a design doc, but the literal one: an honest-to-god queue, accepting every request that showed up and draining them at a fixed rate. It was never built to reject anything. So when the spike hit, it didn't shed a single request — it swallowed all of them, and just started answering later and later. Nothing got a 429. Everything got delayed. And delay, unlike a rejected request, compounds.

Why this is counterintuitive

'We have rate limiting' is a checkbox on an architecture review, filed right next to 'we have logging' — ticked and forgotten. Nobody asks the follow-up question that actually matters: when this thing gets overwhelmed, does it fail open by rejecting fast, or fail slow by queuing and quietly hoping the surge passes before anyone notices? Those are opposite failure modes. They share a label, and the checkbox can't tell them apart.

The mechanism, plainly

• ▹Token bucket: a bucket holds N tokens, refilled on a fixed schedule. Every request spends one. Empty bucket, instant rejection — a 429, no wait, no queue.
• ▹Leaky bucket, the way it was actually built here: every request lands in a buffer and gets processed at a fixed drain rate. Nothing gets turned away. Everything just waits its turn.
• ▹A rejected request is the caller's problem, and a small one — one error, one decision about whether to retry.
• ▹A queued request is everyone's problem. The caller's thread or connection sits open waiting for an answer, tying up the caller's own resources — and whatever that caller was serving (a user, another service, one step in an agent's plan) sits waiting right along with it.
• ▹That's the whole mechanism behind 'one spike becomes everyone's incident': queuing doesn't shrink the burst, it smears it across time, and both ends of the connection stay occupied for the entire smear.

# token bucket: reject fast, no queue
def allow_request(bucket):
    bucket.refill()
    if bucket.tokens < 1:
        return False, bucket.seconds_until_next_token()
    bucket.tokens -= 1
    return True, None

# vs. the incident's leaky-bucket queue
def handle_request(request, queue):
    queue.put(request)          # never rejects, just buffers
    return queue.get_when_drained(request)  # caller blocks here, holding a connection

Why this bites harder in agent chains

This bug is getting more common, not less, because more of the traffic hitting these limiters is agentic now. An agent calling an LLM gateway, which calls a tool, which calls another service, is nothing but a chain of synchronous waits — every hop is a caller blocked on the hop before it. Let the rate limiter in front of your model provider queue instead of reject during a burst — a fan-out of parallel agent calls, a retry storm from one flaky tool — and the delay doesn't stay put. It climbs back up through the whole reasoning loop, and if that agent is just one worker in a fleet, the orchestrator above it starts timing out too. Tail latency in a multi-agent system multiplies, it doesn't add: p99 at each hop compounds into a much uglier p99 for the whole chain, and a queuing limiter multiplies faster than a rejecting one, because it never lets a single hop fail cheap.

The fix and the scar tissue

The fix itself was small: swap the leaky-bucket queue for a token bucket that rejects fast and hands back an explicit Retry-After, so callers know exactly how long to sit tight instead of guessing, or worse, hammering the endpoint again immediately. The scar tissue is the part that actually matters — a whole new class of test, not a config tweak. Inject a traffic spike into staging and assert that p99 stays flat, across the service and everything downstream of it. The old suite only checked that excess requests eventually got a 429. It never once asserted anything about latency under load, because it was written to prove the happy path works — not to catch the failure path breaking.

Day 14 takeaway

The algorithm behind a rate limiter isn't a performance detail you pick and move on from — it's a decision about who pays for the failure. Token bucket hands the cost to the caller: one fast, cheap, visible error. Leaky-bucket queuing hands the cost to the whole system: a slow, expensive degradation that stays invisible right up until it cascades. So when you're designing one of these — or, more likely, inheriting one — especially in front of an LLM API or an agent orchestration layer, don't ask whether you have a rate limiter. Ask which one you actually built.

Day 15 picks up where this leaves off: once you've committed to failing fast, how do you decide what to shed first when there still isn't room for everything?

Extend your knowledge

• ▹Stripe's public engineering writeup on rate limiting is the best production-grade comparison of token bucket, leaky bucket, and sliding-window I know of — read it if you want the trade-offs spelled out with real numbers.
• ▹Google's SRE book (sre.google) has chapters on load shedding and back-pressure that make the systems-level case for 'who absorbs the failure' — worth it even if you never touch Google's stack.
• ▹Running an LLM gateway or an agent fleet? Go check right now whether its rate limiter returns 429s with Retry-After, or just quietly queues under provider-side throttling. That's this exact bug, one layer further up the stack.
• ▹Write the staging spike test yourself before you need it: throw 5-10x normal load at the service for 30 seconds and assert p99 stays within a fixed bound on it and its direct dependents — not just that requests eventually go through.