Four Green Nodes, One Broken p99: What Your Load Balancer Can't See About Agent Traffic

Day 1: Load Balancer, Redefined for the Agent Era

Four healthy replicas. Flat CPU graphs. And a p99 that just paged someone for no visible reason. If you're running agentic LLM traffic through plain nginx, an ALB, or vanilla k8s Services, you've hit this wall — the load balancer isn't broken. It's doing exactly what it was built to do. It just so happens that what it was built to do has zero concept of what your traffic actually costs.

The hook: a phantom p99 spike

Picture the dashboard: four replicas, all green, CPU sitting comfortably in the 20-30% range. And your p99 latency is spiking hard enough to wake someone up at 2am. You dig into node-level metrics and three of the four nodes are basically coasting. The fourth isn't on fire either — no crash, no OOM, no red line anywhere. It's just slow, for everyone unlucky enough to land on it. That's the tell. Nothing is unhealthy. Something is unevenly loaded, and none of your tooling was built to see it.

What a load balancer actually promises

Classic load balancing has exactly one job: spread requests evenly across a pool of backends, using signals like round-robin order, active connection count, or response time. That promise made sense for stateless HTTP — a request for a product page and a request for a login form burn roughly the same CPU-milliseconds and hold a connection open for roughly the same span. Spreading requests evenly was a decent proxy for spreading load evenly, because request cost barely varied.

Agent traffic throws that assumption out the window. A hit to your agent gateway might be a 200-token health-check-style ping, or it might be a 50,000-token tool-use chain holding a connection open for tens of seconds while the model reasons, calls a tool, waits, calls another tool, and reasons again. Same protocol, same endpoint, wildly different cost. The load balancer sees "one request" either way. It has no unit for telling them apart.

The concrete failure: round-robin and least-connections both miss it

Walk through it. Round-robin sends request 1 to node A, request 2 to B, request 3 to C, request 4 to D, then loops. If three long tool-use chains happen to land on node A back to back — entirely possible, since round-robin never looks at what's inside the request — node A is now stuck serving three 50k-token generations at once, while B, C, and D each handle a trickle of 200-token pings. Round-robin has no way to know it just made a bad call, because it never had a signal for "how expensive is this one" in the first place.

Least-connections looks smarter on paper — route new requests to whichever node has the fewest open connections, which sounds load-aware. It isn't. A connection is a connection whether it closes in 50 milliseconds or stays open streaming tokens for 45 seconds. If node A has 3 open connections and node B has 5, least-connections ships new traffic to A — even if A's 3 connections are long, expensive tool-use chains and B's 5 are trivial pings about to close. Fewer connections is not the same as less work. Least-connections is optimizing for the wrong variable, and it has no idea it's doing it.

Why this is invisible to health checks and dashboards

Your health check hits /healthz, gets a 200, moves on. Your CPU dashboard looks calm because token generation on most inference stacks is bound by I/O and memory bandwidth, not CPU, so it never produces the scary utilization number your alerts are wired to. Node A isn't down. It isn't erroring. It might not even register as hot on a single metric your ops team watches. It's just busy — committed to work that takes a long time to finish — and "busy in a way that takes time" doesn't map to any standard health signal anyone built. The node is expensive right now, not unhealthy, and your monitoring stack was built to catch the second thing, not the first.

The reframe: load is projected cost, not connections

For agent workloads, the real unit of load isn't "a connection" or "a request" — it's projected token/time cost. What a request costs depends on things classic LB algorithms never look at: expected output length, tool-use depth, whether it's one completion or a five-step chain, how big the context is. None of that shows up in a TCP connection count or a round-robin counter. That's the actual gap. It's not that round-robin or least-connections are badly implemented — it's that "cost per request" has no slot in their model of the world at all.

# crude cost-aware routing sketch
for each incoming request:
    est_cost = estimate(prompt_tokens, expected_tool_calls, history_depth)
    target = min(nodes, key=lambda n: n.in_flight_cost)
    target.in_flight_cost += est_cost
    route(request, target)
    # on completion: target.in_flight_cost -= est_cost

• ▹Common pitfall: assuming least-connections already handles this. It doesn't — it counts connections, not what each connection costs.
• ▹Common pitfall: throwing more replicas at it without fixing routing. You'll just end up with more idle nodes sitting next to the same overloaded one.
• ▹Common pitfall: sizing capacity off average request cost. Agent traffic cost is heavy-tailed, and averages hide exactly the tail that spikes your p99.
• ▹When classic LB algorithms still hold up: uniform, short-lived, single-turn completions with tight token limits and no tool use — cost variance is low enough that counting requests is a fine proxy again.
• ▹When you need cost-aware routing: multi-step agent chains, variable tool-use depth, mixed workloads — chat pings and long agentic tasks sharing the same pool.

Today's term, defined

Load balancer, agent-era definition: a routing layer whose job is to spread projected cost — not requests, not connections — evenly across backend replicas, where cost gets estimated from signals like token count, tool-use depth, and expected generation time instead of inferred from connection state. Classic algorithms — round-robin, least-connections — stay correct for stateless, low-variance HTTP traffic. They go silently wrong the moment request cost variance climbs, which is the normal state of affairs for LLM agents.

Tomorrow's concept builds straight on this: once you accept that "load" means cost, the next question is who decides where a request goes and how. That's orchestration and routing for agent systems — a layer sitting above plain load balancing that actually knows what an agent is about to do, not just how many connections it's holding.

Extend your knowledge

• ▹Look at how Envoy and other modern proxies support weighted or custom load-reporting extensions, and see how far weighted least-request gets you before you need to feed in your own cost estimate.
• ▹Read how vLLM and TGI handle continuous batching and queue-depth-aware scheduling inside a single node. They've already had to solve a version of this cost-awareness problem internally.
• ▹Instrument your own gateway. Log estimated input+output token count per request next to connection count and node, so you can actually see the cost distribution before you touch the routing.
• ▹Tomorrow's concept: agent orchestration and routing layers — the piece that sits above plain load balancing and understands what a request is about to do, not just how many connections it holds.