Your Gateway Counted 40K Requests. The Bill Said 400K. Here's the Gap.

Day 13 — Your API Gateway Has No Idea Your Agents Are Talking to Each Other

Put a gateway in front of a multi-agent system and call it "observable," and here's what you've actually got: a clear view of the front door and nothing else. Every call that happens after the gateway waves a request through is invisible to it — and in agentic systems, that's usually where most of your cost, your latency, and your failures are quietly piling up.

The dashboard that lied by omission

It started with a billing anomaly. Our LLM provider's invoice for one workload came in 8-10x over what the gateway's request-count dashboard said it should be. The gateway logged 40,000 requests that day — comfortably inside budget. The provider's usage report showed north of 400,000 billed completions. Same day. Same workload. Two completely different stories, depending on which system you asked.

Nobody had touched pricing. Nobody had misconfigured the gateway either — it was doing exactly what it was built to do: count requests crossing the edge. The gap wasn't a bug. It was a blind spot.

Why the gateway was there in the first place

Quick rewind to the [[Day 12]] logic: we put the gateway in front of the agent system to do two jobs — check who's allowed to call in, and cap how hard they can hit us. Front door lock, bouncer. That's the whole mandate. Nobody had ever asked it to understand what happens once a request gets past that door — and it was never sized or designed to.

Tracing one request through the orchestration logs

We grabbed one request ID from the gateway's access log and followed it forward into the orchestrator logs. What the gateway saw as a single POST to /agent/query turned out to be a full reasoning loop: a planner agent calling a retrieval agent, which called two tool-executor agents in parallel, one of which triggered a verification sub-call because its confidence score came back low — which then looped back into the planner for a second pass.

gateway.log:
  req_id=8f2a  POST /agent/query  status=200  latency=1.8s   <- the ONLY line the gateway ever sees

orchestrator.log (same req_id, reconstructed):
  8f2a.1  planner        -> retrieval_agent        120ms
  8f2a.2  retrieval_agent-> vector_db_agent          80ms
  8f2a.3  retrieval_agent-> web_search_agent        310ms
  8f2a.4  planner        -> tool_executor_a         240ms
  8f2a.5  planner        -> tool_executor_b         190ms
  8f2a.6  tool_executor_b-> verifier_agent          410ms   (low confidence, triggered retry)
  8f2a.7  verifier_agent -> planner (2nd pass)      450ms
  ...
  = 12 internal agent-to-agent calls billed as LLM completions

One request in. Twelve calls out. The gateway's "1.8s, 200 OK" wasn't wrong, exactly — it just had nothing to say about where the cost went or where the time was actually spent.

This is structural, not a misconfiguration

Nobody forgot to instrument something. API gateways were built for a client-to-service topology: one external caller, one backend, one response. That model holds up fine for REST APIs and microservices, where a request maps roughly 1:1 to the work it causes. It breaks the moment "the backend" is a reasoning loop that decides, at runtime, how many sub-calls it needs to answer the question.

An agent orchestration step isn't a fixed call graph you can point a gateway at — it's dynamic. The planner might call one agent or seven, depending on the input, the model's confidence, or a retry triggered by a bad tool response. No amount of gateway configuration fixes this, because the gateway was never given visibility into the agent mesh in the first place. It simply isn't equipped to see decisions made after the handoff.

The distributed monolith trap

Here's the part that made it worse: several of our agents were calling each other through the same gateway, not directly. From the outside this looked decoupled — separate services, separate deploys, a clean diagram with boxes and arrows. In practice, every internal agent-to-agent hop paid gateway latency, shared the gateway's rate limits, and could be taken down by the gateway's own failure modes.

That's a distributed monolith: services that are physically separate but behaviorally welded together through a shared chokepoint. One agent's retry storm could rate-limit a completely unrelated agent's traffic, because the gateway saw them both as the same undifferentiated inbound load. Decoupled in the diagram. Coupled in the blast radius.

What we changed

• ▹Kept the gateway doing exactly what it's good at: edge auth, ingress rate limiting, and basic ingress routing for external callers.
• ▹Took agent-to-agent calls off the gateway path entirely — internal mesh traffic now goes direct (or through a lightweight internal proxy), not back through the edge.
• ▹Added per-hop tracing inside the mesh: every agent call carries the parent request ID, so we can reconstruct the full fan-out tree after the fact, not just the entry point.
• ▹Tagged every LLM call with cost attribution back to the originating request and the specific agent that made it — this is what let us actually explain the invoice instead of guessing.
• ▹Set per-agent (not just per-gateway) rate limits and circuit breakers, so one agent's failure mode can't starve the others just because they share an entry point.

The one-sentence rule

If you can't answer "how many calls did this one request actually cause?" — with a number, not a shrug — your gateway is lying to you by omission, and you're flying blind on cost, latency, and failure attribution in exactly the place multi-agent systems break.

Extend your knowledge

• ▹Read up on OpenTelemetry's distributed tracing model (spans, trace context propagation) — it's the standard mechanism for the per-hop tracing described here, and many agent frameworks can be instrumented to emit spans into it via OpenTelemetry-compatible libraries (e.g., OpenLLMetry/Traceloop, Arize's OpenInference), even where it isn't built in natively.
• ▹Look at how your LLM provider's usage/billing dashboard breaks down calls (by API key, tag, or metadata) and check whether you can attribute cost per originating request today — if not, that's your gap to close first.
• ▹Review Day 12 on gateway auth and rate limiting to see the contrast: what the gateway is genuinely good at, versus what got silently assumed to be covered.
• ▹If you're running an orchestrator (LangGraph, Temporal, custom), check whether it already propagates a parent request/trace ID to every sub-agent call by default — many don't out of the box.