Your API Gateway Is Already Refereeing a Fight Between Your Agents — It Just Doesn't Know It

Day 5 — The API Gateway (as an Accidental Agent Orchestrator)

If your org runs agents, your API gateway is already coordinating multiple of them — badly, silently, and with nobody on call aware it's happening. That blind spot is exactly where your duplicate writes and your 'flaky' downstream service actually come from.

The retry storm that wasn't a retry storm

Last month an on-call channel spent two hours chasing a 'flaky' order service. Symptoms: intermittent 500s, duplicate PATCH requests hitting the same order ID, timeouts that cleared themselves before anyone could grab a trace. Textbook downstream instability, so out came the textbook fix — bump the timeout, wrap it in a circuit breaker, page the service owner. The service owner found nothing wrong, because there was nothing wrong with the service. We finally pulled the gateway access logs and lined every hit up by order ID, and the picture flipped completely: two separate agent runs — one working a customer refund, one working a warehouse-triggered 'fix stock mismatch' job — had each independently decided to patch the same order, seconds apart, with conflicting field values. The gateway did exactly what it was built to do: authenticated both, throttled neither (both were well under quota), routed both, logged two clean 200s. From where it sat, there was no conflict — just two valid HTTP requests that happened to land close together.

Day 5 checkpoint: the API gateway is infrastructure you already know cold. It's just doing a job nobody designed it to do.

What an API gateway actually promises

Strip away the vendor branding — Kong, Apigee, AWS API Gateway, your own hand-rolled Envoy config — and every gateway is really only promising four things: routing (send this request where it belongs), auth (prove who's asking), rate limiting (stop one caller from drowning a downstream), and a single choke point for observability — one place to log, one place to throttle, one place to kill traffic when something catches fire. All four promises rest on the same unstated assumption: one request, one caller, no memory between calls. The gateway judges each HTTP request entirely on its own merits. It has no idea what the caller did five minutes ago, whether this call supersedes an earlier one, or whether the caller is even the same 'entity' that hit this endpoint ten seconds before. That's not an oversight — it's the design choice that makes gateways horizontally scalable and boring to run. Statelessness is the feature, not the gap.

Why agents break that assumption

A human clicking 'submit' issues one request as one unit of intent, done. An agent doesn't work that way. It plans, calls a tool, looks at what came back, decides the result was wrong or half-finished, and fires again — sometimes the identical call, sometimes a compensating one, sometimes a different plan entirely that just happens to land on the same endpoint. All of that reasoning — 'I already tried this,' 'this is a retry,' 'I'm now cleaning up after my own earlier move' — lives entirely inside the agent's context window. None of it rides along with the HTTP request. The gateway sees a PATCH to /orders/123. It has no field for 'this is attempt 2 of a self-correcting plan' or 'this call should void that other call I made 30 seconds ago.' It only ever sees HTTP, and HTTP was never built to carry intent.

The concrete failure mode

• ▹Two different agents (or two branches of the same agent — the original call plus its own retry) hit the same endpoint inside a short window.
• ▹Each call is independently well-formed: valid auth token, valid schema, comfortably under rate limits.
• ▹The gateway has no concept of 'these two belong together' — it forwards both as unrelated, legitimate traffic.
• ▹The downstream service applies both writes, in whatever order they arrive, with no idea they were ever racing.
• ▹You end up with a data state no single agent intended — an order that's both cancelled and refunded, a stock count adjusted twice, a ticket closed by one 'fix' and reopened by another.

Why on-call misdiagnoses this every time

Every symptom that surfaces is infra-shaped: timeouts, 429s, duplicate writes, a downstream service that 'sometimes' hands back inconsistent state. Every dashboard on-call has — gateway latency, error rate, rate-limit counters — points a finger at the gateway or the service behind it. None of those dashboards answer the question that actually matters: which two agents were both trying to act on this resource at the same time, and why did neither know the other existed? That question doesn't live in gateway metrics — it lives in agent traces, and most orgs running agents in 2026 still don't correlate agent-level traces with gateway-level request logs. So the incident closes with a timeout bump or a retry-backoff tweak, and the real cause — two independent planners with no shared view of in-flight work — ships again next week, wearing a different pair of agents.

The ownership gap

Rate limiting has an owner — usually platform or infra. Auth has an owner — usually platform or security. Coordination between two agents sharing a gateway has no owner at all. It's not the gateway team's problem — they'll correctly point out the gateway did nothing wrong per spec. It's not obviously the agent builder's problem either — from inside a single agent's context window, everything looks correct, because it doesn't know a second agent exists. It's the same shape as the old 'whose bug is it' fight from distributed systems, except now the two 'services' racing each other are both LLM-driven planners, and nobody designed them to be mutually aware. Nobody wrote this into a job description, because a year ago nobody was routing agent tool-calls through a shared gateway at this volume.

What actually needs to exist instead

Not a smarter, heavier gateway — gateways are good at being dumb and stateless, and that's worth protecting, not fixing. What's missing is a thin coordination layer sitting between the agents and the gateway, doing two things a plain idempotency key can't: tying idempotency to intent instead of just to the request payload, and giving an agent a way to say 'this call supersedes that one' instead of quietly firing a competing call into the void. A standard idempotency key only catches the exact same request being replayed — it does nothing when two different requests, from two different agents, both legitimately want to touch the same resource. That's a different problem, and I'm naming it here rather than solving it — it's the seed for tomorrow's entry.

// A standard idempotency key only dedupes identical retries:
{
  "idempotency_key": "req-8f21",   // unique per HTTP call
  "body": { "status": "cancelled" }
}

// What agent coordination actually needs is intent-scoped:
{
  "intent_key": "order-123:resolve-mismatch",  // shared across agents/attempts
  "supersedes": "order-123:resolve-mismatch:attempt-1",
  "agent_id": "stock-fix-agent-run-9a3",
  "body": { "status": "cancelled" }
}

The reframe to leave with: the gateway isn't the wrong place to spot this — the logs genuinely show you the collision once you know where to look. It's the wrong place to fix it alone, because fixing it means knowing about intent, and intent lives upstream of HTTP. File this as the first crack in 'infrastructure built for stateless clients.' It won't be the last one this series pulls on.

Extend your knowledge

• ▹Pull your own gateway's access logs for a busy endpoint and group by resource ID over a 60-second window — look for pairs of calls from different caller identities that nobody flagged as related.
• ▹Check how your own gateway or API framework handles idempotency keys, if it handles them at all — most implementations only dedupe an identical request being replayed, not two different requests sharing the same intent — that gap is exactly what tomorrow's entry builds on.
• ▹If you run multiple agents against shared write endpoints, check whether your agent framework exposes a run ID or plan ID you could propagate as an intent key today, before building new infra.
• ▹Revisit any recent 'flaky downstream service' incident with agents in the loop and ask the question this lesson poses: which two agents were acting on the same resource, and did anyone check?