Service mesh for AI agents

Day 24: Service Mesh — moving the rules out of reach of the code they restrain

Your agent's system prompt says "never call the payments tool." One pasted support ticket later, it calls payments. Any agent you connect to a tool can call an API you didn't want called, and if the agent can read a rule, it can be talked out of it. A service mesh is the most mature pattern we have for deciding who may call what. It enforces that decision outside the caller's code, where an agent can't reach it.

The hook: a rule that lives next to the thing it restrains

Picture a support agent with one line in its system prompt: "Never call the payments tool." It holds up in every demo. Then a customer pastes in a ticket with hidden instructions, the model reads them as context, and the agent calls payments anyway. Nothing broke. The model did what models do. It weighed one piece of text against another, and the injected text won.

Better wording won't fix this. The rule lived in the same process and the same context window as the thing it was meant to restrain. Moving it into an SDK check helps, but not enough. A different code path, a new framework version or a tool someone else registered can walk right past it. Microservices hit the same wall about ten years ago.

Rewind: why meshes exist at all

When teams broke monoliths into dozens of services, every service suddenly had to handle the network itself. In practice that looked like this:

• ▹The Go service did TLS one way, the Java service did it another way, and the Python service skipped it 'because it's internal'.
• ▹Retries and timeouts were copied between libraries that had different defaults. One service retried 5 times with no backoff and turned a small outage into a big one.
• ▹"Is this caller allowed?" was an if-statement in every service. Some checked a header. Some checked an IP range. Some checked nothing.
• ▹On top of that, every service trusted its own code to get all of this right, on every release, in every language.

The mesh idea was to pull all of that out of the application and into one uniform layer that every request has to pass through. The service keeps the business logic. The layer handles who, how and how often.

The definition in plain words

A service mesh is a layer of small proxies that every request between services passes through, plus a control plane that pushes one shared policy to all of those proxies.

• ▹Data plane: the proxies that do the actual work. In the classic setup, a sidecar (often Envoy) runs next to each service instance. 'Ambient' designs such as Istio ambient mode split the job. A per-node proxy (ztunnel) handles identity and encryption, and an optional 'waypoint' proxy, usually one per namespace, handles L7 rules like HTTP paths and methods. Linkerd ships its own lightweight Rust proxy.
• ▹Control plane: the part that hands out the rules. It issues certificates (identities) and sends authorization policy, routing config and telemetry settings to the proxies. The proxies then send metrics, logs and traces to your observability stack. Istio's istiod and Linkerd's control plane are two examples.
• ▹The property that matters: application code can't override any of it. Traffic is redirected through the proxy at the network level. The app doesn't opt in, and it can't opt itself out. Exclusions are set by whoever runs the platform, not by the app.

Follow one request

• ▹Service A makes an ordinary call to B. It doesn't load certificates or implement retries.
• ▹A's proxy intercepts the call and opens a mutual TLS connection to B's proxy. Both sides present certificates.
• ▹The certificate carries a workload identity, usually in SPIFFE format, such as spiffe://cluster.local/ns/billing/sa/refund-svc. The identity comes from what the workload is, not from its IP address. In Kubernetes, IPs change all the time and tell you nothing about who's calling.
• ▹B's proxy checks policy: is this identity allowed to call this method on this path?
• ▹If it is, the request reaches B. If it isn't, A gets a 403 and B never sees the request.
• ▹Both proxies record the call (who, what, latency, result) for metrics and traces.
• ▹What the application code never had to do: manage certificates, rotate keys, check authorization, implement retries or write access logs. (One caveat: for traces to connect across services, the app still has to forward trace headers from incoming requests to outgoing ones.)

The four jobs of a mesh

• ▹Identity and encryption: mTLS between every pair of workloads, with certificates issued and rotated automatically.
• ▹Authorization: declarative rules for which identity may call which service, path and method.
• ▹Traffic control: retries, timeouts, circuit breaking and rate limits, set in one central place instead of scattered across libraries.
• ▹Observability: metrics and access logs for every call, whether or not the app was instrumented. Proxies emit trace spans too, but joining them into end-to-end traces still needs the app to pass trace headers along.

The bridge to agents

Now swap 'Service A' for 'Agent A' and 'Service B' for an MCP tool server. Earlier in this series we covered tool calling and MCP. The model picks a tool, the runtime sends a request, and the tool server runs it. Once that request leaves the agent's process, it's network traffic like any other.

The governance questions are the same:

• ▹Which caller? Which agent, acting on whose behalf.
• ▹Which target? Which tool server.
• ▹Which operation? Which tool, with which arguments.
• ▹How often? Rate and budget per agent.
• ▹What trail? A record of what the agent actually did, not what it says it did.

Microservices needed a mesh because services shouldn't have to trust each other's code. Agents raise the stakes, because part of their 'code' is generated at runtime from text that anyone can write. If you couldn't fully trust a service a colleague wrote, you shouldn't trust an agent whose next action depends on a customer email.

The opinion: 'please don't' vs 'can't'

My position: a guardrail inside the agent is a request. A guardrail at the network layer is enforced.

• ▹System-prompt rules are persuasion. They shape what the model will probably do, and prompt injection exists precisely to override them.
• ▹SDK and framework checks are better, but they still run in the agent's process. Anything that skips the wrapper skips the check: a raw HTTP client, a new plugin, or a different framework that another team adopted.
• ▹Mesh policy runs in a proxy the agent doesn't control, and it's tied to an identity the agent can't forge, because the certificate is issued to the workload and not to the prompt. Even a fully compromised agent still can't call payments.
• ▹Keep the in-agent guardrails anyway. They give you better UX, earlier refusals and clearer error messages. Just don't treat them as your security boundary.

Before you build your own permissions system inside your agent framework, check whether your mesh already answers 'can caller A reach tool B?'. It often does, and it has been tested in production far longer than anything you'd write this quarter.

What it looks like in practice

• ▹Give each agent its own workload identity. Don't run five agents under one generic 'agents' service account. In Kubernetes, that means one service account per agent type, which gives each agent its own SPIFFE ID.
• ▹Put L7 policy on tool endpoints. Allow billing-agent to call the payments tool server, and deny everyone else by default.
• ▹Rate-limit per agent, not per service. A runaway agent loop shouldn't be able to use up the capacity your human-facing traffic needs. Be aware that Envoy's local rate limit applies to each proxy instance separately, so a true fleet-wide limit needs a global rate-limit service.
• ▹Use mesh telemetry as your audit log. Access logs keyed by agent identity show what each agent actually called, whatever the agent's own (possibly hallucinated) account says.

# Istio: only billing-agent may reach the payments MCP server
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-mcp-allow
  namespace: tools
spec:
  selector:
    matchLabels:
      app: payments-mcp
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/agents/sa/billing-agent"]
    to:
    - operation:
        methods: ["POST"]
        paths: ["/mcp"]
# With an ALLOW policy in place, every other identity is denied.
# Add GET/DELETE if your server uses SSE streams or session termination.
# In ambient mode, L7 rules like this need a waypoint proxy
# (use targetRefs instead of selector).

A pitfall to know about: MCP over HTTP sends JSON-RPC to a single endpoint, so the tool name and arguments are in the request body, not the path. Standard mesh policy matches on identity, path, method and headers. It can answer 'may billing-agent reach the payments server?' but not 'may it call issue_refund with amount 50000?'. For argument-level rules, add an external authorizer such as Envoy ext_authz backed by OPA, or put an MCP-aware gateway behind the mesh. Either way, enforcement still happens outside the agent.

# OPA-Envoy plugin via ext_authz (sketch)
# Requires ext_authz configured to forward the request body
package mcp.authz

default allow := false

allow if {
  input.attributes.source.principal == "spiffe://cluster.local/ns/agents/sa/billing-agent"
  input.parsed_body.method == "tools/call"
  input.parsed_body.params.name == "issue_refund"
  input.parsed_body.params.arguments.amount <= 500
}

The honest cost

Running infrastructure as a CTO taught me that a mesh is one more distributed system, and it fails in its own ways.

• ▹Certificate rotation: when issuance or rotation breaks, nothing happens at first. Then the existing certificates expire, mTLS fails, and services start refusing each other all at once.
• ▹Config push storms: one policy change goes out to every proxy. In large clusters, a bad change or a deployment with lots of churn can overload the control plane or leave proxies with inconsistent config.
• ▹Sidecar startup races: the app starts before its proxy is ready, so its first calls fail. Jobs can't exit cleanly because the sidecar keeps running. Native sidecar containers and ambient mode make these bugs rarer, but they don't get rid of them.
• ▹Debugging gets harder: a 503 could come from your app, the proxy, the policy or an expired certificate, and your on-call engineer needs to know which.
• ▹Rule of thumb: if your on-call engineer can't trace one failed request through the mesh in 10 minutes, you're not ready to run one. A mesh nobody can debug at 2am is a liability, not a guardrail.
• ▹If you have three agents and one tool server, don't adopt a mesh just for them. That's a lot of machinery for a problem a gateway can solve.

The decision rule

• ▹Name the specific policy you can't enforce inside the agent. For example: 'research-agent must never reach the payments server, even if it's fully prompt-injected.'
• ▹If you can name it and you already run a mesh, put the policy in the mesh. You get identity, deny-by-default and audit logging for very little extra work.
• ▹If you can name it but don't run a mesh, start with an API or MCP gateway in front of your tools, with separate credentials for each agent. Adopt a mesh once your number of services, not just agents, justifies it.
• ▹If you can't name it, you don't have a policy problem yet. A simple gateway is enough for now.

Glossary card

Service mesh: the layer that decides who may talk to whom, so the services doing the talking don't have to be trusted to decide.

Coming up on Day 25: today we drew the boundary around what an agent may do. Next, we build on that boundary.

Extend your knowledge

• ▹Read the Istio security docs on AuthorizationPolicy and ambient mode (ztunnel and waypoint proxies) to see how L4 and L7 enforcement are split.
• ▹Read the SPIFFE/SPIRE docs to understand workload identity outside any single mesh. You can reuse it directly for agent identity.
• ▹Work through Linkerd's getting-started guide on a local kind or k3d cluster. Then trace one failed request end to end and time yourself against the 10-minute rule.
• ▹Look at Envoy's ext_authz filter alongside the Open Policy Agent Envoy plugin to prototype argument-level checks on MCP tool calls.