Service discovery

Day 21: Service Discovery: Adopt It When Your Topology Outruns Your Deploys

Every call your system makes needs an address. That's true when one service calls another and when an agent calls a tool. Service discovery is the machinery that keeps those addresses current. It fixes a real problem, and it also gives you a brand-new way to fail. Before you install it, work out which one you're actually getting: the fix or the new failure mode.

Hook: the pager audit

We once pulled two years of 3am incidents and sorted them by root cause. We expected the database or the deploy pipeline to come out on top. The discovery layer was close to the top instead, and that was the layer we'd installed to make us resilient. Close to the bottom was the one problem discovery exists to solve: a service actually moved and its callers couldn't find it.

• ▹Stale registrations. An instance died without deregistering, and the registry kept handing out its address until the TTL expired.
• ▹Agent flapping. The Consul agent on a busy node kept failing gossip failure-detection probes, so a perfectly healthy service dropped out of the catalog and came back every few seconds.
• ▹Split health status. The registry said 'passing' and the load balancer said 'unhealthy', or the other way round. Each was right about something different, and nobody could say which signal callers should trust.
• ▹'Service actually moved.' This was rare, and every time it happened, a config change would have covered it.

So we ripped Consul out. The rest of this lesson explains why that was the right call for us, and how to tell when it would be the wrong call for you.

Rewind: what we thought we were buying

The pitch is simple. A caller needs a host and port for the thing it wants to call, and that address keeps changing. Autoscaling adds and removes instances. Rolling deploys replace old instances with new ones. Failover shifts traffic to another zone or replica. Hardcode the addresses and every one of those changes means editing callers. On Day 20 you saw that a load balancer needs a list of healthy backends to spread traffic across. Discovery is the system that builds and maintains that list. Load balancing picks which instance gets the request. Discovery decides which instances exist in the first place.

The core model: register, health-check, look up, evict

Consul, Eureka, etcd-backed tooling, Kubernetes EndpointSlices: under the hood they all do the same four things. Let's follow one request through them. A new `orders` instance boots, and `checkout` wants to call it.

• ▹Register: the instance (or the platform it runs on) announces its name and address. Easy, and almost never the cause of an incident.
• ▹Health-check: the registry asks whether the instance is alive and can actually serve. This one is hard. 'The process is up' and 'it can serve traffic' are different questions, and the checker reaches the instance over a different network path than your callers use.
• ▹Look up: the caller asks for healthy instances of `orders`. Easy, cacheable, well understood.
• ▹Evict: dead entries get removed, and this is the other hard one. Evict too slowly and callers get stale addresses. Evict too eagerly and a GC pause or a network blip knocks healthy instances out. That's flapping.

Most discovery failures happen in health-checking and eviction. Lookup is rarely the problem. So when you evaluate a discovery tool, don't bother asking how fast lookup is. Ask what happens when the health checker and reality disagree, and how long a dead instance really takes to disappear.

Two shapes: client-side vs server-side discovery

With client-side discovery, the caller queries the registry, gets back a list of instances and picks one itself. This is the Netflix Eureka plus Ribbon style (Spring Cloud LoadBalancer has since replaced Ribbon). With server-side discovery, the caller sends traffic to one stable name and something else does the lookup and the picking. That something could be a load balancer, a DNS record or a Kubernetes Service backed by kube-proxy.

# Client-side: discovery logic lives in your app
instances = registry.healthy("orders")          # registry client library
target = pick(instances, strategy="least_conn") # your LB logic
resp = http.post(f"http://{target}/orders", json=payload)

# Server-side: discovery logic lives in infrastructure
resp = http.post("http://orders.internal/orders", json=payload)
# DNS / load balancer / k8s Service resolves and balances for you

• ▹What client-side adds to your app: a registry client in every language you use, plus caching, retry logic and a load-balancing strategy. In return you get control, like zone-aware routing and smart retries. The catch is that every team now owns code that has to stay correct and up to date.
• ▹What client-side adds to your ops: a highly available registry cluster. For Consul that means a Raft quorum and gossip agents on every node. Congratulations, that cluster is now critical infrastructure.
• ▹What server-side adds to your app: close to nothing. You call a hostname.
• ▹What server-side adds to your ops: a load balancer or platform layer that owns health checks. If it's managed (a cloud LB, a Kubernetes Service), someone else runs the hard parts.
• ▹My rule of thumb: start with server-side. Reach for client-side only when you need routing decisions that only the caller is in a position to make.

Why our registry hurt more than it helped

Here were our numbers. About 15 services, a few deploys a day, almost no autoscaling. Our topology changed more slowly than our deploy pipeline ran, so any address change could ride along with the next deploy, which was never more than a few hours away. What we were actually doing was running a distributed consistency system (quorum, gossip, TTLs, health-check tuning) to answer a question a versioned config file already answered. The registry didn't remove a single point of failure. It added one. Worse, it was a sneaky one, because it could fail partway and still look healthy.

What we replaced it with

• ▹Static hostnames in versioned config, reviewed in pull requests like the rest of the code.
• ▹One managed load balancer per service, owning the health checks. That gives you a single source of truth for 'is it healthy'.
• ▹DNS with short TTLs for the handful of things that really were dynamic. Nothing fancier.

# config/upstreams.prod.yaml  (versioned, reviewed, deployed with the app)
upstreams:
  orders:
    url: https://orders.internal.example     # managed LB in front
    timeout_ms: 800
  payments:
    url: https://payments.internal.example
    timeout_ms: 1500
  llm_gateway:
    url: https://llm-gw.internal.example     # DNS, the one truly dynamic thing
    timeout_ms: 30000

• ▹Better: fewer moving parts, and one health signal where there used to be two. Failures show up in a diff now. If calls go to the wrong place, `git log` tells you who changed it and when.
• ▹Better: on-call debugs a hostname and a load balancer. Every engineer on the team already understands both.
• ▹Worse: scaling is more manual. Putting new capacity behind a new hostname takes a config change and a deploy.
• ▹Worse: failover is slower. Moving traffic to another region is a DNS or config change with propagation delay, not an automatic re-registration.

The decision rule (screenshot this)

• ▹Your instances change faster than you can redeploy config: adopt dynamic discovery.
• ▹You autoscale on minutes-scale demand. LLM inference fleets that scale GPU workers on queue depth are the textbook example.
• ▹Too many teams share one config file to coordinate changes. In my experience that usually starts somewhere around 30 or more services.
• ▹None of the above? Fix deployment hygiene first. Fast, boring deploys make static config cheap, and a slow pipeline is often the real problem dressed up as 'we need discovery'.
• ▹On Kubernetes, you already have server-side discovery through Services and DNS. Use it. Don't stack a second registry on top.

Same lesson, new client: agents

MCP servers, A2A agent cards (published at a well-known URL) and tool catalogs are dynamic discovery with an LLM as the caller. Same four steps. A tool registers. Someone should be health-checking it. The agent looks it up (MCP `tools/list`). Something should evict it when it dies or changes. The rush to build live agent and tool registries right now looks a lot like the 2015 microservices rush, and the same rule holds. If your agent uses five tools that change once a quarter, a static, pinned tool manifest beats a live registry.

# Pinned tool manifest: fail closed at startup if anything drifted
PINNED = {
    "search_tickets": {"server": "https://tools.internal/tickets/mcp", "schema_sha256": "9f2c..."},
    "create_refund":  {"server": "https://tools.internal/billing/mcp", "schema_sha256": "41ab..."},
}

def load_tools(clients):
    tools = []
    for name, pin in PINNED.items():
        live = {t.name: t for t in clients[pin["server"]].list_tools()}
        tool = live.get(name)
        if tool is None:
            raise RuntimeError(f"tool missing: {name}")
        if sha256_of(tool.inputSchema) != pin["schema_sha256"]:
            raise RuntimeError(f"tool schema drifted: {name}")  # review before trusting
        tools.append(tool)
    return tools

Agents make one thing worse. A service that calls a dead backend gets a 503 or a timeout, and everyone notices. An LLM that calls a dead tool often gets the error back as plain text in its context. The model may route around it and hand you a confident, wrong answer, like 'No open tickets found' when the ticket service was simply down. In a microservice system a stale registry entry costs you latency. In an agent system it can cost you correctness, and no pager goes off.

• ▹If you really do need a registry (lots of teams publishing tools, or tools that come and go hourly), carry the old scars over:
• ▹Health probes that make a real tool call. An open port proves very little.
• ▹TTL eviction, so a dead MCP server drops out of the catalog instead of lingering there.
• ▹Capability versioning. Pin tool schemas by version or hash, and treat a schema change as a deploy, not a surprise. Even a reworded tool description changes how the model behaves.
• ▹Structured tool errors nobody can mistake, plus agent logic that treats a failed tool call as a failed step, not as an invitation to improvise.

Exercise for today

• ▹List every service, external API, MCP server and agent tool your system calls.
• ▹For each one, write down how many times its address or capability (endpoint, schema, version) changed in the last 90 days. Pull this from git history and deploy logs. Memory lies.
• ▹Note whether it autoscales, and how quickly.
• ▹Run each through the decision rule and mark it STATIC (config file), DNS/LB (server-side) or REGISTRY (dynamic discovery).
• ▹For everything marked REGISTRY, write down how it's health-checked and how fast a dead instance gets evicted. Can't answer? Then that's your next piece of work.

callee            | addr/capability changes (90d) | autoscale?   | decision
------------------|-------------------------------|--------------|---------
orders            | 0                             | no           | STATIC
llm-inference     | many (GPU workers churn)      | yes, minutes | DNS/LB or REGISTRY
mcp:search_docs   | 1 schema change               | no           | STATIC, pinned hash

Bridge to Day 22: once services and agents can find each other, the next question is what to do when the thing they found is alive but slow. That's where timeouts and circuit breakers come in. They let you fail fast and gracefully, so one slow dependency doesn't drag down every caller.

Extend your knowledge

• ▹Kubernetes docs on Services, EndpointSlices and readiness probes. This is server-side discovery and health-gated eviction built into a platform you may already be running.
• ▹HashiCorp Consul's architecture docs on gossip (Serf) and Raft. Watch for where agent health and catalog state can drift apart, because that's exactly where split health status comes from.
• ▹The Model Context Protocol spec sections on `tools/list` and the `notifications/tools/list_changed` notification (advertised through the `listChanged` capability). Read them, then decide whether your agents should react to tool changes live or pin them.
• ▹Chris Richardson's pattern pages on microservices.io for client-side and server-side discovery. They're the canonical definitions, trade-offs included.