The Health Check That Lied — Why a Wall of Green Doesn't Mean Nothing's Broken

Day 2: The Health Check That Lied to You

Here's how it goes wrong: every replica in the fleet is answering 200 OK on /healthz. Not one red square anywhere. Meanwhile the error rate on real traffic is climbing fast enough that Slack is filling up with angry customers, and on-call is staring at a load balancer target group that insists, instance by instance, that everything is fine. Someone finally says the sentence that should worry you more than any outage ever could: "but health checks are passing." That sentence doesn't mean the load balancer is lying to you on purpose. It means it's faithfully repeating a lie it was told.

What a health check actually verifies (by default)

Out of the box, almost no health check tests capability. It tests liveness — is the process running, is the port accepting connections, and if whoever wrote it was feeling generous, can it ping the database. That's the whole exam. It's a pulse check, not a fitness check. The load balancer isn't asking "can you actually do the job I'm about to route to you?" It's asking "are you, technically speaking, a process right now?"

• ▹Process up: the binary or container hasn't crashed
• ▹Port open: something is listening on the expected socket
• ▹Sometimes: a trivial DB ping (SELECT 1, not the query your app actually runs)
• ▹Almost never: auth works, the real dependency chain resolves, the actual request path succeeds

The failure shape: two roads that never cross

The default /healthz handler is usually its own tiny code path, written once on day one and never touched again. It doesn't call the auth service. It doesn't touch the thread pool that's actually pegged right now. It doesn't hit the downstream API that's rate-limiting you into oblivion, and it doesn't go near the model inference backend that just OOM'd on a big batch. It shares nothing — not one line — with the request path that's currently on fire. So when a shared dependency breaks (a thread pool, a connection pool, a cache, a downstream API), the health check has zero visibility into it. It was never wired to look there in the first place.

Why this is a design flaw, not bad luck

The load balancer's contract with you is dead simple: "I will only route to things you've told me can serve traffic." A stub health handler breaks that contract silently, and the load balancer never notices, because it's doing exactly what it promised — faithfully routing to every instance you personally certified as healthy. The bug was never in the LB. It's in the certification. You told it a lie, and it believed you, because believing you is the entire job.

The fix: treat the health check as a synthetic transaction

A real health check should walk through the same handler chain a real request does — same auth path, same pool acquisition, same downstream calls (or a cheap, faithful stand-in for them) — and only report healthy if that path actually succeeds. This feels like overkill the first hundred times you run it, because nothing ever breaks and you're paying a latency tax for imaginary risk. It stops feeling like overkill the one time a replica has a poisoned connection pool, keeps answering /healthz in 2ms because that handler never touches the pool, and quietly fails 100% of real requests while looking perfectly healthy the entire time.

Before / after: a concrete check you can audit today

# BEFORE — liveness only, proves nothing about capability
@app.get("/healthz")
def healthz():
    return {"status": "ok"}, 200


# AFTER — synthetic transaction through the real path
@app.get("/healthz")
def healthz():
    try:
        with timeout(500):  # ms — cheap, bounded
            token = auth.mint_synthetic_token()          # same auth code path
            conn = pool.acquire(caller="healthcheck")     # same pool as real requests
            model_backend.ping(conn, dry_run=True)         # same client, minimal payload
            pool.release(conn)
        return {"status": "ok"}, 200
    except (PoolExhausted, AuthError, ModelBackendError) as e:
        return {"status": "degraded", "reason": str(e)}, 503

That "dry_run" ping isn't a fake success dressed up to look thorough — it's the smallest real request the system supports, sent through the identical pool, auth, and client code every real user hits. If the pool is exhausted or the backend is unreachable, this call fails exactly like a real one would, and the load balancer pulls the instance out of rotation before it burns through more traffic.

Why this matters more, not less, in the AI era

This problem gets sharper the moment you're load-balancing across LLM inference replicas or an agent fleet. A vLLM or TGI pod can answer /health in microseconds while its KV cache is full, its batch queue is saturated, or it's mid-OOM on a long context — none of that ever touches the liveness handler. An agent worker can look perfectly alive while its downstream model API key is rate-limited, its vector DB connection pool is dead, or a tool call somewhere in its chain is quietly timing out. If your health check doesn't exercise that same call chain, your orchestrator keeps routing tasks to a worker that will fail every single one, and tail latency across the whole multi-agent pipeline blows up while each individual node reports green. For an AI service, "healthy" has to mean "can complete a real inference or tool-call round trip within budget" — not "the process didn't crash."

Where this fits in the 30-day arc

Day 1 gave you the load balancer as a black box that "just routes." Day 2's lesson is that the box is only ever as honest as the signal you feed it — a health check is a claim, and an untested claim is a rumor wearing a badge. Day 3 builds directly on this: once you accept that health signals can lie, the next question is how you design signals — metrics, timeouts, circuit breakers — that fail loud instead of failing quiet.

Extend your knowledge

• ▹Read Kubernetes docs on livenessProbe vs readinessProbe — they encode this exact distinction natively. In my experience, most teams only configure one of them correctly.
• ▹Look at how vLLM and TGI expose health/metrics endpoints and check whether they surface KV cache pressure or queue depth, not just process state.
• ▹Read the Google SRE book's chapter on load balancing and health checking — it's the closest thing to a canonical treatment of why a load balancer's trust in your health signal is a contract, not a formality.
• ▹Audit one of your own services today: open its /healthz handler and trace whether it shares a single line of code with the request path it's meant to protect.