Posts tagged LLM Infrastructure — AI × software engineering, made practical.
This post explains why exponential backoff with jitter fails when many coding agents share one API key, and how to replace it with a token-based lease coordinator, DAG-weighted queueing and checkpointed agent turns.
Consistent hashing promises that adding a node only moves 1/N of your keys — nobody tells you what happens in the gap before that node has anything cached, and why the same trap is even nastier in front of an LLM inference fleet.
Consistent hashing was built to make moving a cache key cheap — but an agent session isn't a cache key, and treating it like one turns every routine scale-up into a silent, expensive case of amnesia.
The tenant_id shard that held up for a year under real human traffic didn't survive three weeks of an unattended AI agent — here's the layer that would have turned the outage into a config change.
Every replica in your fleet is passing its health check while real requests fail, and once you see why, you'll never trust a green dashboard the same way again.