Posts tagged Caching — AI × software engineering, made practical.
Consistent hashing promises that adding a node only moves 1/N of your keys — nobody tells you what happens in the gap before that node has anything cached, and why the same trap is even nastier in front of an LLM inference fleet.
A rising cache-hit ratio and a growing pile of 'site feels slow' tickets can both be true in the same quarter — here's the arithmetic behind that blind spot, and why the exact same trap is now showing up in LLM prompt caching and multi-agent pipelines.
A cached number ninety seconds old looks byte-for-byte identical to a fresh one — until an agent stakes an irreversible decision on it, and nothing in your stack was ever built to notice the difference.
A scheduling agent swore up and down it was March in September — and the root cause wasn't a bad model, it was an idempotency key that never found out the function behind it had changed.
That confident, plausible, wrong edit your agent just made wasn't a hallucination — it was a stale cache hit, and every long agentic session has the exact same one sitting inside it unmonitored.